Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Stable long-term BCI-enabled communication in ALS and locked-in syndrome using 1
LFP signals 2
Tomislav Milekovic1,2,3*†#, Anish A. Sarma2,4,5†, Daniel Bacher2,4, John D. Simeral5,2,4, Jad Saab2,4, Chethan Pandarinath6,11,15, 3
Brittany L. Sorice9, Christine Blabe6, Erin M. Oakley9, Kathryn R. Tringale9, Emad Eskandar7,8, Sydney S. Cash8,9, Jaimie M. 4
Henderson6,10,15‡, Krishna V. Shenoy11,12,13,14,15,16‡, John P. Donoghue5,1,2, Leigh R. Hochberg5,2,4,8,9 5
1Department of Neuroscience, Brown University, Providence, RI, USA. 6 2Brown Institute for Brain Science, Brown University, Providence, RI, USA. 7 3Department of Fundamental Neuroscience, Faculty of Medicine, University of Geneva, Geneva, Switzerland. 8 4School of Engineering, Brown University, Providence, RI, USA. 9 5Center for Neurorestoration and Neurotechnology, Rehabilitation Research & Development, Department of Veterans Affairs, 10
Providence, RI, USA. 11 6Department of Neurosurgery, Stanford University, Stanford, CA, USA. 12 7Department of Neurosurgery, Massachusetts General Hospital, Boston, MA, USA. 13 8Harvard Medical School, Boston, MA, USA. 14 9Department of Neurology, Massachusetts General Hospital, Boston, MA, USA. 15 10Department of Neurology and Neurological Sciences, Stanford University, Stanford, CA, USA. 16 11Department of Electrical Engineering, Stanford University, Stanford, CA, USA. 17 12Neurosciences Program, Stanford University, Stanford, CA, USA. 18 13Department of Neurobiology, Stanford University, Stanford, CA, USA. 19 14Department of Bioengineering, Stanford University, Stanford, CA, USA. 20 15Stanford Neurosciences Institute, Stanford University, Stanford, CA, USA. 21 16Howard Hughes Medical Institute at Stanford University, Stanford, CA USA. 22
†,‡ These authors contributed equally. 23
* Corresponding author: [email protected] 24 # Current address: Department of Basic Neuroscience 25
Faculty of Medicine 26
University of Geneva 27
Campus Biotech, 9 chemin des Mines, 1202 Geneva, Switzerland 28
Running Head: Long-term communication BCI in people with tetraplegia 29
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
ABSTRACT 30
Restoring communication for people with locked-in syndrome remains a challenging clinical problem without a reliable 31
solution. Recent studies have shown that people with paralysis can use brain-computer interfaces (BCIs) based on intracortical 32
spiking activity to efficiently type messages. However, due to neuronal signal instability, most intracortical BCIs have required 33
frequent calibration and continuous assistance of skilled engineers to maintain performance. Here, an individual with locked-in 34
syndrome due to brainstem stroke and an individual with tetraplegia secondary to amyotrophic lateral sclerosis (ALS) used a 35
simple communication BCI based on intracortical local field potentials (LFPs) for 76 and 138 days, respectively, without 36
recalibration and without significant loss of performance. BCI spelling rates of 3.07 and 6.88 correct characters/minute allowed 37
the participants to type messages and write emails. Our results indicate that people with locked-in syndrome could soon use a 38
slow but reliable LFP-based BCI for everyday communication without ongoing intervention from a technician or caregiver. 39
NEW & NOTEWORTHY 40
This study demonstrates, for the first time, stable repeated use of an intracortical brain-computer interface by people with 41
tetraplegia over up to four and a half months. The approach uses local field potentials (LFPs), signals that may be more stable 42
than neuronal action potentials, to decode participants’ commands. Throughout the several months of evaluation, the decoder 43
remained unchanged; thus no technical interventions were required to maintain consistent brain-computer interface operation. 44
KEYWORDS 45
brain-computer interface; communication; people with locked-in syndrome; local field potentials; long-term stability 46
INTRODUCTION 47
Communication is a major goal for people with locked-in syndrome (LIS) (Plum and Posner 1972), who due to their motor 48
impairment cannot move their limbs or speak but remain conscious and mentally engaged (Bauer et al. 1979). Enabling people 49
with LIS to communicate has a positive effect on their quality of life (Bach 1993; Hecht et al. 2002), can promote reintegration 50
into society, improving the ability to lead a fulfilling, productive life (Doble et al. 2003; Laureys et al. 2005; Young and 51
McNicoll 1998). 52
Brain-computer interfaces (BCIs) – devices that translate a user’s neural activity into computer commands – have emerged as 53
promising tools for helping people with LIS communicate. Previous studies have used BCIs for communication with people 54
with LIS (Bacher et al. 2014; Birbaumer et al. 1999; Chaudhary et al. 2017; Gallegos-Ayala et al. 2014; Jarosiewicz et al. 2015; 55
Kennedy et al. 2000; Kubler et al. 1999; Kubler et al. 2001; Kubler et al. 2005; McCane et al. 2014; Nijboer et al. 2008; Sellers 56
et al. 2014; Vansteensel et al. 2016). Some of these studies relied on shaping of neural responses using biofeedback to achieve 57
accuracy permitting communication, a lengthy process lasting for weeks and, in some cases, requiring EEG preparation by 58
technicians before each use (Birbaumer et al. 1999; Chaudhary et al. 2017; Kubler et al. 1999; Kubler et al. 2005; Vansteensel et 59
al. 2016). Other studies relied on daily calibration of BCIs, which required intermittent assistance of highly skilled engineers 60
(Bacher et al. 2014; Gallegos-Ayala et al. 2014; Nijboer et al. 2008; Sellers et al. 2014). A BCI system capable of providing 61
independent communication to people with LIS, where technical support or interaction with a caretaker is not required 62
frequently, still needs to be developed. 63
Among these approaches, BCIs based on intracortically-recorded action potentials have been used to achieve communication by 64
a person with incomplete LIS (Bacher et al. 2014; Jarosiewicz et al. 2015) or ALS (Gilja et al. 2015; Jarosiewicz et al. 2015; 65
Pandarinath et al. 2017); and enable people with tetraplegia to control robotic arms (Aflalo et al. 2015; Collinger et al. 2013; 66
Hochberg et al. 2012; Hochberg et al. 2006; Wodlinger et al. 2015) or their own muscles using functional electrical stimulation 67
(Ajiboye et al. 2017; Bouton et al. 2016). In addition to spiking signals, intracortically implanted microelectrodes also record 68
local field potentials (LFPs), neuronal signals thought to represent the summed activity of neuronal populations in the vicinity of 69
the microelectrode (Buzsaki et al. 2012). LFPs recorded in the motor cortex of people with paralysis modulate with attempted 70
movements in multiple frequency bands (Ajiboye et al. 2012; Hochberg et al. 2006) and have enabled a person with LIS to spell 71
using a BCI (Kennedy et al. 2000). Information about arm actions has been found in non-human primates in both signal 72
modalities (Ajiboye et al. 2012; Bansal et al. 2012; Bansal et al. 2011; Flint et al. 2012a; Flint et al. 2012b; Flint et al. 2013; 73
Heldman et al. 2006; Mehring et al. 2003; Rickert et al. 2005; Stark and Abeles 2007). 74
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Sorted spike recordings from intracortical implants can be unstable across periods of days to months (Dickey et al. 2009; Perge 75
et al. 2013; Perge et al. 2014; Simeral et al. 2011). Signals from individual neurons can be lost due to array movements as small 76
as a few tens of µm in relation to the brain. The number of recorded well-discriminated neurons can also diminish over time, 77
due to degradation of the electrode materials, biological responses to the sensor, or other factors (Barrese et al. 2013; Bjornsson 78
et al. 2006; Shain et al. 2003; Simeral et al. 2011). Partly in response, previous studies have predominantly used frequent 79
recalibration of decoders, often by highly trained engineers and researchers on site, to maintain a high level of performance 80
despite signal instability. While recent studies showed that able-bodied non-human primates can use BCIs with an unchanged 81
decoder for over six months without a significant drop in performance (Flint et al. 2013; Nuyujukian et al. 2014), other recent 82
studies pointed towards comparatively greater intracortical signal variation in people (Jarosiewicz et al. 2015; Perge et al. 2014). 83
One approach to this problem is to build an adaptive decoder (Jarosiewicz et al. 2015). In addition, multi-unit threshold 84
crossings have been proposed as a more stable alternative to sorted spikes (Flint et al. 2016; Fraser et al. 2009; Gilja et al. 2015). 85
Alternatively, LFP signals could be employed to achieve long-term reliable BCI control. 86
Here we demonstrate stable, long-term LFP-based communication in two people with tetraplegia. Our study included an 87
individual with LIS, unable to move his limbs or speak, and an individual with progressive tetraplegia secondary to ALS. They 88
used an LFP-based BCI with a decoder unchanged for 76 and 138 days to communicate with family members, type messages 89
and send emails, without a loss of performance (Figure 1). Our study demonstrates that an intracortical LFP-based BCI can be 90
used for independent communication without the need for re-calibration, thereby reducing the need for caregiver and/or family 91
intervention during communication. 92
Our results provide evidence that intracortical BCIs could be used in-home, for months and with minimal technical oversight, all 93
of which are necessary for a practical communication system. With this new benchmark for stable, LFP-based neural control, 94
future work can continue to address more complex control modalities and higher-resolution neural signals to achieve stable 95
control of more complex neuroprosthetic systems. This study is a step towards a reliable and robust BCI that will allow people 96
with LIS to communicate independently and, therefore, provide greater and more extensive interactions with their friends, 97
family, and caregivers. 98
MATERIALS AND METHODS 99
Participants 100
Permission for these studies was granted by the US Food and Drug Administration (Investigational Device Exemption) and 101
Institutional Review Boards of Stanford University (protocol # 20804), Partners Healthcare/Massachusetts General Hospital 102
(2011P001036), Providence VA Medical Center (2011-009), and Brown University (0809992560). The participants in this study 103
were: T2, a man with classic LIS secondary to a brainstem stroke; and T6, a woman with tetraplegia resultant from ALS. They 104
were enrolled in a pilot clinical trial of the BrainGate Neural Interface System (http://www. 105
clinicaltrials.gov/ct2/show/NCT00912041). As part of the trial, the participants received one 96-channel intracortical multi-106
electrode array (Blackrock Microsystems) in the arm area of the dominant precentral gyrus (Yousry et al. 1997) using a 107
previously described procedure (Hochberg et al. 2006). Both participants were right handed. At the time of the study, T2 was 108
fed through a percutaneous gastrostomy tube and had nighttime-only volume-control lung ventilation to support his breathing. 109
His respiratory drive remained intact throughout the study. T6 had a tracheostomy and on-demand ventilation, though she did 110
not require constant ventilatory support (mostly at night, similar to T2). She was able to eat. Her ALSFRS-R score was 16. T6 111
could speak and use accessibility devices to control a computer with limited hand movement. Neural recordings were collected, 112
stored and processes for online BCI control as described previously (Simeral et al. 2011). 113
Task 114
Participants took part in research sessions in their homes during which they interacted with the FlashSpeller text-entry 115
application through selections: discrete events identified their attempts to perform a particular movement based on their 116
neuronal activity. The FlashSpeller scanned through character-entry options and other options such as backspace, space, word 117
completion, text-to-speech, or email. An option was presented for 1.5s for T6 and 2s for T2. Presentations of two options were 118
separated by a period of 0.1s for T6 and 0.3s for T2. If selected, the option button turned green for 0.9s. As selected movements, 119
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
referred to as “click actions” in the remainder of the text, T2 attempted to squeeze his right hand and T6 flexed her right index 120
finger. 121
Each individual session was divided into blocks lasting up to twelve minutes each. We asked the participant to perform one of 122
four tasks, depending on the block type: (i) normalization – observe the FlashSpeller automatically typing a phrase while not 123
interacting in any way; (ii) open-loop calibration – observe the FlashSpeller automatically typing a phrase while trying to 124
perform click actions as if to spell the phrase; (iii) copy-phrase – control the FlashSpeller using a BCI with either a same-day 125
decoder or the historical decoder to type an assigned phrase, and (iv) free-spelling - control the FlashSpeller using a BCI with a 126
historical decoder to type a message of choice. 127
Three types of sessions were run in this study: collection, comparison and communication (Figure 2). All sessions started with a 128
normalization block. In collection sessions, we collected open-loop data through open-loop calibration blocks that were used 129
afterwards to calibrate a historical decoder. In comparison sessions, after three open-loop calibration blocks were used to 130
calibrate a same-day decoder, the historical and same-day decoder were compared in a double-blind fashion using copy-phrase 131
blocks. Occasionally, comparison sessions would end with one or more free-spelling blocks. In communication sessions, the 132
historical decoder was used for free typing. 133
Preprocessing 134
Neuronal recordings were first down sampled to 15kHz. We then computed power spectral densities for each individual channel 135
to exclude channels that exhibited large peaks at a line noise frequency (60Hz) or its harmonics. We then referenced the 136
recordings to the common average calculated over the remaining low-noise channels (80 channels for T2 and 33 channels for 137
T6). 138
Spectral amplitude normalization 139
At the beginning of every session, the participant completed a normalization block during which he was instructed to observe 140
the monitor during the presentation of an open-loop calibration block, but not to respond to any cues. Recordings from this 141
block were first pre-processed, then low-pass filtered using a 30th order windowed linear-phase finite impulse response filter 142
(WLPFIR) and then downsampled to 1kHz. A short time Fourier transform (STFT; 256ms window; using a Hamming window) 143
was then used to estimate spectral amplitudes in 3.91Hz wide frequency bins in 20ms steps. For each frequency bin, we 144
calculated the average amplitude recorded by a single channel over the normalization block. This produced a normalization 145
matrix of size 96 x 129, where 96 was the number of channels and 129 was the number of frequency bins. 146
Determining frequency bands of the intermediate and high frequency components 147
To estimate the appropriate frequency bands that would provide selection-related neuronal responses, we processed the 148
recordings from all open-loop calibration blocks by pre-processing the data, low-pass filtering it using a 30th order WLPFIR, 149
down sampling it to 1kHz and then using a STFT (256ms window; using a Hamming window) to estimate amplitudes for each 150
of the frequency bins in 50ms steps. Amplitudes were then normalized using the spectral amplitude normalization calculated 151
from the normalization block. Selection-related epochs were then defined as the time from the selection cue until 1.8 seconds 152
after the cue for T2 and until 1.5 seconds after the cue for T6. Amplitudes calculated during the remaining time (no cue or cue 153
that should not be responded to) were taken as baseline. We then calculated the signal to noise ratio (SNR) between cue-related 154
neuronal responses and the baseline activity for each electrode el and for each of the frequency bands, defined by the starting 155
and ending bin of a given frequency band, bS and bE respectively (Milekovic et al. 2012). 156
1ˆ ; , ;
E
S
bel el
i S E i
b bE S
X t b b X t tc bb b
157
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
1
2
1
2
1 ˆ; , ; ,
1 ˆ; , ; , ; ,
1 ˆ, ; ,
1 ˆ, ; , ,
B
B
Kel el
C S E i S E
i
Kel el
C S E i S E C S E
i
el el
B S E S E
t T
el el
B S E S E B S E
t T
t b b X t b bK
t b b X t b b t b bK
b b X t b bT
b b X t b b b bT
158
; , ,; ,
; , ,
el el
C S E B S Eel
S E el el
C S E B S E
t b b b bSNR t b b
t b b b b
159
where Xel(t;b) is the amplitude in the frequency bin b and estimated from a window that ends at time t on an electrode el; K is 160
the number of all selection-related epochs in a session; TB is the set of all time points that belong to the baseline; and T is the 161
cardinal size of TB. To reach a single value representing the signal to noise ratio as a function of a frequency band, we evaluated 162
the frequency band SNR by calculating the maximum SNR over time and averaging it over all electrodes: 163
1, ; ,
max arg ; ,
el
S E MAX S E
elel
el
MAX S Et
SNR b b SNR t b bN
t SNR t b b
164
This procedure was performed for collection sessions from session days 452, 453, 460, 467 and 474 for T2; and session days 165
313, 322, 336, 348 and 355 for T6. 166
These same sessions were used to build the historical decoders for each of the participants. All of these sessions showed three 167
well isolated local maxima of the SNR(bS,bE) (Figure 3). One maximum was present in the very low frequencies, typically 0-168
6Hz. Selection-related neuronal responses in this band were captured by the low frequency component (LFC) obtained by low-169
pass filtering the common—average referenced signals using the filter with a -3dB cutoff frequency of 4.63Hz. We determined 170
the start and end bins for the other two frequency bands for each of the five sessions and used the average as the start and end 171
bins for the intermediate and high frequency components (IFC and HFC, Table 1). 172
Extraction of neural features 173
We then extracted three different components of the LFPs: (i) low-pass filtered component (LFC), (ii) intermediate frequency 174
component (IFC) and (iii) high frequency component (HFC). LFC was extracted by low-pass filtering the neural recordings (-175
3dB cutoff frequency of 4.63Hz). To extract IFC and HFC, we first calculated the spectral amplitudes of neural recordings using 176
a short time Fourier transform (STFT). We then divided the amplitude in each frequency bin by its mean in the normalization 177
block and averaged these normalized amplitudes across the IFC and HFC bands. We then defined a neuronal feature, Φf, at time 178
t as one of LFC, IFC or HFC derived from the electrode el: 179
el
el
f
el
LFC t
t IFC t
HFC t
180
To account for the drifts of the neuronal features, we z-scored the features using recursive estimates of the mean and variance 181
(see “Adaptive feature normalization” section in the Appendix). 182
Decoder calibration and feature selection 183
Data for historical decoder calibration were collected over several collection sessions spanning up to 42 days. This data was then 184
used to rank all 288 available features according to the signal-to-noise ratio (SNR) between peri-click and baseline feature 185
values (see “SNR used for selection of neural features” section in the Appendix). We then selected the top 5 LFC, top 5 IFC, 186
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
and top 5 HFC features, followed by the top 35 remaining features for a total of 50. This procedure selected 16 LFC, 5 IFC and 187
29 HFC features for T2, and 5 LFC, 5 IFC and 40 HFC features for T6. 188
These features were then used to calibrate a regularized linear discriminant analysis (rLDA) classifier that calculated the 189
selection probability for a given set of feature values (see “Calibration of decoding algorithms” section in the Appendix). 190
Parameters of this historical decoder were optimized using the normalized mutual information CYX between the decoded 191
selection and selection cues provided during the open-loop calibration blocks of the collection sessions. A same-day decoder 192
was calibrated to data from three open-loop calibration blocks recorded on that day following the same procedure. 193
Decoding algorithm 194
A regularized linear discriminant analysis (rLDA) classifier, i.e. either a historical or a same-day decoder, was used to calculate 195
the probability of a selection every 20ms. When the probability crossed a threshold of 95%, a selection command was sent to the 196
text-entry application. After the selection was detected, a refractory period of 0.5 seconds was imposed, during which another 197
selection would not be detected. In addition, the probability had to drop to 75% or below before another selection could be 198
detected. 199
Cross-session stability of LFP responses 200
To investigate the stability of LFP signals over time, we calculated the correlation coefficient between mean single-feature LFP 201
responses during click actions recorded on two sessions s1 and s2, rf(s1,s2) (see “Measuring cross-session stability of LFP 202
responses” section in the Appendix). To extract an overall estimate of stability of LFP signals between two sessions, we then 203
calculated the median of rf(s1,s2) over all features, Mr(s1,s2). To estimate the trends of LFP signal stability, we examined the 204
relationship between the Mr(s1,s2) and the number of days between two sessions. A slope b of the line fitted to Mr(s1,s2) as a 205
function of time using ordinary least squares gave us the estimate on the rate of instability of the LFP signals. 206
Performance measures 207
Decoding accuracy was measured using normalized mutual information CYX (Coombs et al. 1970), a metric based on 208
information theory, between the decoded selections and either the selection cues provided to the participants, in open-loop 209
blocks, or the assumed selections a participant wanted to perform based on the final spelled phrase in copy-phrase and free-210
spelling blocks. CYX was used to optimize the parameters of a decoding algorithm during calibration and to obtain a measure of 211
spelling performance that was independent of the spelled phrase. The spelling performance of our communication interface was 212
measured by counting the number of correctly typed characters per minute. For a more elaborate derivation of these measures, 213
see “Measuring spelling and decoding performance” section in the Appendix. 214
Retrospective comparison to a hypothetical single electrode 215
It has been shown that people can use BCIs based on neural signals recorded from the surface of the cortex 216
(electrocorticography) for rapid spelling (Brunner et al. 2011) and control of computer cursors and robotic arms (Wang et al. 217
2013). For a simple, switch-based communication demonstrated in this study, a BCI based on single electrocorticographic 218
electrode might perform comparably to a BCI based on LFPs recorded from the whole implanted intracortical array, while 219
covering a similar cortical surface. We explored this secondary question by simulating the case where only one broadband 220
signal - the average of LFP signals recorded across the intracortical array - was used for neural decoding. In a retrospective 221
analysis, this average LFP signal was used to define intermediate and high frequency bands and to calibrate an average signal 222
decoder based on open-loop calibration blocks of the data collection sessions. We then used this decoder to decode participant’s 223
actions during the open-loop blocks of the comparison sessions. Since participant did not receive any decoding feedback during 224
these blocks, this allowed for a fair comparison between this average signal decoder and the historical decoder that used LFPs 225
from multiple electrodes. Furthermore, while the closed-loop blocks could vary in size, the open-loop calibration sequence was 226
standardized for each participant over all sessions. Therefore, the same amount of data was used to calibrate (fifteen OLC 227
blocks) and test (three OLC blocks) both historical and average signal decoders on each session. We compared the decoders by 228
calculating CYX from offline decoding. In order to prevent an undue penalty on the average signal decoder from particularly 229
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
noisy signals originating from one or more electrodes, the average LFP was calculated using only the LFPs from the channels 230
that contributed to the common average in the multiple-electrode case. 231
RESULTS 232
Interaction of participants with a FlashSpeller communication BCI 233
Our study shows that two people with tetraplegia, one of whom was locked-in, can spell text at a rate of about a word per minute 234
using a FlashSpeller BCI with a decoder that remained unchanged, without a drop in performance, for time periods of several 235
months. This BCI decoder was calibrated on data collected during five one-day sessions (spanning a period of up to 42 days) 236
and was then used unchanged for communication for up to 138 days afterwards. To spell messages on a computer monitor, two 237
participants - an individual with LIS secondary to brainstem stroke (T2) and an individual with tetraplegia secondary to ALS 238
(T6) - used a custom text-entry application, called FlashSpeller, with an LFP-based BCI (Figure 1, 4; Table 2). FlashSpeller was 239
designed for individuals with limited or no eye movement, which occurs commonly in people with LIS. In the middle of the 240
screen, a large grey button displayed options for 1.5-2s. To select the currently displayed option, i.e. to “click”, the participant 241
attempted or executed a chosen movement. LFP responses elicited during this “click actions” were recorded from the motor 242
cortex and decoded as clicks by the BCI. 243
The study lasted for 99 and 181 days in total for T2 and T6, respectively. At the beginning of the study, we ran “data collection” 244
sessions to accumulate a large number of neuronal responses during participants’ click actions, which were used to calibrate a 245
historical decoder. Data collection sessions were followed by “comparison” sessions, in which the historical decoder was 246
compared to a decoder calibrated on data recorded on the same day (“same-day” decoder) by measuring the participants’ 247
performance in spelling assigned phrases during “copy- spelling” blocks. Often, at the end of comparison sessions, participants 248
used the historical decoder to freely spell messages during “free-spelling” blocks. We also ran separate “communication” 249
sessions, in which the participants used the historical decoder to spell messages. The historical decoder was calibrated only once 250
for each participant and was used in all comparison sessions and all communication sessions. 251
To create BCI decoders, we identified three different features of LFPs that modulated in relation to participants’ attempts to 252
perform click actions (Figure 5, 6a). The low-pass filtered component (continuous signal filtered using a low-pass filter below 253
4.63Hz) reflected a slow, large-amplitude deflection related to the selection actions. Activity in the intermediate frequency 254
component (normalized spectral amplitude in 14-33Hz for T2 and 21-41Hz for T6) showed a decrease following the cue, while 255
the high frequency component (normalized spectral amplitude in 53-193Hz for T2 61-283Hz for T6) showed an increase in 256
activity. Modulations of these features could be distinguished from LFPs recorded on individual trials and were used to detect 257
participants’ intentions to select the presented options (Figure 4, Movie 1-8). 258
In addition to quantifying the performance and stability of our BCI system, we investigated different causes that led to stable 259
long-term spelling performance, formulated in the following questions. (i) Was the stable performance a direct consequence of 260
the stability of LFP responses that occurred during click actions? (ii) Does the resolution of intracortical recordings substantially 261
alter the decoding performance as compared to a hypothetical electrocortigraphic electrode? (iii) Was the design of our decoder 262
responsible for stable decoding performance despite the instability of LFP responses during click actions? 263
Stability of LFP responses to click actions over the whole microelectrode array 264
LFP responses during click actions in features used for decoding were similar over the duration of the study (Figure 5, 6a). We 265
also measured the stability of LFP signals recorded over the whole MEA over time as follows. For each session, we calculated 266
the mean LFP response time-course for each feature, ranging from the click cue to 1.5s after the click cue, across all click 267
actions recorded during that session’s open-loop calibration blocks (34 and 68 cue responses for sessions T2 466 and T2 467, 268
respectively; 51 cue responses for all other sessions). For each feature’s mean,we then calculated the correlation coefficient 269
between any pair of sessions, hereafter referred to as “single-feature cross-session correlation coefficient”. Finally, we 270
calculated the median cross-session correlation coefficient across all 288 features (96 electrodes times three feature types), thus 271
obtaining a single value for each session pair. 272
This median single-feature cross-session correlation coefficient did not change significantly with inter-session period for T2 (b 273
= -0.02year-1; P = 0.78; Spearman’s rank correlation test) while it significantly diminished with the inter-session period for T6 274
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
(T6: b = -0.21year-1; P < 0.05; testing non-zero correlation between the correlation coefficient and inter-session period, 275
Spearman’s rank correlation test; Figure 6b). 276
Stability of detection based on LFP neuronal responses 277
We tested the stability of the LFP-based detection of clicks by using a decoder, calibrated on all open-loop calibration blocks 278
from one collection or comparison session, to detect clicks on open-loop calibration blocks from another session in the future or 279
in the past. The standardized calibration sequence for each participant over all sessions ensured that the same amount of data 280
was used to calibrate (three OLC blocks) and test (three OLC blocks) each decoder. Since participants did not control the 281
spelling during open-loop blocks and, therefore, did not receive any feedback on their performance, they could not adapt their 282
strategy to modify the decoding performance. Decoding performance was measured using normalized mutual information (CYX) 283
between the decoded clicks and the clicks that would lead to errorless spelling (Coombs et al. 1970; Milekovic et al. 2013). 284
For both participants, offline detection performance, as measured by CYX, was variable, but did not degrade as a function of the 285
number of days since decoder calibration, ΔT (Figure 7). Correlation between CYX and ΔT was not significant (T2: P=0.82; T6: 286
P=0.46, Spearman’s rank test). Average CYX calculated this way was 0.68 and 0.72 for T2 and T6, respectively. 287
Spelling performance and other performance measures 288
Both study participants controlled the FlashSpeller using LFP signals and a historical decoder to communicate by spelling 289
messages (Figure 8). The average spelling performance, as measured by the number of correct characters per minute (CCPM), 290
during free-spelling blocks using word completion was 3.07±0.20CCPM and 6.88±0.33CCPM for T2 and T6, respectively. 291
These rates enabled participants to spell full sentences within several minutes. The period over which the unchanged historical 292
decoder was used by the participants for communication spanned 76 and 138 days for T2 and T6, respectively. During this 293
period the participants’ spelling performance did not change significantly (P value for testing non-zero correlation between 294
spelling performances during free-spelling blocks and historical decoder age: T2: P = 0.40; T6: P = 0.73; Spearman’s rank 295
correlation test). 296
Spelling performance takes into account both the accuracy of the selection detections and the difficulty of the typed phrases. It 297
may be possible that participants chose to type easier phrases during the sessions in which the decoders were not as accurate, 298
thus masking the instability of LFP responses used for detection of selections. We used CYX to compare the performance of 299
decoders irrespective of the typed phrases (Figure 9). Over all sessions with free-spelling blocks in which word completion was 300
used, CYX in those blocks did not change significantly (P value for testing non-zero correlation between CYX during free-spelling 301
blocks and historical decoder age: T2: P = 0.33; T6: P = 0.13; Spearman’s rank correlation test). The historical decoder spelling 302
performance was not significantly different from the same-day decoder performance (mean spelling performance difference 303
during copy-phrase blocks over all comparison session: T2: 0.19±0.18CCPM, P = 0.10; T6: 0.03±0.22CCPM, P = 0.85; t-test). 304
In addition, spelling performance of both the historical and same-day decoders during copy-phrase blocks did not change 305
significantly over the time we conducted the comparison sessions (P value for testing non-zero correlation between spelling 306
performances during copy-phrase blocks and historical decoder age: same-day decoder: T2: P = 0.68; T6: P = 0.95; historical 307
decoder: T2: P = 0.18; T6: P = 0.48; Spearman’s rank correlation test). Further, CYX did not change significantly over the same 308
time (P value for testing non-zero correlation between CYX during copy-phrase blocks and historical decoder age: same-day 309
decoder: T2: P = 0.78; T6: P = 0.67; historical decoder: T2: P = 0.42; T6: P = 0.76; Spearman’s rank correlation test), thus 310
further confirming that LFP responses remained stable over the tested period. 311
The participants self-assessed the performance of both historical and same-day decoders on a scale from 1 to 10 (best) after each 312
copy-phrase block (Figure 10). Scores were the same for both decoders for all participants. On average, T2 scored the historical 313
decoder with 9.50±0.15 and the same-day decoder with 9.34±0.37, T6 scored the historical decoder with 9.23±0.43 and the 314
same-day decoder with 8.94±0.47. The score differences in all participants were not significant (T2: P = 1; Wilcoxon signed 315
rank test; T6: P = 0.47; Wilcoxon ranksum test). The score in one of the blocks in session T6 473 was not recorded. 316
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Contribution of intracortical resolution 317
We compared decoding performance of the average signal decoder, which used the average LFP across the array and 318
represented a simulated single electrocorticographic electrode signal, to the historical decoder. For both participants, the 319
historical decoder was more effective (P < 0.05, bootstrap test; Figure 11). 320
DISCUSSION 321
Our results demonstrate stable, long-term LFP-based communication with an unchanged decoder in two people with severe 322
paralysis. The unchanged decoder was effective for as long as 138 days. Our participants included one person with long-323
standing locked-in syndrome (T2), who was unable to move or verbally communicate, and one person with progressive 324
tetraplegia secondary to ALS (T6). Importantly, over the duration of the study, spelling performance did not change 325
significantly. Both participants used the BCI to communicate with family members, caretakers and researchers by typing full 326
sentences. T2 also used the BCI to write and send emails - the participant first selected the email recipient and title using a 327
caretaker-assisted spelling board, then wrote the email body using FlashSpeller. When writing emails a “send” button was 328
present at the end of each FlashSpeller cycle, which could be selected by T2. T2 requested the use of a send button, because he 329
enjoyed using the FlashSpeller BCI system for personal email communication. T6 used other electronic communication systems 330
for email, and preferred to use the FlashSpeller to communicate messages to the research community. Decoders calibrated on 331
the same day did not achieve higher performance when compared to the unchanged historical decoder in double-blind 332
comparisons. During these comparisons, participants scored the performance of historical and same-day decoders equally well. 333
Together, these results show that an LFP-based communication BCI could soon provide people with locked-in syndrome with a 334
slow but reliable ability to communicate with minimum technical supervision or caretaker intervention. This LFP-based 335
decoding could also be used to enhance higher-throughput, intracortical spike-based communication interfaces (Bacher et al. 336
2014; Gilja et al. 2015; Jarosiewicz et al. 2015; Pandarinath et al. 2017). 337
While severely paralyzed, our participants were still able to move their eyes and head and neck muscles. Since eye movements 338
and head and neck muscle activity were not controlled for, artifacts originating from eye, head and neck movements could have 339
been introduced into our recordings. Nonetheless, such artifacts were likely to be common across signals recorded from a small 340
patch of motor cortex covered by one MEA. Therefore, we minimized their influence by using common-average re-referencing 341
step in our signal processing. 342
Stability of LFP signals 343
We show that LFPs recorded from a chronically implanted microelectrode array can serve as the basis for stable long-term 344
control of a switch-based communication interface. A major concern with chronic multielectrode recordings has been their 345
suitability for long-term prosthetic control. It is difficult to reliably detect and isolate activity of single units within a day or from 346
one day to the next (Dickey et al. 2009; Perge et al. 2013). Furthermore, changes to the recording environment can lead to a 347
long-term decline in the number of detectable single units (Chestek et al. 2011; Flint et al. 2012b; Moran 2010). However, when 348
available, isolated single units appear to have a predictable long-term relationship to behaviour, suggesting a stable underlying 349
map (Chestek et al. 2007). As such, two signal types have been studied as potentially more stable than isolated single units: 350
multi-unit activity derived from threshold crossings of a high-frequency band (0.5-7.5kHz) signal (Chestek et al. 2011; Flint et 351
al. 2013), and LFPs (Flint et al. 2012b; Perge et al. 2014). At present, the evidence suggests that while threshold crossings are 352
more stable neuroprosthetic control signal than was expected a decade ago, LFPs are still more stable from day to day (Flint et 353
al. 2016) and may persist when threshold crossings are lost (Flint et al. 2012b; Simeral et al. 2011). Therefore, for studies of 354
long-term reliability of neuroprosthetic control, LFPs continue to be a signal of interest. 355
In our study, it was important to distinguish between contributions to stable communication performance from signal 356
processing, signal selection, and changes in participant control strategy. We found that LFP signals recorded over the whole 357
microelectrode array showed long-term stability only for participant T2, suggesting that T2's stable communication rates 358
resulted from signal processing. For participant T6, we found significant changes of LFP modulations over time across the entire 359
array (Figure 6b). This result shows that the underlying signals, i.e. the processed LFPs, could not account alone for T6's stable 360
communication rates. Either decoder design (that is, feature selection and feature weighting) or T6's control strategy might 361
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
account for this discrepancy. We next tested decoding performance on open-loop blocks, during which participants could not 362
adapt control strategies due to lack of feedback (Figure 7). We found that open-loop decoding performance did not degrade over 363
time for T2 or T6, leading us to conclude that signal selection and feature weights, rather than a change in control strategy, 364
accounted for T6's stable communication rates. Therefore, population-level approaches can continue to decode a stable 365
behaviour-to-neuroprosthetic-command map even when the fidelity of the signal has declined on average. 366
Selection of LFP features to achieve BCI control 367
One objective of this study was to develop an LFP feature selection method that readily generalized across participants. 368
Following the data-driven process to identify informative LFP frequency bands, we derived three features on each channel (288 369
in total). We considered the case where features in a given frequency band were separately informative but highly cross-370
correlated, while separately less informative features from other frequency bands might provide independent novel information. 371
One way to capture the diversity of information over all available features is to use all features for decoding. However, this 372
approach increases the risk of overfitting the decoder to the training data. Furthermore, we were constrained by the need to 373
perform all signal processing and decoding computations online, which made it necessary to perform feature selection. 374
Previous studies in decoding movement direction from motor cortical spiking activity and LFPs in monkeys and humans 375
showed that the marginal benefit of added features diminishes substantially in the 30-50 range (Bansal et al. 2012; Bansal et al. 376
2011; Gilja et al. 2015; Wessberg et al. 2000). Therefore, we made a conservative estimate that 50 features will still contain the 377
majority of movement attempt information. We scored the “informativeness” of each feature by its SNR. To capture diverse 378
information potentially available in different frequency bands, we first selected five features with the highest SNR from each 379
frequency band. We supplemented those 15 features with additional 35 features with the highest SNR irrespective of the 380
frequency band to select the total of 50 features. We further mitigated the risk of overfitting the decoder to the training data by 381
using regularization. 382
Utility of switch-based BCIs 383
Although the primary finding of this study relates to the stability of neurophysiological signals, the efficacy of simple switch-384
based communication BCIs is also worth underscoring. ALS at moderate and severe stages may result in cortical and/or other 385
changes that could disrupt BCI function, including attention deficits (Pinkhardt et al. 2008; Schreiber et al. 2005; Vieregge et al. 386
1999) and other cognitive impairments (Cipresso et al. 2012). Furthermore, people with ALS can experience problems with 387
vision due to poor conjugate gaze control and corneal injury (Murguialday et al. 2011). People with LIS secondary to brainstem 388
stroke can also develop oculomotor and visual impairments (Caplan 1980; Sellers et al. 2014). Lastly, people with 389
communication deficits related to neurological disease or injury (for instance, related to traumatic brain injury) may have a 390
range of sensory and cognitive abilities, but could still benefit from a reliable method of communication for basic needs. The 391
switch-based communication method described here could, for example, also be used with an auditory interface. For all these 392
reasons, simple switch-based communication BCIs may be better suited for some people with LIS, ALS, or other neurological 393
disorders, than more complex communication BCIs designed for high communication throughput, including those previously 394
reported by our group. Additionally, approaches and algorithms that can reliably decode state information from intracortical 395
recordings may point towards more expansive use of BCIs for non-typing communication goals such as control of household 396
technology and emergency alarms, for which reliability is paramount (Kageyama et al. 2014). 397
The development of a fully implantable amplification and telemetry system is a likely precursor to the widespread adoption of 398
BCI technologies based on intracortical recordings. Safety and efficacy of such devices are currently being studied in animal 399
models (Borton et al. 2013; Capogrosso et al. 2016; Yin et al. 2014); and translation for use in people is ongoing. Future work 400
should also aim to implement the signal normalization block as an ongoing, self-updating process. 401
Comparison to other communication BCI studies 402
Other approaches using different signal modalities have investigated communication BCIs (Chaudhary et al. 2016; Wolpaw and 403
Wolpaw 2012). Nevertheless, recent studies have demonstrated the superior performance of intracortical communication BCIs 404
in people with tetraplegia when compared to prior studies (Gilja et al. 2015; Jarosiewicz et al. 2015; Pandarinath et al. 2017). 405
However, for successful clinical applications, these systems have to provide stable performance with minimal technical 406
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
oversight, despite the instability of intracortical signal recordings (Jarosiewicz et al. 2015; Perge et al. 2014). An approach taken 407
by a recent study from our group used an adaptive decoder that conformed to the signal changes to generate stable performance, 408
which allowed the participant with tetraplegia to use a communication BCI without a loss in performance and without 409
reinitializing the decoder over a period of 42 days (Jarosiewicz et al. 2015). In the current study, we followed an alternate 410
approach of harnessing intracortical LFP signals, which are thought to be more stable than the neuronal spiking signals. Despite 411
changes in LFP signals recorded across the entire array in one participant, the online decoding performance remained stable for 412
both participants throughout the duration of the study, over a period of 76 days for one and 138 days for the other. 413
In a recent study, a person with LIS used a BCI based on ECoG signals to spell text at a rate of 1.82 CCPM over 70 days with an 414
unchanged decoder and without a drop in performance (Vansteensel et al. 2016). In our current study, a person with LIS used an 415
intracortical LFP-based BCI with an unchanged decoder and without a drop of performance over a similar period (76 days), but 416
achieved 69% higher performance (3.07 CCPM). A previous study from our group has shown that people with LIS can use a 417
communication BCI based on intracortical spiking signals, but used daily calibration of decoders to preserve performance 418
(Bacher et al. 2014). Another study from our group demonstrated stable spelling performance using an intracortical BCI in an 419
individual with tetraplegia who could still speak (Jarosiewicz et al. 2015). This study thus complements recent intracortical BCI 420
communication studies. While a previous study (Jarosiewicz et al. 2015) adapted the decoder based on data collected on the 421
current day, in this study the historical decoder was calibrated with data pooled across about a month. Therefore, our calibration 422
strategy inherently favored features that remained stable over longer periods. This strategy might also benefit spike-based 423
control. The adaptation strategy used in the previous study relies on reasonable neural control in a point-and-click task after a 424
days-long period of disuse, and would likely benefit from the methods developed here. LFP signals and the associated data 425
collection, decoder calibration, signal selection and signal processing strategies developed in our study could contribute not only 426
to point-and-click BCI communication but also to more general neuroprosthetic control. 427
Other studies have investigated the use of communication BCIs based on other signal modalities as a communication tool for 428
people with LIS (Bacher et al. 2014; Birbaumer et al. 1999; Kubler et al. 1999; Kubler et al. 2005; Nijboer et al. 2008; Sellers et 429
al. 2014). A prior study showed that two individuals with LIS secondary to ALS could communicate using an EEG-based BCI, 430
with one of the participants typing 510 characters for over 16 hours (Birbaumer et al. 1999). Another recent study has shown 431
that an individual with LIS secondary to brainstem stroke (state similar to that of T2) can communicate using an EEG-based 432
BCI with a performance of below 1 CCPM (Sellers et al. 2014). In our study, all participants, including a person with LIS, typed 433
at a higher rate. 434
Our system provided higher performance than reported by any EEG- or ECoG-based spellers tested on a similar population of 435
subjects, i.e. people with Amyotrophic Lateral Sclerosis and people with Locked-in Syndrome (Arya et al. 2013; Bai et al. 2010; 436
Brunner et al. 2010; Chen et al. 2015; Fazel-Rezai et al. 2012; Hamer et al. 2002; Hedegard et al. 2014; Hill et al. 2006; 437
Hinterberger et al. 2005; Kubler and Birbaumer 2008; Mainsah et al. 2015; Marchetti and Priftis 2014; McCane et al. 2014; 438
Miner et al. 1998; Nijboer et al. 2008; Pires et al. 2012; Vansteensel et al. 2016) (see Table 3 for detailed list of spelling rates 439
reported in other studies). In the two above studies in which the communication rates were similar (McCane et al. 2014; Pires et 440
al. 2012), our BCI communication system was used to type considerably longer and more complex messages, demonstrating that 441
people with paralysis can use our BCI system for spelling, and that intracortical recording may be an effective solution. 442
Two recent studies showed that near infrared spectroscopy (NIRS) can be used to detect recognition of the veracity of a Boolean 443
choice in people with complete LIS (Chaudhary et al. 2017; Gallegos-Ayala et al. 2014), which suggests that some people with 444
complete LIS might remain capable of communication with adequate technology. However, the rate of yes/no detections was 445
slow (around 20-25s per response, responses about 75% correct), possibly reflecting the low signal-to-noise ratio of NIRS 446
responses. Other attempts to provide individuals with complete LIS secondary to ALS with the ability to communicate using 447
electrocorticography-based BCIs were not yet successful (Hill et al. 2006; Marchetti and Priftis 2014). It remains to be seen 448
whether a LFP-based communication BCI could be used by someone with complete LIS, but our study with participants who 449
have ALS or classic LIS suggests that this approach could be effective. 450
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
ACKNOWLEDGMENTS 451
We thank participants T2 and T6 for their dedication to this research. We also thank L. Barefoot, D. Rosler, S. Mernoff and M. 452
Bowker for their contributions to this research. The research was supported by the National Institutes of Health: National 453
Institute on Deafness and Other Communication Disorders - NIDCD (R01DC009899); Rehabilitation Research and 454
Development Service, Department of Veterans Affairs (B6453R and B6459L); Massachusetts General Hospital (MGH) - Deane 455
Institute for Integrated Research on Atrial Fibrillation and Stroke; Joseph Martin Prize for Basic Research; The Executive 456
Committee on Research (ECOR) of Massachusetts General Hospital; Doris Duke Charitable Foundation and the Swiss National 457
Science Foundation Ambizione program (PZOOP2_168103/1). The content of this paper is solely the responsibility of the 458
authors and does not necessarily represent the official views of the National Institutes of Health, the Department of Veterans 459
Affairs or the United States Government. 460
AUTHOR CONTRIBUTIONS 461
TM designed the experiment with input from AAS, DB, JDS, JPD and LRH. TM and AAS performed pilot analyses to develop 462
the historical decoder approach. TM conceived and implemented the decoder calibration method, with input from AAS. AAS 463
implemented the adaptive feature normalization method, with input from TM. DB designed and implemented the 464
communication interface. TM and AAS performed the data analyses and drafted the manuscript, which was further edited by all 465
authors. SSC is clinical co-investigator of the pilot clinical trial and assisted in the clinical oversight of the participants. EE and 466
JH planned and executed the electrode array implants and supported the clinical research components of the study. AAS, JDS, 467
JS and DB made key contributions to the BrainGate software and hardware infrastructure supporting these research sessions. 468
EG, KT, BLS, CB, CP, KVS, and JMH contributed to data collection. JPD and LRH conceived, planned and continue to direct 469
the ongoing BrainGate research. LRH is IDE sponsor-investigator of the BrainGate2 pilot clinical trial. 470
471
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
FIGURE CAPTIONS 472
Fig. 1. LFP-based communication brain-computer interface enables a person with a locked-in syndrome (T2) and a person with 473
tetraplegia and ALS (T6) to write messages over long time periods without decoder recalibration and with minimal technical 474
intervention. Top. The BCI used a historical decoder that was calibrated to data recorded over a period of 23 and 42 days (blue) 475
and then remained unchanged during the use of the BCI for communication for 76 and 138 days afterwards (red) for T2 and T6, 476
respectively. Middle. The participants sat in front of a computer monitor that displayed a BCI-controlled speller FlashSpeller. 477
The participants selected the options by attempting or executing selected movements, which were accompanied by LFP 478
responses decoded as selections. The grey arrow traces the processing of the intracortical neural signals. Raw signals from each 479
of 96 electrodes (first box) were processed into low-pass filtered, intermediate frequency and high frequency component 480
features (second box). Out of these 288 features, we selected 50 (third box). These features were analyzed by the decoder that 481
outputted a “click” probability (fourth box). If the probability crossed the threshold, the option was selected (fifth box). If no 482
selections were detected during the option presentation time (1.5s-2s), the FlashSpeller presented the next option in the cue. 483
Bottom. The FlashSpeller is a two tier speller. The first tier is composed of groups of characters, space and backspace options. 484
Once a group of characters is selected, FlashSpeller enters into the second tier composed of individual characters. Selecting a 485
character adds it to the entered text and restarts the first tier cycle. If all second tier options have been passed, the FlashSpeller 486
continues the first tier cycle. If all first tier options have been passed, the first tier cycle restarts from the backspace option. 487
Additional “word completion” option was added during free-spelling, which allowed completion of a partially spelled word 488
(middle; purple panel). 489
Fig. 2. Schemes of the sessions used in our study. Top left panel shows the scheme of the data collection sessions. These started 490
with a normalization block and proceeded with two to four open-loop calibration blocks. Recordings from five consecutive data 491
collection sessions were used to calibrate the historical decoder, which remained unchanged throughout all subsequent sessions. 492
Bottom panel shows the scheme for the comparison sessions. These started with a normalization block and three open-loop 493
calibration blocks. Same-day decoder was calibrated to the three open-loop calibration blocks and the historical decoder was 494
loaded from file. The two decoders were then blind randomized and used in the 4 pairs of copy phrase blocks, each decoder 495
once in each pair. The historical decoder was used for typing in the free-spelling blocks that followed. Top right panel shows the 496
scheme of the Communication session in which a normalization block was followed by free-spelling blocks in which the 497
historical decoder was used for typing. Comparison and communication sessions ended when the participant did not express his 498
intention to start a new block. 499
Fig. 3. SNR for five collection sessions for both participants, averaged over all channels, which have been used to determine 500
intermediate and high frequency component (IFC and HFC) frequency bands. SNR has been normalized to the maximum value 501
for each session. The black circles mark the frequency band that gave the highest SNR in the intermediate and high frequency 502
ranges. 503
Fig. 4. Example of selection probabilities and selected options throughout the first 316s of the first free-spelling block, session 504
T6 day 493. During this time, T6 spelled the phrase, “Free speech is one of the most important rights”. Top to bottom. 505
Successive panels show increasing time resolution of a region outlined in red. Three top rows in each panel display the periods 506
during which the FlashSpeller button was: green after selecting an option (Feedback), presenting an option (Option), and empty 507
during transition between options (Transition). Fourth row shows the click probability with a probability threshold. Rows five 508
through seven show three neural features selected for decoding, one for each component (LFC – low-pass filtered component; 509
IFC – intermediate frequency component; HFC – high frequency component). The last row shows all fifty neural features 510
selected for decoding. 511
Fig. 5. Examples of T2’s neuronal responses to attempts to squeeze his right fist recorded on a single electrode (el. 95) during 512
three open-loop calibration blocks for three sessions 48 or more days apart. Panels show average spectrogram (top row of 513
panels) and single trials of low-pass filtered (second row), intermediate frequency (third row) and high frequency (fourth row) 514
component LFP features. Single trials are shown in units of standard deviation (s.d.). For better visualization, the colorbar scale 515
spans the values from 0.5th to 99.5th percentile. 516
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Fig. 6. Long term stability of participants’ LFP responses during click actions. (A) Mean (thick lines) and 95% confidence 517
intervals (thin lines) of single electrode low-pass filtered component (LFC; top panels), intermediate frequency component (IFC; 518
middle panels) and high frequency component (HFC; bottom panels) LFP features calculated over all trials recorded in open-519
loop calibration blocks are shown for three selected sessions at least one month apart (blue, red and black traces in temporal 520
order). Left and right panels show signals for participant T2, electrode 95; and participant T6, electrode 80, respectively. 521
Average spectrograms and single trial LFP responses for these sessions are shown in Figure 5. s.d. – standard deviation; el. – 522
electrode. (B) Black dots show the median of single-feature cross-session LFP response correlation coefficients (rf(s1,s2)) over 523
all channels for all compared session pairs s1 and s2 against the inter-session period. Grey dots show the mean Mr(s1,s2) when 524
the LFP responses are triggered on random events. Grey lines show the linear ordinary least squares fit to Mr(s1,s2) with the 525
slope b shown at the bottom of the panel. 526
Fig. 7. Demonstration of the detection stability based on stability of LFPs. Top panels show the CYX calculated by calibrating a 527
decoder on all open-loop calibration blocks from one session and validating it on all open-loop calibration blocks of another 528
sessions. The same amount of data was used to calibrate and validate each decoder. Bottom panels show the same CYX values as 529
a function of the decoder age ΔT (time between the session used to calibrate the decoder and the session used to validate it). 530
Black lines show the linear ordinary least squares fit to the CYX values. P value of testing for significant correlation between ΔT 531
and CYX is given. 532
Fig. 8. Spelling performance of participants when using LFP-based FlashSpeller communication BCI. Dots on the left show 533
participants’ spelling performance, as measured by correct characters per minute, as a function of decoder age when using same-534
day and historical decoders during copy-phrase blocks (yellow and blue dots, respectively) and when using historical decoder 535
during free spelling with the word completion option (purple dots). Broken lines show the linear ordinary least squares fit over 536
all sessions with P values for testing significant correlation between the spelling performance and the age of the historical 537
decoder shown below the lines. Bars on the right side show the mean spelling performance over all sessions. Error bars show 538
95% confidence intervals. ns – non significant difference (P ≥ 0.05); * - P < 0.05. 539
Fig. 9. Performance of participants’ interaction with the FlashSpeller application as measured by the normalized mutual 540
information (CYX). Left panels show CYX when using same-day and historical decoders during copy-phrase blocks (yellow and 541
blue bars, respectively) and when using historical decoder during free spelling with and without the word completion option 542
(full and empty purple bars). Maximum theoretical performance, described as participant always selecting the intended spelling 543
option or character, is 1. Right panels show the CYX as a function of time (colored dots) and its linear ordinary least squares fit 544
(broken lines). Top right corner displays p values for testing significant correlation between the CYX and time. Performance is 545
shown as mean CYX. Error bars show 95% confidence intervals. Error bars show 95% confidence intervals. ns – non significant 546
difference (P ≥ 0.05); * - P < 0.05. 547
Fig. 10. Participants’ self-assessed quantitative scores of historical and same-day decoders during copy-phrase blocks. Each 548
colored dot shows the participant’s score for a single copy-phrase block. Score was given on a scale from 1 to 10 (best). Bars on 549
the right show the mean score over all scored blocks and error bars show the 95% confidence intervals. ns – non significant 550
difference (P ≥ 0.05). 551
Fig. 11. Hypothetical performance of an average signal decoder compared to the performance of the historical decoder. We 552
simulated the case where the intracortical microelectrode array was replaced with a single electrocorticographic electrode by 553
averaging the recorded signal over all electrodes. (A) As with the historical decoder, we identified IFC and HFC frequency 554
bands for each of the participants from the peri-click SNR in five collection sessions. Note that, to construct the simulated single 555
electrocorticographic electrode signal, we averaged the signals over all low-noise microelectrode array channels prior to 556
computing the SNR. For T6, the HFC frequency band did not emerge in any of the sessions. Therefore, we used only LFC and 557
IFC features to calibrate and use the average signal decoder for T6. (B) Bar plots show decoding accuracy, as measured by CYX, 558
of the historical and average signal decoder for each participant on individual session days and the mean across all sessions 559
(overall). For all participants, the historical decoder outperformed the simulated average signal decoder (p<.05, bootstrap test). 560
The same amount of data was used to calibrate and validate both decoders. 561
562
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
TABLES AND TABLE CAPTIONS 563
564
Sessions
Intermediate frequency SNR local maximum
High frequency SNR local maximum
First bin Last bin First bin Last bin
T2
452 5 (14Hz) 9 (33Hz) 14 (49Hz) 42 (162Hz)
453 5 (14Hz) 9 (33Hz) 16 (57Hz) 60 (232Hz)
460 5 (14Hz) 8 (29Hz) 14 (49Hz) 45 (174Hz)
467 5 (14Hz) 9 (33Hz) 15 (53Hz) 41 (158Hz)
474 6 (18Hz) 8 (29Hz) 14 (49Hz) 61 (236Hz)
Average 5 (14Hz) 9 (33Hz) 15 (53Hz) 50 (193Hz)
T6
313 6 (18Hz) 9 (33Hz) 16 (57Hz) 55 (213Hz)
322 7 (21Hz) 9 (33Hz) 18 (64Hz) 90 (350Hz)
336 7 (21Hz) 14 (53Hz) 18 (64Hz) 81 (314Hz)
348 7 (21Hz) 9 (33Hz) 17 (61Hz) 86 (334Hz)
355 7 (21Hz) 13 (49Hz) 17 (61Hz) 53 (205Hz)
Average 7 (21Hz) 11 (41Hz) 17 (61Hz) 73 (283Hz)
565
Table 1. SNR local maxima for five collection sessions used to determine the intermediate and high frequency component (IFC 566
and HFC) frequency bands (average) for each participant. Numbers in parentheses show the rounded lower edge frequency of 567
the first frequency bin or the rounded top edge frequency of the last frequency bin. 568
569
Session Phrase Time (min) Number of correct
characters
T2 510 Thankyou for your time friday. We had a g ood time. ****. 21.8 56
T2 524 A tale told by an idiot, full of sound and fury, signifying nothing 15.0 67
T2 529 Would it help if i came saturday in smaller chair? 13.3 50
T6 403 Embracing als is an important step towards living with als. 8.5 59
T6 441 I want to thank all my caregivers who made the trip to hawaii possible 11.0 70
T6 493 Free speech is one of the most important rights we have whether you are able or
disabled, and w
11.9 95
Where technolgy is a 3.8 20
Available, there should be a constitutional mandate to make available such techn 11.9 80
Technology to people in need free of charge. 6.2 43
570
Table 2. Examples of sentences typed by participants T2 and T6 using the historical decoder during the free-spelling blocks. In 571
the phrase from session T2 510, four characters that contained personal information have been redacted (*). In session T6 493, 572
the participant typed a sentence over four free-spelling blocks. Typing was interrupted either due to the block maximum length 573
of 12 minutes or by the participant. 574
575
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Able-bodied participants Participants with ALS without LIS Participants with LIS
Speller type Switch P300 / SSVEP Switch P300 / SSVEP Switch P300 / SSVEP
Scalp EEG 0-0.86* (Bai et al. 2010) 2.6 (McCane et al. 2015);
5.48 (Pires et al. 2012);
5.83 (Chen et al. 2015)
0 (Hill et al. 2006);
1.2* (Miner et al. 1998)
<2.1 (Nijboer et al. 2008);
4.97 (Pires et al. 2012);
1.16 (Mainsah et al. 2015)
0.5 (Birbaumer et al. 1999);
<2 (Kubler et al. 2001)
2.1 (McCane et al. 2014);
<1 (Sellers et al. 2014)
ECoG 0.51 (Hinterberger et al. 2005) 17 (Brunner et al. 2011) 0 (Hill et al. 2006);
1.82 (Vansteensel et al. 2016)
0 (Murguialday et al. 2011)
Intracortical
LFP-based
(this study)
6.88 3.07
Intracortical
spike-based
using other
speller types
10.44-17.40 (Jarosiewicz et al. 2015);
13.5-31.6 (Pandarinath et al. 2017)
9.4 (Bacher et al. 2014);
1.95-11.75 (Jarosiewicz et al. 2015)
Table 3. Comparison of spelling performance between our study (red) and other spelling BCI studies, including all studies listed in three recent reviews (Fazel-576
Rezai et al. 2012; Kubler and Birbaumer 2008; Marchetti and Priftis 2014). Performance of BCI-based spellers in able-bodied people, people with ALS who 577
retain the ability to speak, and people with LIS resulting from ALS or stroke, as measured in correct characters per minute. When only selections per minute were 578
reported, we include this value as an upper bound. When only binary selections were allowed we estimated a typing rate based on a simple switch keyboard 579
design, with five selections required per character (marked with *s). 580
581
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
APPENDIX 582
Adaptive Feature Normalization 583
To account for the drifts of the neuronal features, Φf, we used a recursive algorithm to estimate the their mean, μf , and variance, 584
σf, using a time constant τ of 240 seconds (Bruce 1969). 585
f f f
tt t t t
586
22
f f f f
tt t t t t
587
where Δt was the temporal resolution of our online data processing set to 20ms. Estimated mean of each feature was subtracted 588
from the current feature value and the result was divided by the estimated standard deviation, thereby normalizing all features to 589
zero-mean and unit-variance. To prevent division by zero, a value of 10-6 was added to the variance estimate. 590
6ˆ
10
f f
f
f
t tt
t
591
SNR used for selection of neural features 592
To reduce the time needed for algorithm calibration and to reduce the effect of excessive dimensionality (Hastie et al. 2009), we 593
selected a subset of neural features that were used for decoder calibration. Selection was done by ranking the features according 594
to the SNR between peri-action and baseline feature values (see Milekovic et al. 2012 for similar use of SNR): 595
ˆ
ˆ
ˆ
ˆ
SQUEEZE SQUEEZE
SQUEEZE SQUEEZE
BASELINE BASELINE
BASELINE BASELINE
i
i
t E t tc
t std t tc
E t
std t
596
SQUEEZE BASELINE
SQUEEZE BASELINE
t tSNR t
t t
597
.
max max max arg ,SQUEEZE
VSBASELINE
S Et
SNR SNR t t SNR t t t t 598
Where tci is the time of the trial i, E is the expectation operator and std is the standard deviation operator, both acting over trials, 599
tS (300ms) and tE (1.8s for T2 and 1.5s for T6) are the start and the end of the investigated epoch. 600
Calibration of decoding algorithms 601
The rLDA classifier was calibrated in a following way. First, we calibrated a set of classifiers that captured the neuronal features 602
specific to the participant’s selection actions, recorded in the selected set of open-loop calibration blocks. In these blocks, the 603
participant was cued to perform selection click action at times tci (see Task section for details). Given a set of normalized 604
features (Φf1, …, Φfm), the peri-selection feature vector Fi was defined as: 605
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
1 1
1
2 1
2
ˆ
ˆ
ˆ
ˆ
ˆ
f i
f i n
f i
i
f i n
fm i n
tc t
tc t
tc tF
tc t
tc t
M
M
M
606
where t1, ..., tn are the selected time points relative to the time of the cue. Therefore, the feature vector contained n∙m features. 607
The time points t1, ..., tn were always equidistant and defined by a set of parameters: (i) the time of the first feature in relation to 608
the time of the cue, t1, (ii) the number of time points, n and (iii) the temporal distance between the first and the last feature, tn-t1. 609
Each classifier was built using two classes of feature vectors: the “select” class, which contained peri-select feature vectors 610
(Sclass), and the baseline class, which contained feature vectors between selections (Bclass). 611
1 1
1
2 1
2
, 1,...,
ˆ
ˆ
ˆ,
, | 0.5
ˆ
ˆ
class i
f
f n
f
class
i
f n
fm n
S F i N
k t t
k t t
kk t tB
i k tc k t s
k t t
k t t
M
¥
M
M
612
where δt is the time resolution of baseline feature vectors, set to 80ms; and N is the number of selection cues present in the data 613
used to train the classifier. 614
These classes were used to build a regularized linear discriminant analysis (rLDA) classifier (Friedman 1989). Regularization 615
was implemented in a same way as in Milekovic et al. (Milekovic et al. 2013). The probability of every feature vector F(t) in the 616
test dataset to belong to the “select” class was then calculated using the classifier. 617
1 1 1 2 1ˆ ˆ ˆ ˆ,..., , ,...,f f n f fm nF t t t t t t t t t 618
Due to the internal latency of the decoder, feature vector F(t) could only lead to a selection detection at time t+tn. Based on the 619
probability of the selection, refractory period and the detection hysteresis, we assigned “detected selection” (ds) labels to all 620
times in which the selection was detected and assigned “detected as non-selection” (dns) labels to all remaining time points. 621
To validate each of the classifiers, we divided the set of blocks used to calibrate the classifiers into two equally long subsets, one 622
containing the first halves and another containing the second halves of each block. First, we chose a set of parameter values 623
consisting of: (i) a time of the first feature in relation to the selection cue, t1, (ii) a number of features, n and (iii) a time distance 624
between the first and the last feature, tn-t1, and (iv) regularization coefficient, γ. An rLDA classifier was then calibrated to the 625
first part of the dataset and used to detect the events on the second part of the dataset and vice-versa. Detection performance was 626
estimated by calculating the normalized mutual information CYX (see section “Performance measures” for details) for both 627
combinations and then averaging between the two. Values of the parameters were then changed and the process was repeated, 628
until all parameter values from the parameter grid were tested. The parameter values that were used to calibrate the decoder that 629
gave the maximum CYX were then used to calibrate a new decoder to the whole dataset. This decoder was then used to detect 630
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
selections during the copy-phrase and free-spelling blocks of the comparison and communication sessions (online) or to detect 631
selections in the open-loop calibration blocks of all completed sessions in a posterior analysis (offline). 632
Performing the selection of parameter values that give the highest performance is a computationally demanding process. The 633
historical decoder used during the participants’ comparison and communication sessions and all decoders used for offline 634
analysis were calibrated using a parameter grid with numerous values. This was possible since there were no restrictions on the 635
amount of time needed for calibration. However, same-day decoder used during comparison sessions had to be calibrated during 636
the session itself, within several minutes. Therefore, instead of selecting the parameter values from a grid of values, online 637
same-day decoders were calibrated using the parameter values selected through the calibration of the historical decoder. 638
Measuring cross-session stability of LFP responses 639
rf(s1,s2) was calculated as follows: 640
1 2
1 2
1 11 2
1 1ˆ ˆ1, 2 , ,s sN N
s s
f f f
n n
r s s CC t tN N
641
where CC is the correlation coefficient operator calculated over time t running from 0 to 1.5 seconds after the cue, and Ns1, Ns2, 642
and are the numbers of responses and the features for sessions s1 and s2, respectively. rf(s1,s2) was calculated for 643
all 288 possible features. We used only the recordings from the open-loop calibration blocks for the calculation. Since the 644
calibration was standardized for each participant, the same amount of data from each session - three equally long OLC blocks 645
containing 17 cues each – was used to calculate CC. 646
To estimate the Mr(s1,s2) values for no evoked LFP responses, we randomly drew as many “random” cues from a given block 647
as there were active cues. The draws were made from a uniform distribution. These random cues were then used to calculate 648
Mr(s1,s2). This procedure was repeated 1000 times to calculate the mean Mr(s1,s2) for random cues, termed RMr(s1,s2). 649
Measuring spelling and decoding performance 650
Accuracy of a decoding algorithm is often described by its positive and negative predictive values (Fletcher et al. 2014). 651
However, using these measures to obtain a single metric that can be used to optimize a decoder requires establishing an arbitrary 652
tradeoff between these two values. Another way to measure decoding performance, which takes both positive and negative 653
predictive values into account, is to use a metric based on information theory. We used the normalized mutual information CYX 654
(Coombs et al. 1970), which was calculated using the assumed intended actions of the participant, “real selection” (rs) and “real 655
non-selection” (rns), and the actions detected by BrainGate system, “detected selection” (ds) and “detected non-selection” (dns). 656
We first divided the corresponding blocks into epochs of the same length as the presentation time that we labeled as “rs” and 657
“rns”. During the open-loop calibration blocks, we assumed that the participant always responded to the cues. Therefore, every 658
epoch after the selection cue was labeled as “rs” and all other epochs were labeled as “rns”. According to the selection 659
detections, we labeled each epoch with one of the four labels: true positive – (rs,ds); false positive – (rns,ds); false negative - 660
(rs,dns); and true negative – (rns,dns). We calculated CYX by computing the mutual information between detections and real 661
events, divided by the entropy of real events, as described previously (Milekovic et al. 2013). 662
The spelling performance of our text-entry interface was calculated by counting the total number of correctly typed characters 663
(NCC) and dividing it by the time that participant spent from the start of the spelling until s/he finished the phrase (value given in 664
correct characters per minute, CCPM). To calculate the confidence intervals of the measure, we split the spelling period into 665
epochs that the participant spent to type every single character, τi(char). Occasionally, even though the participants could erase 666
the incorrectly typed character, they would continue spelling. For the purpose of calculating spelling performance, this character 667
was not counted as having been typed, and the time taken to type it was added to the time it took to type the next character. Due 668
to the design of the text-entry interface, there is a minimum time necessary to reach each of the characters, τMIN(char). Note that 669
τMIN(char) can be different for different characters. Once the character is reached, the time it takes for the participant to type it, 670
κi, is independent of the character, even if the character has been missed. The spelling performance (SP) can then be written as: 671
1 ˆs
f 2 ˆs
f
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
1 1 1
CC CC CC
CC CC CC
N N N
i MIN i MIN i
i i i
N N NSP
char char phrase
672
We estimated the standard deviation of the spelling performance calculated from a given dataset by bootstrapping with 100,000 673
re-samples (Moore et al. 2009). We assumed the normal distribution of the bootstrapped spelling performance values and used 674
the t-distribution to calculate the 95% confidence intervals (Bohm and Zech 2010). During the free-typing blocks, by instantly 675
completing the word through word completion, the participant was able to select several characters at once. To calculate the 676
confidence intervals in these blocks, we bootstrapped the pairs, consisting of the number of the characters spelled in this 677
particular step nCCi and the corresponding κi. When comparing the performance of two different detection algorithms, we again 678
used bootstrapping with 100,000 re-samples on a SPA - SPB statistic and used the t-distribution to calculate the 95% confidence 679
intervals. 680
681
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
REFERENCES 682
Aflalo T, Kellis S, Klaes C, Lee B, Shi Y, Pejsa K, Shanfield K, Hayes-Jackson S, Aisen M, Heck C, Liu C, and Andersen 683
RA. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348: 906-910, 684
2015. 685
Ajiboye AB, Simeral JD, Donoghue JP, Hochberg LR, and Kirsch RF. Prediction of imagined single-joint movements in a 686
person with high-level tetraplegia. IEEE Trans Biomed Eng 59: 2755-2765, 2012. 687
Ajiboye AB, Willett FR, Young DR, Memberg WD, Murphy BA, Miller JP, Walter BL, Sweet JA, Hoyen HA, Keith 688 MW, Peckham PH, Simeral JD, Donoghue JP, Hochberg LR, and Kirsch RF. Restoration of reaching and grasping 689
movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. The 690
Lancet 2017. 691
Arya R, Mangano FT, Horn PS, Holland KD, Rose DF, and Glauser TA. Adverse events related to extraoperative invasive 692
EEG monitoring with subdural grid electrodes: a systematic review and meta-analysis. Epilepsia 54: 828-839, 2013. 693
Bach JR. Amyotrophic lateral sclerosis. Communication status and survival with ventilatory support. American journal of 694
physical medicine & rehabilitation / Association of Academic Physiatrists 72: 343-349, 1993. 695
Bacher D, Jarosiewicz B, Masse NY, Stavisky SD, Simeral JD, Newell K, Oakley EM, Cash SS, Friehs G, and Hochberg 696 LR. Neural Point-and-Click Communication by a Person With Incomplete Locked-In Syndrome. Neurorehabil Neural Repair 697
2014. 698
Bai O, Lin P, Huang D, Fei DY, and Floeter MK. Towards a user-friendly brain-computer interface: initial tests in ALS and 699
PLS patients. Clin Neurophysiol 121: 1293-1303, 2010. 700
Bansal AK, Truccolo W, Vargas-Irwin CE, and Donoghue JP. Decoding 3D reach and grasp from hybrid signals in motor 701
and premotor cortices: spikes, multiunit activity, and local field potentials. J Neurophysiol 107: 1337-1355, 2012. 702
Bansal AK, Vargas-Irwin CE, Truccolo W, and Donoghue JP. Relationships among low-frequency local field potentials, 703
spiking activity, and three-dimensional reach and grasp kinematics in primary motor and ventral premotor cortices. J 704
Neurophysiol 105: 1603-1619, 2011. 705
Barrese JC, Rao N, Paroo K, Triebwasser C, Vargas-Irwin C, Franquemont L, and Donoghue JP. Failure mode analysis 706
of silicon-based intracortical microelectrode arrays in non-human primates. J Neural Eng 10: 066014, 2013. 707
Bauer G, Gerstenbrand F, and Rumpl E. Varieties of the locked-in syndrome. Journal of neurology 221: 77-91, 1979. 708
Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kubler A, Perelmouter J, Taub E, and Flor H. A 709
spelling device for the paralysed. Nature 398: 297-298, 1999. 710
Bjornsson CS, Oh SJ, Al-Kofahi YA, Lim YJ, Smith KL, Turner JN, De S, Roysam B, Shain W, and Kim SJ. Effects of 711
insertion conditions on tissue strain and vascular damage during neuroprosthetic device insertion. J Neural Eng 3: 196-207, 712
2006. 713
Bohm G, and Zech G. Introduction to statistics and data analysis for physicists. Verlag Deutsches Elektronen-Synchrotron, 714
2010. 715
Borton DA, Yin M, Aceros J, and Nurmikko A. An implantable wireless neural interface for recording cortical circuit 716
dynamics in moving primates. J Neural Eng 10: 026010, 2013. 717
Bouton CE, Shaikhouni A, Annetta NV, Bockbrader MA, Friedenberg DA, Nielson DM, Sharma G, Sederberg PB, 718
Glenn BC, Mysiw WJ, Morgan AG, Deogaonkar M, and Rezai AR. Restoring cortical control of functional movement in a 719
human with quadriplegia. Nature 533: 247-250, 2016. 720
Bruce MM. Estimation of variance by a recursive equation National Aeronautics and Space Administration, 1969. 721
Brunner P, Joshi S, Briskin S, Wolpaw JR, Bischof H, and Schalk G. Does the 'P300' speller depend on eye gaze? J Neural 722
Eng 7: 056013, 2010. 723
Brunner P, Ritaccio AL, Emrich JF, Bischof H, and Schalk G. Rapid Communication with a "P300" Matrix Speller Using 724
Electrocorticographic Signals (ECoG). Front Neurosci 5: 5, 2011. 725
Buzsaki G, Anastassiou CA, and Koch C. The origin of extracellular fields and currents--EEG, ECoG, LFP and spikes. Nat 726
Rev Neurosci 13: 407-420, 2012. 727
Caplan LR. "Top of the basilar" syndrome. Neurology 30: 72-79, 1980. 728
Capogrosso M, Milekovic T, Borton D, Wagner F, Moraud EM, Mignardot J-B, Buse N, Gandar J, Barraud Q, Xing D, 729
Rey E, Duis S, Jianzhong Y, Ko WKD, Li Q, Detemple P, Denison T, Micera S, Bezard E, Bloch J, and Courtine G. A 730
brain–spine interface alleviating gait deficits after spinal cord injury in primates. Nature 539: 284-288, 2016. 731
Chaudhary U, Birbaumer N, and Ramos-Murguialday A. Brain-computer interfaces for communication and rehabilitation. 732
Nature Reviews Neurology 12: 513-525, 2016. 733
Chaudhary U, Xia B, Silvoni S, Cohen LG, and Birbaumer N. Brain-Computer Interface-Based Communication in the 734
Completely Locked-In State. Plos Biology 15: 2017. 735
Chen X, Wang Y, Nakanishi M, Gao X, Jung TP, and Gao S. High-speed spelling with a noninvasive brain-computer 736
interface. Proc Natl Acad Sci U S A 112: E6058-6067, 2015. 737
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Chestek CA, Batista AP, Santhanam G, Yu BM, Afshar A, Cunningham JP, Gilja V, Ryu SI, Churchland MM, and 738
Shenoy KV. Single-neuron stability during repeated reaching in macaque premotor cortex. J Neurosci 27: 10742-10750, 2007. 739
Chestek CA, Gilja V, Nuyujukian P, Foster JD, Fan JM, Kaufman MT, Churchland MM, Rivera-Alvidrez Z, 740
Cunningham JP, Ryu SI, and Shenoy KV. Long-term stability of neural prosthetic control signals from silicon cortical arrays 741
in rhesus macaque motor cortex. J Neural Eng 8: 045005, 2011. 742
Cipresso P, Carelli L, Solca F, Meazzi D, Meriggi P, Poletti B, Lule D, Ludolph AC, Silani V, and Riva G. The use of 743
P300-based BCIs in amyotrophic lateral sclerosis: from augmentative and alternative communication to cognitive assessment. 744
Brain and behavior 2: 479-498, 2012. 745
Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, McMorland AJ, Velliste M, Boninger 746
ML, and Schwartz AB. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381: 557-564, 747
2013. 748
Coombs CH, Dawes RM, and Tversky A. Mathematical psychology. An elementary introduction. [By] Clyde H. Coombs ... 749
Robin M. Dawes ... Amos Tversky. Englewood Cliffs, New Jersey: Prentice-Hall, 1970. 750
Dickey AS, Suminski A, Amit Y, and Hatsopoulos NG. Single-unit stability using chronically implanted multielectrode 751
arrays. J Neurophysiol 102: 1331-1339, 2009. 752
Doble JE, Haig AJ, Anderson C, and Katz R. Impairment, activity, participation, life satisfaction, and survival in persons 753
with locked-in syndrome for over a decade: follow-up on a previously reported cohort. The Journal of head trauma 754
rehabilitation 18: 435-444, 2003. 755
Fazel-Rezai R, Allison BZ, Guger C, Sellers EW, Kleih SC, and Kubler A. P300 brain computer interface: current 756
challenges and emerging trends. Front Neuroeng 5: 14, 2012. 757
Fletcher RH, Fletcher SW, and Fletcher GS. Clinical epidemiology : the essentials. Philadelphia: Wolters Kluwer/Lippincott 758
Williams & Wilkins Health, 2014, p. 253 p. 759
Flint RD, Ethier C, Oby ER, Miller LE, and Slutzky MW. Local field potentials allow accurate decoding of muscle activity. 760
J Neurophysiol 108: 18-24, 2012a. 761
Flint RD, Lindberg EW, Jordan LR, Miller LE, and Slutzky MW. Accurate decoding of reaching movements from field 762
potentials in the absence of spikes. J Neural Eng 9: 046006, 2012b. 763
Flint RD, Scheid MR, Wright ZA, Solla SA, and Slutzky MW. Long-Term Stability of Motor Cortical Activity: Implications 764
for Brain Machine Interfaces and Optimal Feedback Control. J Neurosci 36: 3623-3632, 2016. 765
Flint RD, Wright ZA, Scheid MR, and Slutzky MW. Long term, stable brain machine interface performance using local field 766
potentials and multiunit spikes. J Neural Eng 10: 056005, 2013. 767
Fraser GW, Chase SM, Whitford A, and Schwartz AB. Control of a brain-computer interface without spike sorting. J Neural 768
Eng 6: 055004, 2009. 769
Friedman JH. Regularized Discriminant-Analysis. J Am Stat Assoc 84: 165-175, 1989. 770
Gallegos-Ayala G, Furdea A, Takano K, Ruf CA, Flor H, and Birbaumer N. Brain communication in a completely locked-771
in patient using bedside near-infrared spectroscopy. Neurology 82: 1930-1932, 2014. 772
Gilja V, Pandarinath C, Blabe CH, Nuyujukian P, Simeral JD, Sarma AA, Sorice BL, Perge JA, Jarosiewicz B, 773 Hochberg LR, Shenoy KV, and Henderson JM. Clinical translation of a high-performance neural prosthesis. Nature medicine 774
2015. 775
Hamer HM, Morris HH, Mascha EJ, Karafa MT, Bingaman WE, Bej MD, Burgess RC, Dinner DS, Foldvary NR, Hahn 776 JF, Kotagal P, Najm I, Wyllie E, and Luders HO. Complications of invasive video-EEG monitoring with subdural grid 777
electrodes. Neurology 58: 97-103, 2002. 778
Hastie T, Tibshirani R, and Friedman JH. The elements of statistical learning : data mining, inference, and prediction. New 779
York: Springer, 2009. 780
Hecht M, Hillemacher T, Grasel E, Tigges S, Winterholler M, Heuss D, Hilz MJ, and Neundorfer B. Subjective 781
experience and coping in ALS. Amyotrophic lateral sclerosis and other motor neuron disorders : official publication of the 782
World Federation of Neurology, Research Group on Motor Neuron Diseases 3: 225-231, 2002. 783
Hedegard E, Bjellvi J, Edelvik A, Rydenhag B, Flink R, and Malmgren K. Complications to invasive epilepsy surgery 784
workup with subdural and depth electrodes: a prospective population-based observational study. J Neurol Neurosurg Psychiatry 785
85: 716-720, 2014. 786
Heldman DA, Wang W, Chan SS, and Moran DW. Local field potential spectral tuning in motor cortex during reaching. 787
IEEE Trans Neural Syst Rehabil Eng 14: 180-183, 2006. 788
Hill NJ, Lal TN, Schroder M, Hinterberger T, Wilhelm B, Nijboer F, Mochty U, Widman G, Elger C, Scholkopf B, 789
Kubler A, and Birbaumer N. Classifying EEG and ECoG signals without subject training for fast BCI implementation: 790
comparison of nonparalyzed and completely paralyzed subjects. IEEE Trans Neural Syst Rehabil Eng 14: 183-186, 2006. 791
Hinterberger T, Birbaumer N, and Flor H. Assessment of cognitive function and communication ability in a completely 792
locked-in patient. Neurology 64: 1307-1308, 2005. 793
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P, 794
and Donoghue JP. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485: 372-375, 795
2012. 796
Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, and 797
Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442: 164-171, 2006. 798
Jarosiewicz B, Sarma AA, Bacher D, Masse NY, Simeral JD, Sorice B, Oakley EM, Blabe C, Pandarinath C, Gilja V, 799
Cash SS, Eskandar EN, Friehs G, Henderson JM, Shenoy KV, Donoghue JP, and Hochberg LR. Virtual typing by people 800
with tetraplegia using a self-calibrating intracortical brain-computer interface. Science translational medicine 7: 313ra179, 2015. 801
Kageyama Y, Hirata M, Yanagisawa T, Shimokawa T, Sawada J, Morris S, Mizushima N, Kishima H, Sakura O, and 802
Yoshimine T. Severely affected ALS patients have broad and high expectations for brain-machine interfaces. Amyotrophic 803
Lateral Sclerosis and Frontotemporal Degeneration 15: 513-519, 2014. 804
Kennedy PR, Bakay RA, Moore MM, Adams K, and Goldwaithe J. Direct control of a computer from the human central 805
nervous system. IEEE Trans Rehabil Eng 8: 198-202, 2000. 806
Kubler A, and Birbaumer N. Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking 807
in completely paralysed patients? Clin Neurophysiol 119: 2658-2666, 2008. 808
Kubler A, Kotchoubey B, Hinterberger T, Ghanayim N, Perelmouter J, Schauer M, Fritsch C, Taub E, and Birbaumer 809 N. The thought translation device: a neurophysiological approach to communication in total motor paralysis. Exp Brain Res 124: 810
223-232, 1999. 811
Kubler A, Kotchoubey B, Kaiser J, Wolpaw JR, and Birbaumer N. Brain-computer communication: unlocking the locked 812
in. Psychol Bull 127: 358-375, 2001. 813
Kubler A, Nijboer F, Mellinger J, Vaughan TM, Pawelzik H, Schalk G, McFarland DJ, Birbaumer N, and Wolpaw JR. 814
Patients with ALS can use sensorimotor rhythms to operate a brain-computer interface. Neurology 64: 1775-1777, 2005. 815
Laureys S, Pellas F, Van Eeckhout P, Ghorbel S, Schnakers C, Perrin F, Berre J, Faymonville ME, Pantke KH, Damas 816
F, Lamy M, Moonen G, and Goldman S. The locked-in syndrome : what is it like to be conscious but paralyzed and 817
voiceless? Prog Brain Res 150: 495-511, 2005. 818
Mainsah BO, Collins LM, Colwell KA, Sellers EW, Ryan DB, Caves K, and Throckmorton CS. Increasing BCI 819
communication rates with dynamic stopping towards more practical use: an ALS study. J Neural Eng 12: 016013, 2015. 820
Marchetti M, and Priftis K. Brain-computer interfaces in amyotrophic lateral sclerosis: A metanalysis. Clin Neurophysiol 821
2014. 822
McCane LM, Heckman SM, McFarland DJ, Townsend G, Mak JN, Sellers EW, Zeitlin D, Tenteromano LM, Wolpaw 823
JR, and Vaughan TM. P300-based brain-computer interface (BCI) event-related potentials (ERPs): People with amyotrophic 824
lateral sclerosis (ALS) vs. age-matched controls. Clin Neurophysiol 126: 2124-2131, 2015. 825
McCane LM, Sellers EW, McFarland DJ, Mak JN, Carmack CS, Zeitlin D, Wolpaw JR, and Vaughan TM. Brain-826
computer interface (BCI) evaluation in people with amyotrophic lateral sclerosis. Amyotroph Lateral Scler Frontotemporal 827
Degener 15: 207-215, 2014. 828
Mehring C, Rickert J, Vaadia E, Cardosa de Oliveira S, Aertsen A, and Rotter S. Inference of hand movements from local 829
field potentials in monkey motor cortex. Nat Neurosci 6: 1253-1254, 2003. 830
Milekovic T, Ball T, Schulze-Bonhage A, Aertsen A, and Mehring C. Detection of Error Related Neuronal Responses 831
Recorded by Electrocorticography in Humans during Continuous Movements. PLoS One 8: e55235, 2013. 832
Milekovic T, Ball T, Schulze-Bonhage A, Aertsen A, and Mehring C. Error-related electrocorticographic activity in humans 833
during continuous movements. J Neural Eng 9: 026007, 2012. 834
Miner LA, McFarland DJ, and Wolpaw JR. Answering questions with an electroencephalogram-based brain-computer 835
interface. Arch Phys Med Rehabil 79: 1029-1033, 1998. 836
Moore DS, McCabe GP, and Craig B. Introduction to the practice of statistics. Basingstoke: W.H.Freeman, 2009. 837
Moran D. Evolution of brain-computer interface: action potentials, local field potentials and electrocorticograms. Curr Opin 838
Neurobiol 20: 741-745, 2010. 839
Murguialday AR, Hill J, Bensch M, Martens S, Halder S, Nijboer F, Schoelkopf B, Birbaumer N, and Gharabaghi A. 840
Transition from the locked in to the completely locked-in state: a physiological analysis. Clin Neurophysiol 122: 925-933, 2011. 841
Nijboer F, Sellers EW, Mellinger J, Jordan MA, Matuz T, Furdea A, Halder S, Mochty U, Krusienski DJ, Vaughan TM, 842
Wolpaw JR, Birbaumer N, and Kubler A. A P300-based brain-computer interface for people with amyotrophic lateral 843
sclerosis. Clin Neurophysiol 119: 1909-1916, 2008. 844
Nuyujukian P, Kao JC, Fan JM, Stavisky SD, Ryu SI, and Shenoy KV. Performance sustaining intracortical neural 845
prostheses. J Neural Eng 11: 066003, 2014. 846
Pandarinath C, Nuyujukian P, Blabe CH, Sorice BL, Saab J, Willett FR, Hochberg LR, Shenoy KV, and Henderson JM. 847
High performance communication by people with paralysis using an intracortical brain-computer interface. Elife 6: 2017. 848
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Perge JA, Homer ML, Malik WQ, Cash S, Eskandar E, Friehs G, Donoghue JP, and Hochberg LR. Intra-day signal 849
instabilities affect decoding performance in an intracortical neural interface system. J Neural Eng 10: 036004, 2013. 850
Perge JA, Zhang S, Malik WQ, Homer ML, Cash S, Friehs G, Eskandar EN, Donoghue JP, and Hochberg LR. 851
Reliability of directional information in unsorted spikes and local field potentials recorded in human motor cortex. J Neural Eng 852
11: 046007, 2014. 853
Pinkhardt EH, Jurgens R, Becker W, Molle M, Born J, Ludolph AC, and Schreiber H. Signs of impaired selective 854
attention in patients with amyotrophic lateral sclerosis. Journal of neurology 255: 532-538, 2008. 855
Pires G, Nunes U, and Castelo-Branco M. Comparison of a row-column speller vs. a novel lateral single-character speller: 856
assessment of BCI for severe motor disabled patients. Clin Neurophysiol 123: 1168-1181, 2012. 857
Plum F, and Posner JB. The diagnosis of stupor and coma. Contemp Neurol Ser 10: 1-286, 1972. 858
Rickert J, Oliveira SC, Vaadia E, Aertsen A, Rotter S, and Mehring C. Encoding of movement direction in different 859
frequency ranges of motor cortical local field potentials. J Neurosci 25: 8815-8824, 2005. 860
Schreiber H, Gaigalat T, Wiedemuth-Catrinescu U, Graf M, Uttner I, Muche R, and Ludolph AC. Cognitive function in 861
bulbar- and spinal-onset amyotrophic lateral sclerosis. A longitudinal study in 52 patients. Journal of neurology 252: 772-781, 862
2005. 863
Sellers EW, Ryan DB, and Hauser CK. Noninvasive brain-computer interface enables communication after brainstem stroke. 864
Science translational medicine 6: 257re257, 2014. 865
Shain W, Spataro L, Dilgen J, Haverstick K, Retterer S, Isaacson M, Saltzman M, and Turner JN. Controlling cellular 866
reactive responses around neural prosthetic devices using peripheral and local intervention strategies. IEEE Trans Neural Syst 867
Rehabil Eng 11: 186-188, 2003. 868
Simeral JD, Kim SP, Black MJ, Donoghue JP, and Hochberg LR. Neural control of cursor trajectory and click by a human 869
with tetraplegia 1000 days after implant of an intracortical microelectrode array. J Neural Eng 8: 025027, 2011. 870
Stark E, and Abeles M. Predicting movement from multiunit activity. J Neurosci 27: 8387-8394, 2007. 871
Vansteensel MJ, Pels EGM, Bleichner MG, Branco MP, Denison T, Freudenburg ZV, Gosselaar P, Leinders S, Ottens 872
TH, Van Den Boom MA, Van Rijen PC, Aarnoutse EJ, and Ramsey NF. Fully Implanted Brain–Computer Interface in a 873
Locked-In Patient with ALS. New Engl J Med 0: null, 2016. 874
Vieregge P, Wauschkuhn B, Heberlein I, Hagenah J, and Verleger R. Selective attention is impaired in amyotrophic lateral 875
sclerosis--a study of event-related EEG potentials. Brain research Cognitive brain research 8: 27-35, 1999. 876
Wang W, Collinger JL, Degenhart AD, Tyler-Kabara EC, Schwartz AB, Moran DW, Weber DJ, Wodlinger B, 877
Vinjamuri RK, Ashmore RC, Kelly JW, and Boninger ML. An Electrocorticographic Brain Interface in an Individual with 878
Tetraplegia. PLoS One 8: e55344, 2013. 879
Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs SJ, Srinivasan MA, and 880
Nicolelis MA. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408: 361-365, 2000. 881
Wodlinger B, Downey JE, Tyler-Kabara EC, Schwartz AB, Boninger ML, and Collinger JL. Ten-dimensional 882
anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J Neural Eng 12: 883
016011, 2015. 884
Wolpaw JR, and Wolpaw EW. Brain-computer interfaces : principles and practice. Oxford; New York: Oxford University 885
Press, 2012. 886
Yin M, Borton DA, Komar J, Agha N, Lu Y, Li H, Laurens J, Lang Y, Li Q, Bull C, Larson L, Rosler D, Bezard E, 887 Courtine G, and Nurmikko AV. Wireless Neurosensor for Full-Spectrum Electrophysiology Recordings during Free Behavior. 888
Neuron 84: 1170-1182, 2014. 889
Young JM, and McNicoll P. Against all odds: Positive life experiences of people with advanced amyotrophic lateral sclerosis. 890
Health Soc Work 23: 35-43, 1998. 891
Yousry TA, Schmid UD, Alkadhi H, Schmidt D, Peraud A, Buettner A, and Winkler P. Localization of the motor hand 892
area to a knob on the precentral gyrus. A new landmark. Brain 120 ( Pt 1): 141-157, 1997. 893
894
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
SUPPORTING INFORMATION 895
Movie 1. Session T6 473, 3rd copy-phrase block using historical decoder. T6 used the LFP-based BCI with a historical decoder 896
to control the FlashSpeller in order to type “Pack my box with”. Upon selection, the button with options turned green. Left 897
panel. Video showing the participant monitor during the block as seen by the participant. Right panels. Neural features recorded 898
during the block and the probability of selection for the historical decoder synchronized with the video shown on the left panel. 899
Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the epoch marked by the white line. Top 900
right panel. All 50 features selected for decoder calibration are shown as the color image. Bright colors shows higher amplitude 901
and darker colors show lower amplitude. Three middle right panels. One of each low-filtered component (LFC), intermediate 902
frequency component (IFC) and high-frequency component (HFC) chosen out of 50 features selected for decoder calibration. 903
Bottom right panel. Probability of selection for the historical decoder. Cyan line marks the threshold set at 0.95. 904
Movie 2. Session T2 523, 1st copy-phrase block using historical decoder. T2 used the LFP-based BCI with a historical decoder 905
to control the FlashSpeller in order to type “The quick brown fox”. Upon selection, the button with options turned green. Left 906
panel. Video showing the participant monitor during the block as seen by the participant. Right panels. Neural features recorded 907
during the block and the probability of selection for the historical decoder synchronized with the video shown on the left panel. 908
Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the epoch marked by the white line. Top 909
right panel. All 50 features selected for decoder calibration are shown as the color image. Bright colors shows higher amplitude 910
and darker colors show lower amplitude. Three middle right panels. One of each low-filtered component (LFC), intermediate 911
frequency component (IFC) and high-frequency component (HFC) chosen out of 50 features selected for decoder calibration. 912
Bottom right panel. Probability of selection for the historical decoder. Cyan line marks the threshold set at 0.95. 913
Movie 3. Session T6 473, 3rd copy-phrase block using same-day decoder. T6 used the LFP-based BCI with a same-day decoder 914
to control the FlashSpeller in order to type “Pack my box with”. Upon selection, the button with options turned green. Left 915
panel. Video showing the participant monitor during the block as seen by the participant. Right panels. Neural features recorded 916
during the block and the probability of selection for the same-day decoder synchronized with the video shown on the left panel. 917
Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the epoch marked by the white line. Top 918
right panel. All 50 features selected for decoder calibration are shown as the color image. Bright colors shows higher amplitude 919
and darker colors show lower amplitude. Three middle right panels. One of each low-filtered component (LFC), intermediate 920
frequency component (IFC) and high-frequency component (HFC) chosen out of 50 features selected for decoder calibration. 921
Bottom right panel. Probability of selection for the same-day decoder. Cyan line marks the threshold set at 0.95. 922
Movie 4. Session T2 523, 1st copy-phrase block using same-day decoder. T2 used the LFP-based BCI with a same-day decoder 923
to control the FlashSpeller in order to type “The quick brown fox”. Upon selection, the button with options turned green. Left 924
panel. Video showing the participant monitor during the block as seen by the participant. Right panels. Neural features recorded 925
during the block and the probability of selection for the same-day decoder synchronized with the video shown on the left panel. 926
Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the epoch marked by the white line. Top 927
right panel. All 50 features selected for decoder calibration are shown as the color image. Bright colors shows higher amplitude 928
and darker colors show lower amplitude. Three middle right panels. One of each low-filtered component (LFC), intermediate 929
frequency component (IFC) and high-frequency component (HFC) chosen out of 50 features selected for decoder calibration. 930
Bottom right panel. Probability of selection for the same-day decoder. Cyan line marks the threshold set at 0.95. 931
Movie 5. Session T2 524, 1st free-spelling block without word completion. T2 used the LFP-based BCI with a historical 932
decoder to control the FlashSpeller in order to type “A story told by an idiot” (Shakespeare paraphrase, Macbeth, Act V, Scene 933
V). Upon selection, the button with options turned green. Left panel. Video showing the participant monitor during the block as 934
seen by the participant. Right panels. Neural features recorded during the block and the probability of selection for the historical 935
decoder synchronized with the video shown on the left panel. Each panel shows a 3s epoch with the frame of the video 936
synchronized to the middle of the epoch marked by the white line. Top right panel. All 50 features selected for decoder 937
calibration are shown as the color image. Bright colors shows higher amplitude and darker colors show lower amplitude. Three 938
middle right panels. One of each low-filtered component (LFC), intermediate frequency component (IFC) and high-frequency 939
component (HFC) chosen out of 50 features selected for decoder calibration. Bottom right panel. Probability of selection for the 940
historical decoder. Cyan line marks the threshold set at 0.95. 941
Movie 6. Session T2 523, 1st free-spelling block with word completion. T2 used the LFP-based BCI with a historical decoder to 942
control the FlashSpeller in order to type and send an email. The beginning of the email spelled “Tell team thanks for fixing”. 943
Upon selection, the button with options turned green. Left panel. Video showing the participant monitor during the block as seen 944
by the participant. Right panels. Neural features recorded during the block and the probability of selection for the historical 945
decoder synchronized with the video shown on the left panel. Each panel shows a 3s epoch with the frame of the video 946
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
synchronized to the middle of the epoch marked by the white line. Top right panel. All 50 features selected for decoder 947
calibration are shown as the color image. Bright colors shows higher amplitude and darker colors show lower amplitude. Three 948
middle right panels. One of each low-filtered component (LFC), intermediate frequency component (IFC) and high-frequency 949
component (HFC) chosen out of 50 features selected for decoder calibration. Bottom right panel. Probability of selection for the 950
historical decoder. Cyan line marks the threshold set at 0.95. 951
Movie 7. Session T6 493, 1st free-spelling block with word completion. T6 used the LFP-based BCI with a historical decoder to 952
control the FlashSpeller in order to type “Free speech is one of the most important rights”. Upon selection, the button with 953
options turned green. Left panel. Video showing the participant monitor during the block as seen by the participant. Right 954
panels. Neural features recorded during the block and the probability of selection for the historical decoder synchronized with 955
the video shown on the left panel. Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the 956
epoch marked by the white line. Top right panel. All 50 features selected for decoder calibration are shown as the color image. 957
Bright colors shows higher amplitude and darker colors show lower amplitude. Three middle right panels. One of each low-958
filtered component (LFC), intermediate frequency component (IFC) and high-frequency component (HFC) chosen out of 50 959
features selected for decoder calibration. Bottom right panel. Probability of selection for the historical decoder. Cyan line marks 960
the threshold set at 0.95. 961
Movie 8. Session T6 493 normalization block. Each session started with a normalization block during which the participants 962
looked at the monitor while the FlashSpeller automatically spelled the phrase “The brain”. Data from the normalization block 963
were used to calculate the STFT normalization matrix and to initialize the parameters for the automatic feature normalization. 964
Left panel. Video showing the participant monitor during the normalization block as seen by the participant. Right panels. 965
Neural features recorded during the block and the probability of selection for the historical decoder synchronized with the video 966
shown on the left panel. Each panel shows a 3s epoch with the frame of the video synchronized to the middle of the epoch 967
marked by the white line. Top right panel. All 50 features selected for decoder calibration are shown as the color image. Bright 968
colors shows higher amplitude and darker colors show lower amplitude. Three middle right panels. One of each low-filtered 969
component (LFC), intermediate frequency component (IFC) and high-frequency component (HFC) chosen out of 50 features 970
selected for decoder calibration. Bottom right panel. Probability of selection for the historical decoder. Cyan line marks the 971
threshold set at 0.95. 972
Movie 9. Session T6 473, 2nd open-loop calibration block. In the open-loop calibration blocks, the FlashSpeller automatically 973
spelled the phrase “The brain”. The participants observed the spelling and were asked to perform a click action when the correct 974
options appeared on the screen. The correct options were presented with a highlighted blue background of the button. Data from 975
the open-loop calibration blocks was used to calibrate the historical decoder (collection sessions) or to calibrate the same-day 976
decoder (comparison sessions). Left panel. Video showing the participant monitor during the open-loop calibration block as 977
seen by the participant. Right panels. Neural features recorded during the block and the probability of selection for the historical 978
decoder synchronized with the video shown on the left panel. Each panel shows a 3s epoch with the frame of the video 979
synchronized to the middle of the epoch marked by the white line. Top right panel. All 50 features selected for decoder 980
calibration are shown as the color image. Bright colors shows higher amplitude and darker colors show lower amplitude. Three 981
middle right panels. One of each low-filtered component (LFC), intermediate frequency component (IFC) and high-frequency 982
component (HFC) chosen out of 50 features selected for decoder calibration. Bottom right panel. Probability of selection for the 983
historical decoder. Cyan line marks the threshold set at 0.95. 984
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
TimeDecoder calibrated from data recorded over up to 42 days ...
... then used unchanged for up to 138 days laterHistorical decoder
Feature 1Feature 2
0
1
Prob
abili
ty Historical decoder
50 selected features...
Wideband recordingsfrom 96 electrodes
Volta
ge
...
Time
96 high frequencycomponent features
...
96 low-pass �lteredcomponent features
...
96 intermediate frequencycomponent features
...
Decoded selection
Prob
abili
ty
Threshold
Time
Selection
No selection
e
t
Word completion
<< BK
CL DRET
SP >> ETAO GYPBV KJXQZ .,’?:!-^RLDC UMWFINSH
<< BK
FlashSpeller Custom text-entry application designed for selection-based BCIs
Signal processing for LFP-based Flash Speller brain-computer interface
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
Cued aut
Samdecoder
Load from Historicaldecoder
m Ov laz g Pac m o i Five doze li uor g
decoder 1
decoder 2
decoder 1
decoder 2
decoder 1
decoder 2
decoder 1
decoder 2
Double bli
Build fr
decoder…
Par t rela ed
Comparison session
OLC
Cued autPar t rela ed…
Data collection session
Par t rela ed…
Communication session
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
T2 45249-162Hz
14-33Hz
T2 45357-232Hz
14-33Hz
T2 46049-174Hz
14-29Hz
T2 46753-158Hz
14-33Hz
T2 47449-236Hz
18-29Hz
T6 31357-213Hz
18-33Hz
T6 32264-350Hz
21-33Hz
T6 33664-330Hz
21-53Hz
T6 34861-334Hz
21-33Hz
T6 35561-205Hz
21-45Hz
0
Bot
tom
ban
d fr
eque
ncy
Top band frequency
0100Hz
0 0.2 0.4 0.6 0.8 1
Frequency band SNR normalized to maximum
100H
z
- Frequency band that gave the highest SNR in the intermediate and high frequency ranges
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
-5 0 5 10
Normalized feature
FeedbackOptionTransition
DUFP F nrcw r
Probability
LFCchannel 65
IFCchannel 14
HFCchannel 80
18 20 22 24 26 28 30 32 34 36 38
Selectedfeatures
Block time (s)
Threshold
FeedbackOptionTransition
DUFP
F nrcw
r [wor
ds]
Free
SP >>
etos
s dufp
p etose et
ose [w
ords
]
spee
ch
Probability
LFCchannel 65
IFCchannel 14
HFCchannel 80
Selectedfeatures
20 30 40 50 60 70 80 90 100
Threshold
D F nr r
4
FeedbackOptionTransition
Probability
LFCchannel 65
IFCchannel 14
HFCchannel 80
Selectedfeatures
50 100 150 200 250 300
Free speech is one of the most important rights
Threshold
os tos
[w sp
T6 493, first free-spelling block
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
0
50
100
150
200
T2 452, el. 95
Fre
quen
cy (
Hz)
Spe
ctro
gram
T2 500, el. 95 (+48 days) T2 550, el. 95 (+98 days)
10
20
30
40
50
Low
-pas
s fil
tere
dco
mpo
nent
Tria
ls
10
20
30
40
50
Inte
rmed
iate
freq
uenc
yco
mpo
nent
Tria
ls
0 1 2
10
20
30
40
50
Time from click cue (s)
Hig
h fr
eque
nncy
com
pone
nt
Tria
ls
0 1 2Time from click cue (s)
0 1 2Time from click cue (s)
Fea
ture
(s.
d.)
Fea
ture
(s.
d.)
Fea
ture
(s.
d.)
0246
02468
0
2
4
1
2
3
4
Nor
mal
ized
am
plitu
de
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
0 25 50 75 100
0
0.2
0.4
0.6
0.8
1
Cor
rela
tion
coef
ficie
nt
25 50 75 100 125 150
Number of days between compared sessions (day)
b = -0.02 year-1 b = -0.21 year-1
b, bC Slope of the linear fitLinear fit over Mr(s1,s2)Mean of Mr(s1,s2) for random events - RMr(s1,s2)Median of rf(s1,s2) over all channels - Mr(s1,s2)
T2 T6
0123
Fea
ture
(s.
d.)
012
0
Fea
ture
(s.
d.)
0
1
0 1 2
0
1
2
Fea
ture
(s.
d.)
0 1 2
0
1
2
3
Time from click cue (s)
el. 95 el. 80T2 T6
+98 days (T2 550)+48 days (T2 500)0 days (T2 452)
+151 days (T6 473)+75 days (T6 397)0 days (T6 322)
LFC
IFC
HFC
A
B
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
452453460467474487500510515523529545550
452
453
460
467
474
487
500
510
515
523
529
545
550
313
322
336
348
355
369
376
397
418
441
474
Cal
libra
tion
sess
ion
313
322
336
348
355
369
376
397
418
441
473
Test session Test session
0
0.2
0.4
0.6
0.8
1
0 25 50 75 1000
0.2
0.4
0.6
0.8
1
25 50 75 100 125 1500
0.2
0.4
0.6
0.8
1
Decoder age T (day)
P = 0.82 P = 0.46
CYX
T2 T6
Nor
mal
ized
mut
ual
info
rmat
ion
C YX
Decoder age T (day)
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
42 48 63 86 96 118 1380
2
4
6
8
Spe
lling
per
form
ance
(co
rrec
t cha
ract
ers
per
min
ute,
CC
PM
)
T6
Overall
*ns*
Decoder age (days)
13 26 36 41 49 55 56 71 760
1
2
3
4
T2
Overall
*ns*
Historical decoder with word completionFree Spelling blocks
Historical decoderCopy Phrase blocks
Same day decoderCopy Phrase blocks
P = 0.68 P = 0.18 P = 0.40
P = 0.95 P = 0.48 P = 0.73
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
487 500 510 515 523 524 529 530 545 550 Overall0
0.5
1
[13] [26] [36] [41] [49] [50] [55] [56] [71] [76]
T2
13 26 36 41 49 55 56 71 760.4
0.6
0.8
397 403 418 441 451 474 493 Overall0
0.5
1
[42] [48] [63] [86] [96] [119] [138]
T6
42 48 63 86 96 119 138
0.7
0.8
0.9
Nor
mal
ized
mut
ual i
nfor
mat
ion
C YX
Session (days from implantation) [Decoder age (days)] Decoder age (days)
P = 0.78 P = 0.42 P = 0.33
P = 0.67 P = 0.76 P = 0.13
Historical decoderCopy Phrase blocks
Historical decoder with word completionFree Spelling blocks
Same day decoderCopy Phrase blocks
Historical decoderFree Spelling blocks
ns **
ns **
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
487 500 510 515 523 529 545 550 Overall0
5
10
[13] [26] [36] [41] [49] [55] [71] [76]
T2
ns
397 418 441 473 Overall[42] [63] [86] [118]
T6
ns
Par
ticip
ant's
sco
res
Session (days from implantation) Decoder age [days]
Historical decoderSame day decoderCopy Phrase blocks:
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.
T2 45257-209Hz
18-29Hz
T2 45353-229Hz
21-29Hz
T2 46053-287Hz
18-29Hz
T2 46749-330Hz
21-29Hz
T2 474127-174Hz
21-29Hz
T6 313
21-29Hz
T6 322
21-33Hz
T6 336
21-29Hz
T6 348
21-29Hz
T6 355
21-25Hz
00
Bot
tom
ban
d fr
eque
ncy
Top band frequency
100H
z
0 0.2 0.4 0.6 0.8 1
Frequency band SNR normalized to maximum100Hz
369
376
397
418
441
474 Overall
*
0
1
487
500
510
515
523
529
545
550 Overall
*
Nor
mal
ized
mut
ual
info
rmat
ion C YX
T2 T6
Average signal decoder
Historical decoder
A
B
Session (days from implant placed)
Downloaded from www.physiology.org/journal/jn by ${individualUser.givenNames} ${individualUser.surname} (171.066.209.004) on April 30, 2018.Copyright © 2018 American Physiological Society. All rights reserved.