METHODS ARTICLEpublished: 28 April 2014

doi: 10.3389/fnhum.2014.00244

Decoding of four movement directions using hybridNIRS-EEG brain-computer interfaceM. Jawad Khan1, Melissa Jiyoun Hong2 and Keum-Shik Hong1,3*

1 Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, Republic of Korea2 Department of Education Policy and Social Analysis, Columbia University, New York, NY, USA3 School of Mechanical Engineering, Pusan National University, Busan, Republic of Korea

Edited by:

Francesco Di Russo, University ofRome “Foro Italico,” Italy

Reviewed by:

Mederic Descoins, NYU LangoneMedical Center, USARodolphe J. Gentili, University ofMaryland, USA

*Correspondence:

Keum-Shik Hong, Department ofCogno-Mechatronics Engineering,Pusan National University,2 Busandaehak-ro, Guemjeong-gu,Busan 609-735, Republic of Koreae-mail: kshong@pusan.ac.kr

The hybrid brain-computer interface (BCI)’s multimodal technology enables precisionbrain-signal classification that can be used in the formulation of control commands. In thepresent study, an experimental hybrid near-infrared spectroscopy-electroencephalography(NIRS-EEG) technique was used to extract and decode four different types of brain signals.The NIRS setup was positioned over the prefrontal brain region, and the EEG over theleft and right motor cortex regions. Twelve subjects participating in the experiment wereshown four direction symbols, namely, “forward,” “backward,” “left,” and “right.” Thecontrol commands for forward and backward movement were estimated by performingarithmetic mental tasks related to oxy-hemoglobin (HbO) changes. The left and rightdirections commands were associated with right and left hand tapping, respectively. Thehigh classification accuracies achieved showed that the four different control signals canbe accurately estimated using the hybrid NIRS-EEG technology.

Keywords: electroencephaelography, near-infrared spectroscopy, hybrid brain-computer interface, motor

execution, arithmetic mental task, linear discriminant analysis

INTRODUCTIONBrain-computer interface (BCI) is a methodology that corre-lates brain activities with external devices. The recent researchand trend have demonstrated the enormous potential of theBCI approach (Matthews et al., 2008; Nicolas-Alonso andGomez-Gil, 2012; Ortiz-Rosario and Adeli, 2013). The materialadvances in the cutting-edge technology, moreover, has reducedthe cost of BCI equipment. The BCI domain comprehendsboth invasive and non-invasive methods. Invasive methods suchas electrical-corticography (ECoG), though showing promisingsignal-acquisition results, are not recommended, as they entailvery significant risks. Non-invasive methods are much safer alter-natives in this regard (Min et al., 2010).

The major non-invasive modalities include electroencephalog-raphy (EEG), magneto encephalography (MEG), functional mag-netic resonance imaging (fMRI), and functional near-infraredspectroscopy (fNIRS). Each has its own strengths and limitations;the selection of one over another for brain-imaging applicationswill rely on the cost of the equipment as well as the spatial andtemporal resolution required for the given objective (Min et al.,2010).

EEG is a medical imaging technique that gauges brain activityby measuring, via metal electrodes positioned on the scalp, thevoltage fluctuations on the scalp resulting from neurons’ actionpotentials (Niedermeyer and Lopes da Silva, 1999; Rehan andHong, 2012). The drawback of EEG is the poor spatial resolutionthat does not allow an accurate localization, that is, identificationof the brain source signal (Ball et al., 2009).

NIRS is another non-invasive brain-imaging technique thatalternatively utilizes the near-infrared (NIR) spectrum oflight (wavelength 600–1000 nm) to measure the hemodynamic

response represented by oxygenated hemoglobin (HbO), deoxy-genated hemoglobin (HbR), cytochrome oxidase (CytOx) andwater (H2O) concentration changes (Nagdyman et al., 2003; Iraniet al., 2007; Bhutta et al., 2014). In most analyses, two hemo-dynamic variations due to brain activites are focused: increasedoxygenation (resulting from the increased neural activity) anddecreased deoxygenation (Matsuyama et al., 2009). Increasedoxygen consumption in the course of performing increasingly dif-ficult mental tasks has been demonstrated (Verner et al., 2013).fNIRS has also shown the ability to detect the fast optical response(Hu et al., 2011), however the hemodynamic response is mostlyused for analysis.

EEG offers good temporal resolution (∼0.05 s) but poor spa-tial resolution (∼10 mm), while fNIRS provides only moderatetemporal resolution (∼1 s) and also moderately better spatialresolution (∼5 mm) (Nicolas-Alonso and Gomez-Gil, 2012).Another advantage of fNIRS to EEG is its robustness to noise(Waldert et al., 2012). The objective of a hybrid BCI (Pfurtschelleret al., 2010) is either to improve the classification accuracy or/andto generate more control commands than the case of a singlemodality. The reason why Fazli et al. (2012) could improve theclassification accuracy by using a hybrid EEG and fNIRS config-uration is that they used the union of two cases (i.e., detected byeither EEG or fNIRS). This was possible because the window sizefor which the features are identified has been set to include bothfNIRS and EEG data. Even for some cases that EEG could notdetect, fNIRS could detect them. For motor execution, the aver-age classification accuracy by EEG alone was 90%, but EEG +HbR provided 93%, see Table 1 in Fazli et al. (2012). The previousstudies on single modality have shown that the classification accu-racy for two commands using fNIRS was about 65% (Stangl et al.,

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 1

HUMAN NEUROSCIENCE

Khan et al. Hybrid NIRS-EEG brain-computer interface

2013), and that for four commands using EEG (rotations andmovements of the left/right wrists) was about 65% (Vuckovic andSepulveda, 2012). The objective in this paper, however, is to gen-erate more commands without losing the classification accuracy.In this paper, four commands will be generated: two from theprefrontal cortex and two from the motor cortex by configuringfNIRS and EEG in such a way that each control signal is generatedfrom its associated brain region. In this way, a new command canbe generated in every 0.6 s and the achieved classification accuracywas over 80%.

The BCI techniques to control a wheel chair are diverse: eyemovement and blinking based (Gneo et al., 2011; Lin and Yang,2012), emotions based (Fattouh et al., 2013), and event relatedand state control (Galán et al., 2008; Huang et al., 2012; Carlsonand Millán, 2013). All these are reactive BCI, in which outputfrom the brain is generated in reaction to an external stimulation.The novelty in this work is the proposition of a hybrid config-uration of EEG and fNIRS for active BCI, whose classificationaccuracy is over 80%. Using four brain tasks (left/right motor exe-cution, mental counting, and mental arithmetic), four commandswere generated. In the proposed configuration, EEG electrodesare placed on the motor cortex region and NIRS optodes onthe prefrontal cortex region. The left and right directions weredecoded by tapping of the left or the right hand, and the mentalarithmetic and the mental counting was used to decode back-ward and forward directions. The classification accuracies of the12 subjects justify that the proposed configuration is suiltable forBCI and direction decoding which can be used for the generationof control commands for movement executions for the patientssuffering from lower-limb disorders.

FIGURE 1 | Experimental paradigm. One complete data set over the spanof a minute, consisting of four rest periods, two task periods detected byNIRS from the prefrontal brain region and two motor execution periodsdetected from C3 and C4 regions of brain using EEG.

METHODSPARTICIPANTS AND EXPERIMENTAL PARADIGMTwelve healthy volunteers (all male;10 right handed, two lefthanded; aged 24–34 years) participated in the experiment. Theexperiment was conducted under the declaration of Helsinki andconsent was taken from the subjects priror to the start of exper-iment. The experiment was performed in a confined room toreduce disturbance from the environment. The subjects were satin a comfortable chair with their arms on arm rests and instructedto relax. A screen nearly 70 cm away from the subjects was placedon which left, right, forward, and backward arrows were dis-played. On display of each arrow, a time marker starts at thebottom of screen indicationg the start and end of stimulus. Forright and left directions, the subjects were asked to tap their asso-ciated hands 10 times during the time period shown on the screenfor 10 s. For forward and backward directions, the subjects were

FIGURE 2 | Optode location for EEG and NIRS. (A) 8 EEG electrodesplacement over the Cz of the brain, (B) 12 channel locations on theprefrontal brain region using three sources and eight detectors.

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 2

Khan et al. Hybrid NIRS-EEG brain-computer interface

asked to do mental counting for 10 s and arithmetic subtractionfor 10 s. A training session was performed before the start tomake the subjects familier with the paradigm. The total durationof each experiment for each subject was 5 min, divided into restand activity periods. The time duration of each data sample was60 s. The initial 5 s of the experiment was the rest period, afterwhich the subjects were shown left or right direction symbolsand requested to physically tap their left or right hand accord-ingly, at a frequency of 1 Hz over a 10 s interval. The subjectswere also instructed to increase the strength of tap for left handfor better discrimination of signals. The next 5 s was another restperiod, subsequent to which the subjects were shown a forwardor backward direction symbol for an interval of 10 s. The subjectswere instructed to perform arithmetic subtraction, on a task sheetplaced 25 cm away prior to the start of the experiment, upon thedisplay of the backward direction symbol. For the mental arith-metic task, the subjects were asked to mentally perform a seriesof arithmetic calculations that were given on the sheet in a pseu-dorandom order. These calculations consisted of subtraction of atwo-digit number (between 10 and 20) from a three-digit numberthroughout the task period with successive subtraction of a two-digit number from the result of the previous subtraction (e.g.,695-19, 706-12, 894-15, etc). This mental activation period wasfollowed by a 5-s rest period, after which the subjects were againshown the left or right direction symbol and instructed, onceagain, to tap their hand at a 1 Hz frequency over the time spanof 10 s. If in a previous case the subjects were shown the left direc-tion symbol, the next time the indicated direction symbol was theopposite one. The directed hand activity was followed by 5 s rest,which was followed in turn by 10 s of mental activation. In thiscase as well, the next shown direction was reversed, according towhich the subjects were asked to perform an arithmetic counting

for indicated forward direction. For the mental counting task, thesubjects were asked to mentally count down from number “99”backwards. For “Stop” the subjects were asked not to perform anyactivity. A check on EEG and NIRS data values was placed to dis-tinguish the activation and resting state by defining the baselineand activity. The movement command can only be produced ifthere is activity in any one of the four brain regions else the stateshould be termed as “Stop” state. Figure 1 shows one completesample process; the second sample was taken immediately afterthe obtainment of the first.

SENSOR CONFIGURATIONEight EEG electrodes were placed on the motor cortex regionon the scalp and 12 channel NIRS was placed on the prefrontalbrain region. The reason for using NIRS on the prefrontal cor-tex is because it can discriminate between two activities from theprefrontal region with high classification accuracies (Naito et al.,2007; Power et al., 2010; Verner et al., 2013; Naseer et al., 2014)whereas the same cannot be done using EEG (Knyazev, 2013).Meanwhile NIRS signals are affected by dense hairs (Gervainet al., 2011) making EEG a better option for the detection ofbrain activities from the motor cortex region. Furthermore, ifboth modalities are positioned at the same brain location, theyinduce noise in each other thus reducing the strength of obtainedsignals for BCI (Safaie et al., 2013). Using the current setup, foursignals were obtained thus enhancing the performance of NIRSby combining with EEG setup.

DATA ACQUISITIONThe brain activities related to the mental tasks and motor exe-cutions were measured from the NIRS and EEG, respectively.Eight Ag/AgCl EEG electrodes were placed on C3, C4, T3, T4,

FIGURE 3 | System block diagram. The complete process from signal acquisition to control commands generation.

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 3

Khan et al. Hybrid NIRS-EEG brain-computer interface

P3, P4, F3, and F4 locations according to the International10–20 system (Homan et al., 1987; Pivick et al., 1993; Jurcaket al., 2007), and the data were recorded by g-MOBIlab+biosignal acquisition device (Christoph Guger, Austria) at asampling rate of 256 Hz. The NIRS-System (DYNOT, NIRxMedical Technologies, USA) was used in this experiment withwave length of 760 and 830 nm, respectively. A total of threesources and eight detectors forming a combinational pair of12 channels were used in the experiment. This assembly wasplaced on Fp1 and Fp2 regions of the brain and the optodeswere placed in the way that they cover the whole prefrontalarea in order to maximize the probability of locating the acti-vated region of brain. The sampling frequency used for theacquisition of NIRS signals was 1.81 Hz. Figure 2 shows thesource-detector locations for the optode placement for EEG andNIRS.

DATA ANALYSISThe fNIRS signals were obtained using the modifiedBeer-Lambert law (Coyle et al., 2007; Hu et al., 2010, 2012;

Kamran and Hong, 2013; Naseer and Hong, 2013).

A (t; λ) = lnIin (λ)

Iout (t; λ)= α (λ) × c (λ) × l × d (λ) + η, (1)

[�cHbO(t)

�cHbR(t)

]=

[αHbO (λ1) αHbR (λ1)

αHbO (λ2) αHbR (λ2)

]−1 [�A (t; λ1)

�A (t; λ2)

]

· 1

l × d(λ), (2)

where A is the absorbance of light (optical density), Iin is the inci-dent intensity of light, Iout is the detected intensity of light, α isthe specific extinction coefficient in μM−1cm−1, c is the absorberconcentration in μM, l is the distance between the source anddetector in cm, d is the differential path-length factor, and η is theloss of light due to scattering. In order to remove noise from thehemodynamic response, different techniques are used (Santosaet al., 2013). In the present study, respiration- and pulse-related

FIGURE 4 | Brain-signal acquisition: Each spike from the left or right motor cortex represents a hand tapping with the right or left hand, respectively. Theconcentration change in the prefrontal brain region reflects the change from rest to mental counting (“forward”) and mental subtraction (“backward”).

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 4

Khan et al. Hybrid NIRS-EEG brain-computer interface

noises were removed from the data using Gaussian low-pass fil-tering and wavelet transform (Ye et al., 2009; Hu et al., 2013).The β-band falling within the 12–30 Hz range was obtained byonline band-pass filtering of the EEG signals (Zaepffel et al.,2013; Kaiser et al., 2014). The epochs lasting 10 s were estimatedby segmenting the recordings from +1 to +11 s relative to theonset of the tapping stimulus, thus yielding five epochs for eachleft and right hand activity (Delorme and Makeig, 2004; Subasiand Gursoy, 2010; Turnip et al., 2011; Turnip and Hong, 2012).Linear discriminant analysis was used as the classifier. The fea-tures in the case of EEG were the mean values of peak amplitudesof channels C3 and C4 whereas in the case of NIRS, the meanvalues of HbO and HbR were used (Lotte et al., 2007; Zhanget al., 2013; Naseer et al., 2014). An online analysis was performedon the data by downsampling EEG data to 1.82 Hz to synchro-nize with the NIRS sampling rate. For both modalities, one datasample was obtained approximately after 0.5 s, which was thenprocessed at 250 Hz to obtain the control command. Figure 3shows the complete process for control command genereationfor BCI.

RESULTSThe signals extracted from the left and right brain hemispheres(the C3 and C4 regions) and the frontal brain hemisphere (theFp1 and Fp2 regions) are shown in Figure 4. The signal fromthe motor cortex region reflects the action potential generateddue to the firing of neurons when an motor execution task wasperformed. The signal from the prefrontal region is the hemo-dynamic change of HbO from the rest to mental executionperiod.

The association between movement and rest state was devel-oped by taking the common rest state as “Stop” indication. Thisis also beneficial as using this methodology only one commandcan be generated at one time and thus reduces the chance of miss-classification. The movement command can only be produced ifthere is activity in any one of the four brain regions else the stateshould be termed as “Stop” state as shown in Figure 5 and Table 1.

The left- and right-signal classifications are shown in Figure 6.Similary forward and backward mental tasks are represented inFigure 7. Figures 5, 6 show changes of state of data with respectto time. The results indicate a significant difference between the

FIGURE 5 | Stop condition: Common rest state of the three signals is selected as stop command. The “Stop” command can only be generated if absoluterest is detected from three brain regions.

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 5

Khan et al. Hybrid NIRS-EEG brain-computer interface

states. Table 2 lists the classification accuracies for all trials of the12 subjects.

DISCUSSIONThe first objective of this study, which was achieved, was to detectfour different classes of signal for generation of control com-mands for movement estimation suitable for BCI purposes. Twoclasses of signal were obtained by EEG using motor executionfrom the left and right motor cortex regions, and two classes are

Table 1 | Brain activities of EEG and NIRS for different control

commands.

Brain activity Command

EEG NIRS

Left hand tapping Rest LeftRight hand tapping Rest RightRest Mental arithmetic BackRest Mental counting ForwardRest Rest Stop

obtained by NIRS from the prefrontal brain region using mentalcounting and arithmetic. These two tasks are simple tasks thatare very easy to perform and can be detected from the prefrontalcortex. For the decoding application they serve their purposewell and have already been used in several BCI studies (Naitoet al., 2007; Power et al., 2012; Naseer et al., 2014). Figure 4clearly shows that each tap recorded from the left and right brainhemispheres due to the motor execution was recorded as a signalspike from the reference position. The time interval between eachtap was maintained consistent at approximately 1 s. In order todifferentiate between the left- and right-executed signals, eachtrial was separated by a time interval of 20 s. It was observed thatthe HbO concentration changes during the mental task began toincrease 2 s after the subjects were prompted, and that the HbOlevel required almost 12 s to settle after the termination of thatsignal. Accordingly, the time gap between the two NIRS tasks wasset at 20 s.

The classification of left and right hand tapping is per-formed by increasing the magnitude of left hand tapping fromthe right hand tapping. It has been reported in Jochumsenet al. (2013) and Robinson et al. (2013) that the neuronal

FIGURE 6 | EEG data classification for left and right control commands. (A) Shows the classification of trials for “Left” and “Stop” command signals forSubject 7, (B) Shows the classification of trials for “Right” and “Stop” command signals for Subject 6.

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 6

Khan et al. Hybrid NIRS-EEG brain-computer interface

FIGURE 7 | NIRS data classification for forward and backward control commands. (A) Shows the classification of trials for “Forward” and “Stop”command signals for Subject 3, (B) Shows the classification of trials for “Backward” and “Stop” command signals for Subject 7.

response due to fast and slow movement is different and canbe distinguished with high classification accuracy. Figures 6,7 show clearly that the four corresponding signals are sep-arable and, thus, utilizable for generation of BCI controlcommands.

The observed classification accuracies for the 12 subjects sig-nificantly show that the proposed research is suitable for BCIpurpose. In comparison to other studies in which two controlcommands are generated based on motor and arithmetic taskswith low classification accuracies results (Stangl et al., 2013), theresults have shown potential for control command generationwith high classification accuracy. Moreover four stage classifier isrequired if only one modality is used for four signals acquisitionwhich may results in significant decrease in accuracy (Vuckovicand Sepulveda, 2012). Using the current setup, the same resultswith better accuracy are achieved: The accuracies of “Forwardvs. Stop” and “Backward vs. Stop” trials were 80.2% and 83.6%,respectively, due to the selection of oxy-hemoglobin (HbO) as aclassification feature for both control commands. This is the first

time that four intended movements are decoded using a hybridBCI. Wolpaw and McFarland (2004) and the previous two hybridEEG-NIRS studies (Fazli et al., 2012; Kaiser et al., 2014) haveshown discrimination of two signals that can be used to gen-erate two control commands whereas in this paper four controlcommands have been generated.

The generated commands can be used for those people whocannot perform motor imagery (Vidaurre and Blankertz, 2010) orin such cases that the patient cannot touch any machine directly,for instance, prosthetic legs, working with remote controlleddevices, etc. Naseer et al. (2014) have shown binary decisiondecoding for rehabilitation, whereas in this research the fourcontrol commands can be associated to four different decisions.Moreover, the previous hybrid EEG based researches for rehabili-tation have shown the use of P300 and steady state visual evokedpotentials (Li et al., 2013; Xu et al., 2013) to generate four con-trol commands based on reactive tasks. As motor imagery andmotor execution activate the same brain area (Beisteiner et al.,1995; Porro et al., 1996), the same goal can be achived by this

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 7

Khan et al. Hybrid NIRS-EEG brain-computer interface

Table 2 | Classification accuracies: four control command accuracies

of 12 subjects for left vs. stop, right vs. stop, forward vs. stop, and

backward vs. stop.

Subject Left vs. Right vs. Forward vs. Back vs.

Stop (%) Stop (%) Stop (%) Stop (%)

1 98.4 98.7 81.1 86.52 99.8 99.2 79.1 84.13 93.2 94.7 81.2 85.44 97.7 96.5 80.1 80.85 98.1 88.2 81.2 86.96 91.1 99.5 84.2 86.67 99.2 98.1 77.2 81.18 98.0 98.2 83.1 87.19 91.2 90.4 80.2 83.110 85.0 84.1 74.1 76.311 94.5 95.2 82.8 86.112 90.1 94.0 78.2 79.3

Mean 94.7 ± 4.6 94.7 ± 4.8 80.2 ± 2.7 83.6 ± 3.5

research using active tasks. Also, it is known that fNIRS candecode multiple signals from the same brain area: for instance,mental counting and music imagery (Power et al., 2012) and pic-ture imagery and mental arithmetic (Naito et al., 2007) from thesame prefrontal cortex. Therefore, there is a potential that thetotal number of commands in the current EEG and fNIRS config-uration can be increased to up to seven. Further research can becarried out using advanced adaptive filtering techniques (Rehanand Hong, 2013), optimal feature sets, and/or combined EEG-NIRS features to achieve better classification results that can beused by patients with lower limb disorder to control wheelchairor prostheses.

CONCLUSIONIn this research, a hybrid fNIRS and EEG configuration for decod-ing four movment commands was proposed. The NIRS setup wasused to decode the prefrontal activities based on hemodynamicchanges of HbO using mental arithmetic and mental count-ing as tasks. The EEG response detected from the motor cortexregion was used to decode other two direction commands. Bothmodalities were synchronized to obtain control commands at thesame time. The results of classification accuracies were highlyencouraging and certainly will prove fruitfully applicable to BCIsystems and purposes. The use of wirelss systems and transla-tion of the control commands into machine codes will enableeffective control of a robotic system suitable for rehabilitationpurposes.

AUTHOR CONTRIBUTIONSM. Jawad Khan has conducted all the experiments andcarried out the data processing. Melissa J. Hong has exam-ined the data and participated in revising the manuscript.Keum-Shik Hong has suggested the theoretical aspectsof the current study and supervised all the process fromthe beginning. All the authors have approved the finalmanuscript.

ACKNOWLEDGMENTSThis work was supported by the National Research Foundationof Korea funded by the Ministry of Education, Scienceand Technology, Korea (grant no. MEST-2012-R1A2A2A01046411).

REFERENCESBall, T., Kern, M., Mutschler, I., Aertsen, A., and Schulze-Bonhage, A.

(2009). Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46, 708–716. doi: 10.1016/j.neuroimage.2009.02.028

Beisteiner, R., Höllinger, P., Lindinger, G., Lang, W., and Berthoz, A. (1995). Mentalrepresentations of movements. Brain potentials associated with imaginationof hand movements. Electroencephalogr. Clin. Neurophysiol. 96, 183–193. doi:10.1016/0168-5597(94)00226-5

Bhutta, M. R., Hong, K.-S., Kim, B.-M., Hong, M.-J., Kim, Y.-H., and Lee,S.-H. (2014). Note: three wavelengths near-infrared spectroscopy system forcompensating the light absorbance by water. Rev. Sci. Instrum. 85, 1–3. doi:10.1063/1.4865124

Carlson, T., and Millán, J. R. (2013). Brain-controlled wheelchairs: a roboticarchitecture. IEEE Robot. Automat. Mag. 20, 65–73. doi: 10.1109/MRA.2012.2229936

Coyle, S. M., Ward, T. E., and Markham, C. M. (2007). Brain-computer interfaceusing a simplified functional near-infrared spectroscopy system. J. Neural Eng.3, 219–226. doi: 10.1088/1741-2560/4/3/007

Delorme, A., and Makeig, S. (2004). EEGlab: an open source toolbox for anal-ysis of single trail EEG dynamics. J. Neurosci. Methods 134, 9–21. doi:10.1016/j.jneumeth.2003.10.009

Fattouh, A., Horn, O., and Bourhis, G. (2013). Emotional BCI control of a smartwheelchair. Int. J. Comput. Sci. 10, 32–36.

Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R.,et al. (2012). Enhanced performance by a hybrid NIRS-EEG braincomputer interface. Neuroimage 59, 512–529. doi: 10.1016/j.neuroimage.2011.07.084

Galán, F., Nuttin, M., Lew, E., Ferrez, P. W., Vanacker, G., Philips, J., et al. (2008).A brain-actuated wheelchair: asynchronous and non-invasive brain-computerinterfaces for continuos control of robots. Clin. Neurohysiol. 119, 2159–2169.doi: 10.1016/j.clinph.2008.06.001

Gervain, J., Mehler, J., Werker, J. F., Nelson, C. A., Csibra, G., Lloyd-Fox, S., et al. (2011). Near-infrared spectroscopy: a report from theMcDonnell infant methodology consortium. Dev. Cogn. Neurosci. 1, 22–46. doi:10.1016/j.dcn.2010.07.004

Gneo, M., Severini, G., Conforto, S., Schmid, M., and D’Alessio, T. (2011). Towardsa brain-activated and eye-controlled wheel chair. Int. J. Bioeletromagn. 13,44–45.

Homan, R. W., Herman, J., and Purdy, P. (1987). Cerebral location of international10-20 system electrode placement. Electroencephalogr. Clin. Neurophysiol. 66,376–382. doi: 10.1016/0013-4694(87)90206-9

Hu, X.-S., Hong, K.-S., and Ge, S. S. (2011). Recognition of stimulus-evokedneuronal optical response by identifying chaos levels of near-infrared spec-troscopy time series. Neurosci. Lett. 504, 115–120. doi: 10.1016/j.neulet.2011.09.011

Hu, X.-S., Hong, K.-S., and Ge, S. S. (2012). fNIRS-based online deceptiondecoding. J. Neural Eng. 9:2. doi: 10.1088/1741-2560/9/2/026012

Hu, X.-S., Hong, K.-S., and Ge, S. S. (2013). Reduction of trail-to-trial variabilityin functional near-infrared spectroscopy signals by accounting for resting-state functional connectivity. J. Biomed. Opt. 18, 1–9. doi: 10.1117/1.JBO.18.1.017003

Hu, X.-S., Hong, K.-S., Ge, S. S., and Jeong, M.-Y. (2010). Kalman estimator-and general linear model-based on-line brain activation mapping bynear-infrared spectroscopy. Biomed. Eng. Online 9:82. doi: 10.1186/1475-925X-9-82

Huang, D., Qian, K., Fei, D.-Y., Jia, W., Chen, X., and Bai, O. (2012).Electroencephalography (EEG)-based brain-computer interface (BCI): a2-D virtual wheelchair control based on event-related desynchorniza-tion/synchronization and state control. IEEE Trans. Neural Syst. Rehabil. Eng.20, 379–388. doi: 10.1109/TNSRE.2012.2190299

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 8

Khan et al. Hybrid NIRS-EEG brain-computer interface

Irani, F., Platek, S. M., Bunce, S., Ruocco, A. C., and Chute, D. (2007).Functional near-infrared spectroscopy (fNIRS): an emerging neu-roimaging technology with important applications for the study ofbrain disorders. Neuropsychologist 21, 9–37. doi: 10.1080/13854040600910018

Jochumsen, M., Niazi, I. K., Mrachacz-Kersting, N., Farina, D., and Dremstrup,K. (2013). Detection and classification of movement-related cortical poten-tials associated with task forece and speed. J. Neural Eng. 10, 5, 1–9. doi:10.1088/1741-2560/10/5/056015

Jurcak, V., Tsuzuki, D., and Dan, I. (2007). 10/20, 10/10, and 10/5 system revis-ited: their validity as head-surface-based positioning system. Neuroimage 34,1600–1611. doi: 10.1016/j.neuroimage.2006.09.024

Kaiser, V., Bauernfeind, G., Kreilinger, A., Kaufmann, T., Kubler, A., Neuper, C.,et al. (2014). Cortical effects of user training in a motor imagery based brain-computer interface measured by fNIRS and EEG. Neuroimage 85, 432–444. doi:10.1016/j.neuroimage.2013.04.097

Kamran, M. A., and Hong, K.-S. (2013). Linear parameter varying model andadaptive filtering technique for detecting neuronal activities: an fNIRS study.J. Neural Eng. 10:5. doi: 10.1088/1741-2560/10/5/056002

Knyazev, G. G. (2013). EEG correlates of self-referential processing. Front. Hum.Neurosci. 7:264. doi: 10.3389/fnhum.2013.00264

Li, Y. Q., Pan, J. H., Wang, F., and Yu, Z. (2013). A hybrid BCI system combiningP300 and SSVEP and its application to wheel chair control. IEEE Trans. Biomed.Eng. 60, 3156–3166. doi: 10.1109/TBME.2013.2270283

Lin, J.-S., and Yang, J.-S. (2012). Wireless brian-computer interface for electricwheelcahirs with EEG and eye-blinking signals. Int. J. Innovat. Comput. Inform.Control 8, 6011–6024.

Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., and Arnaldi, B. (2007). A reviewof classification algorithms for EEG-based brain-computer interfaces. J. NeuralEng. 4:2. doi: 10.1088/1741-2560/4/2/R01

Matsuyama, H., Asama, H., and Otake, M. (2009). “Design of differential near-infrared spectroscopy based brain machine interface,” in IEEE InternationalSymposium on Robot and Human Interactive Communication, (Toyama),775–780. doi: 10.1109/ROMAN.2009.5326215

Matthews, F., Pearlmutter, B. A., Ward, T. E., Soraghan, C., and Markham, C.(2008). Hemodynamics for brain computer interfaces. IEEE Signal Process. Mag.25, 87–94. doi: 10.1109/MSP.2008.4408445

Min, B. K., Marzelli, M. J., and Yoo, S.-S. (2010). Neuroimaging-basedapproaches in brain-computer interface. Trends Biotechnol. 28, 552–560. doi:10.1016/j.tibtech.2010.08.002

Nagdyman, N., Fleck, T. P. K., Ewert, P., Abdul-Khaliq, H., Redlin, M., and Lange, P.E. (2003). Cerebral oxygenation measured by near-infrared spectroscopy duringcirculatory arrest and cardiopulmonary resuscitation. Br. J. Anesth. 91, 438–442.doi: 10.1093/bja/aeg181

Naito, M., Michioka, Y., Ozawa, K., Ito, Y., Kiguchi, H., and Kanazaw, T.(2007). A communication means for totally locked-in ALS patientsbased on changes in cerebral blood volume measured with near-infraredlight. IEICE Trans. Infrom. Syst. 90, 1028–1037. doi: 10.1093/ietisy/e90-d.7.1028

Naseer, N., and Hong, K.-S. (2013). Classification of functional near-infraredspectroscopy signals corresponding to right- and left-wrist motor imagery fordevelopment of a brain-computer interface. Neurosci. Lett. 553, 84–89. doi:10.1016/j.neulet.2013.08.021

Naseer, N., Hong, M. J., and Hong, K.-S. (2014). Online binary decision decodingusing functional near-infrared spectroscopy for the development of brain-computer interface. Exp. Brain Res. 232, 555–564. doi: 10.1007/s00221-013-3764-1

Nicolas-Alonso, L. F., and Gomez-Gil, J. (2012). Brain-computer interfaces, areview. Sensors 12, 1211–1279. doi: 10.3390/s120201211

Niedermeyer, E., and Lopes da Silva, F. H. (1999). Electroencephalography: BasicPrinciples, Clinical Applications and Related Fields. Philadelphia, PA: LippincottWilliam and Wilkins.

Ortiz-Rosario, A., and Adeli, H. (2013). Brain-computer interface technologies:from signal to action. Rev. Neurosci. 24, 537–552. doi: 10.1515/revneuro-2013-0032

Pfurtscheller, G., Allison, B. Z., Brunner, C., Bauernfeind, G., Solis-Escalante,T., Scherer, R., et al. (2010). The hybrid BCI. Front. Neurosci. 4:30. doi:10.3389/fnpro.2010.00003

Pivick, R. T., Broughton, R. J., Coppola, R., Davidson, R. J., Fox, N., and Nuwer,M. R. (1993). Guidelines for recording and quantitative analysis of electroen-cephalographic activity in research contexts. Psychophysiology 30, 547–558. doi:10.1111/j.1469-8986.1993.tb02081.x

Porro, C. A., Francescato, M. P., Cettolo, V., Diamond, M. E., Baraldi, P., Zuiani,C., et al. (1996). Primay motor and sensory cortex activation during motor per-formance and motor imagery: a functional magnetic resonance imaging study.J. Neurosci. 16, 7688–7698.

Power, S. D., Falk, T. H., and Chau, T. (2010). Classification of prefrontal activitydue to mental arithmetic and music imagery using hidden Markov models andfrequency domain near-infrared spectroscopy. J. Neural Eng. 7:2. doi: 10.1088/1741-2560/7/2/026002

Power, S. D., Khushki, A., and Chau, T. (2012). Automatic single-trial discrim-ination of mental arithmetic, mental singing and no-control state form theprefrontal activity: towards the three state NIRS-BCI. BMC Res. Notes 5:141.doi: 10.1186/1756-0500-5-141

Rehan, M., and Hong, K.-S. (2012). Robust synchronization of delayedchaotic FitzHugh-Nagumo neurons under external electrical stimu-lation. Comput. Math. Method Med. 2012, 1–11. doi: 10.1155/2012/230980

Rehan, M., and Hong, K.-S. (2013). Modeling and automatic feedback control oftremor: adaptive estimation of deep brain stimulation. PLoS ONE 8:e62888. doi:10.1371/journal.pone.0062888

Robinson, N., Vinod, A. P., Ang, K. K., Tee, K. P., and Guan, C. T.(2013). EEG-based classification of fast and slow hand movements usingwavelet-CSP algorithm. IEEE Trans. Biomed. Eng. 60, 2123–2132. doi:10.1109/TBME.2013.2248153

Safaie, J., Grebe, R., Moghaddam, H. A., and Wallois, F. (2013). Towards a fullyintegrated wearable EEG-NIRS bimodal acquisition system. J. Neural Eng. 10:5.doi: 10.1088/1741-2560/10/5/056001

Santosa, H., Hong, M. J., Kim, S.-P., and Hong, K.-S. (2013). Noise reductionin functional near-infrared spectroscopy signals by independent componentanalysis. Rev. Sci. Instrum. 84:7. doi: 10.1063/1.4812785

Stangl, M., Bauernfeind, G., Kurzmann, J., Scherer, R., and Neuper, C. (2013).A heamodynamic brain-computer interface based on real-time classifica-tion of near infrared spectroscopy signals during motor imagery andmental arithmetic. J. Near Infrared Spectrosc. 21, 157–171. doi: 10.1255/jnirs.1048

Subasi, A., and Gursoy, M. I. (2010). EEG signal classification using PCA, ICA,LDA and support vector machines. Expert Syst. Appl. 37, 8659–8666. doi:10.1016/j.eswa.2010.06.065

Turnip, A., and Hong, K.-S. (2012). Classifying mental activities from EEG-P300signals using adaptive neural network. Int. J. Innovat. Comput. Inform. Control8, 6429–6443.

Turnip, A., Hong, K.-S., and Jeong, M.-Y. (2011). Real-time feature extractionof EEG-based P300 using adaptive nonlinear principal component analysis.Biomed. Eng. Online 10:83. doi: 10.1186/1475-925X-10-83

Verner, M., Herrmann, M. J., Troche, S. J., Robers, C. M., and Rammsyaer, T. M.(2013). Cortical oxygen consumption in mental arithmetic as a function taskdifficulty: a near-infrared spectroscopy approach. Front. Hum. Neurosci. 7:217.doi: 10.3389/fnhum.2013.00217

Vidaurre, C., and Blankertz, B. (2010). Towards a cure for BCI illiteracy. BrainTopogr. 23, 194–198. doi: 10.1007/s10548-009-0121-6

Vuckovic, A., and Sepulveda, F. (2012). A two-stage four-class BCI based on ima-genary movements of the left and right wrist. Med. Eng. Phys. 34, 964–971. doi:10.1016/j.medengphy.2011.11.001

Waldert, S., Tüshaus, L., Kaller, C. P., Aertsen, A., and Mehring, C. (2012). fNIRSexhibits weak tuning to hand movement direction. PLoS ONE 7:e49266. doi:10.1371/journal.pone.0049266

Wolpaw, J. R., and McFarland, D. J. (2004). Control of two-dimensional movementsignal by a noninvasive brain-comuter interface in humans. Proc. Natl. Acad. Sci.U.S.A. 101, 17849–17852. doi: 10.1073/pnas.0403504101

Xu, M. P., Qi, H. Z., Wan, B. K., Yin, T., Liu, Z. P., and Ming, D. (2013). A hybridBCI speller paradigm combining P300 potential and SSVEP blocking feature.J. Neural Eng. 10:2. doi: 10.1088/1741-2560/10/2/026001

Ye, J. C., Tak, S., Jang, K. E., Jung, J., and Jang, J. (2009). NIRS-SPM: statisticalparametric mapping for near-infrared spectroscopy. Neuroimage 44, 428–447.doi: 10.1016/j.neuroimage.2008.08.036

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 9

Khan et al. Hybrid NIRS-EEG brain-computer interface

Zaepffel, M., Trachel, R., Kilavik, B. E., and Brochier, T. (2013). Modulation of EEGbeta power during planning and execution of grasping movements. PLoS ONE8:e60060. doi: 10.1371/journal.pone.0060060

Zhang, R., Xu, P., Guo, L., Zhang, Y., Li, P., and Yao, D. (2013). Z-score lineardiscriminant analysis for EEG based brain-computer interfaces. PLoS ONE8:e74433. doi: 10.1371/journal.pone.0074433

Conflict of Interest Statement: The authors declare that the researchwas conducted in the absence of any commercial or financialrelationships that could be construed as a potential conflict ofinterest.

Received: 13 November 2013; accepted: 03 April 2014; published online: 28 April 2014.Citation: Khan MJ, Hong MJ and Hong K-S (2014) Decoding of four movement direc-tions using hybrid NIRS-EEG brain-computer interface. Front. Hum. Neurosci. 8:244.doi: 10.3389/fnhum.2014.00244This article was submitted to the journal Frontiers in Human Neuroscience.Copyright © 2014 Khan, Hong and Hong. This is an open-access article distributedunder the terms of the Creative Commons Attribution License (CC BY). The use, dis-tribution or reproduction in other forums is permitted, provided the original author(s)or licensor are credited and that the original publication in this journal is cited, inaccordance with accepted academic practice. No use, distribution or reproduction ispermitted which does not comply with these terms.

Frontiers in Human Neuroscience www.frontiersin.org April 2014 | Volume 8 | Article 244 | 10