Telematics and Informatics 32 (2015) 180–192

Contents lists available at ScienceDirect

Telematics and Informatics

journal homepage: www.elsevier .com/locate / te le

Brain–computer interface: The next frontier of telemedicinein human–computer interaction

http://dx.doi.org/10.1016/j.tele.2014.07.0010736-5853/� 2014 Elsevier Ltd. All rights reserved.

⇑ Tel.: +886 5 2720411x34621; fax: +886 5 2721501.E-mail addresses: shenswy@gmail.com, shen@csie.ncku.edu.tw

Wei-Yen Hsu ⇑Department of Information Management, National Chung Cheng University, No. 168, Sec. 1, University Rd., Min-Hsiung Township, Chiayi County 621, Taiwan

a r t i c l e i n f o a b s t r a c t

Article history:Received 9 April 2014Received in revised form 30 June 2014Accepted 3 July 2014Available online 15 July 2014

Keywords:Brain–computer interfaceTelemedicineHuman–computer interactionDiscriminative area selection

The study proposes a novel brain–computer interface scheme for the next frontier of tele-medicine in human–computer interaction, where the goal is to improve the interactionsbetween users and computers in telemedicine. The system consists of discriminative areaselection, feature extraction and classification. Discriminative area selection is proposed toobtain the optimal discriminative area, which can decrease the time length of event-related area to achieve more efficient computation and higher accuracy. A fuzzy Hopfieldneural network is used to classify the features extracted by means of wavelet-fractalapproach. Experimental results show that the proposed system is robust and performs bet-ter than several previous methods. It is also suggested being suitable for the applications oftelemedicine in human–computer interaction.

� 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Since beginning of computer use, input devices have been limited to a few well-understood instruments, such as punchcards, the joystick, keyboard, and mouse. In the past decade, however, the potential of unconventional input devices hasbecome increasingly apparent, and a wide variety of sensors and mechanisms are in the process of development(Williamson et al., 2009). Currently, the touch panel, somatosensory, and voice control are the most popular new inputdevices for human–computer interaction (HCI). Though commercial applications of the brain–computer interface (BCI)are still problematic, researchers currently involved in HCI have recognized the importance of emotional assessment usingelectroencephalographic (EEG) methods (Channel et al., 2009).

2. Objective

BCI is a new communication system that provides an alternative channel for directly transmitting messages from thehuman brain to computers by analyzing the brain’s mental activities (Williamson et al., 2009). Further, BCI might be a highlyfeasible method to assist limb-disabled users. BCI systems based on the single-trial analysis of EEG signals associated withfinger lifting (FL) or motor imagery (MI) have grown rapidly in the last decade. EEG analysis is based on discriminating theleft and right FL/MI using event-related brain potentials (ERP). For this, the raw EEG data are continuous signals in thetime-domain that can be transformed by means of filters. These include, spatial filters, and an effective selection of the mostappropriate frequency-bands in the frequency domain is known to improve the classification accuracy (Aler et al., 2012).

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 181

However, the EEG data are non-stationary and their characteristics vary with time. This suggests that there are special char-acteristics of event-related desynchronization and synchronization in l and b rhythms over the sensorimotor cortices duringthe tasks (Williamson et al., 2009).

3. State of the art

Generally, BCI systems consist of two main processes, feature extraction and classification. Feature extraction is an impor-tant process since it greatly affects the effectiveness of EEG classification accuracy. Feature extraction is based on bandpower and usually computes the powers at the a and b bands, whose frequency components are in mental tasks(Obermaier et al., 2001). A great many feature extraction approaches have been proposed, among which, band power(Obermaier et al., 2001) and autoregressive (AR) parameters (Burke et al., 2005) are the most commonly used. The EEG timeseries is fitted with an AR model to produce the parameters.

Fractal geometry provides a proper mathematical model to describe complex and irregular shapes that exist in nature byusing fractal features (Mandelbrot, 1982). Fractal dimensions are one of the most commonly used fractal features and theyhave been applied to various fields. In the present study, we decompose selected discriminative area into multi-scale signalsby means of discrete wavelet transform (DWT). The wavelet-fractal features (WFFs) are then extracted via the proposedmodified fractal dimensions. In addition to multi-scale characteristics, WFFs also contain important fractal information inthe time-scale space.

For the classification process, supervised classifiers are usually adopted to recognize single-trial FL/MI EEG data. Some ofthese classifiers, such as linear discriminant analysis (Saprikis, 2013), multilayer perceptron (Telfer et al., 1993), support vec-tor machines (Doukas et al., 2011), and artificial neural networks (ANN) (Sim et al., 2014), are quite popular and generallyused for this type of classification. However, they are supervised, and their parameters need to be trained in advance beforebeing applied to on-line applications. The fuzzy Hopfield neural network (FHNN) is an unsupervised approach that combineswith the fuzzy clustering method (FCM) (Dunn, 1973) and the Hopfield neural network (HNN) (Hopfield, 1982), and thenpartitions a collection of feature vectors into a number of subgroups based on minimizing the trace of a within-cluster scat-ter matrix (Hsu, 2012). Hsu (2012) also indicated that the classification of FL/MI EEG data with an unsupervised FHNN maylead to better classification accuracy than can be obtained with conventional supervised classifiers. Therefore, the presentstudy employed ANN, HNN, and FHNN for the single-trial classification.

4. Materials and methods

4.1. Data acquisition and description

Three datasets were used to evaluate the performance of the proposed scheme.In the first dataset, EEG signals associated with FL signals were recorded from four untrained subjects (three males and

one female, two left-handed and two right-handed) in a shielded room. As illustrated in Fig. 1 there were 13 silver/silverchloride electrodes, including ten scalp EEG channels (C3, C5, FC3, C1, CP3, C4, C2, C6, FC4, and CP4), two EMG channels

Fig. 1. Location of EEG electrodes for the first data set.

182 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

for monitoring left and right muscle activities, and one channel on the forehead to record possible electrooculographic arti-facts and eye blinks during the experiment. All electrodes were referenced to the A1 lead at the left earlobe. Before beingsampled at the rate of 256 Hz, the EEG data were filtered by an analog band-pass filter with cutoff frequencies at 0.5 Hzand 100 Hz, and amplified by a multiple of 10,000. During the experiments, each subject was asked to perform two trialsthat included left and right FL in each test. Fig. 2 illustrates the experimental protocol. Each trial was 10 s in length, sothe tests were 20 s long. For each lifting trial, the first 4 s were quiet and then an acoustic stimulus was given as a cue tosignal the beginning of left FL. At the same time, each subject was asked to execute FL. An example of a test is shown inFig. 3. Each subject recorded 60 tests; making a total of 120 trials for each subject. No trials were removed during theEEG data processing stage. Data segments for FL were acquired from second �2 to second 2, where second 0 stands forthe trigger of movement by detecting the peak EMG signal after linear envelope processing (only the data recorded betweenseconds �2 and 2 were considered to be event-related).

For the second and third datasets, EEG signals associated with MI were recorded. The second dataset is the Graz datasetfor the BCI (2003) competition, which was recorded from three subjects during a feedback experimental recording proce-dure. The third dataset is also the Graz dataset for the BCI (2005) competition, also recorded from three subjects, but usinga Neuroscan EEG amplifier. The left and right mastoids served as a reference and ground, respectively.

For the second dataset, the task was to control a bar by means of imagined left or right hand movements. The order of leftand right cues was random. The data were recorded on three subjects, the first of which performed 280 trials, while othertwo each performed 320 trials. The length of each trial was 8–9 s. The first 2 s of the trial were quiet; then an acoustic stim-ulus indicated the beginning of the experiment at t = 2 s, and a fixation cross + was displayed for 1 s. At t = 3 s, an arrow (leftor right) was displayed as a cue. (Only the data recorded between 3 and 8 s were considered to be event-related.) At the sametime, each subject was asked to move a bar by imagining left or right hand movements according to the direction of the cue.The recordings were made using a g.tec amplifier and Ag/AgCl electrodes. All signals were sampled at 128 Hz and filteredbetween 0.5 and 30 Hz. The EEG electrodes, channels C3 and C4, are located in the international standard 10–20 system.

For the third dataset, the EEG data were sampled at 250 Hz and filtered between 1 and 50 Hz. The subjects were asked toperform imaginary movements prompted by a visual cue. Each trial started with an empty black screen. At t = 2 s, a shortbeep tone was presented and a cross ‘ + ’ appeared on the screen to notify the subjects. Then at t = 3 s an arrow lasting

Fig. 2. Experimental protocol for the first data set.

Fig. 3. Example of a test for the first data set.

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 183

for 1.25 s pointed to either the left or right direction. Each direction indicates the subjects to imagine either a left or righthand movement. The imagery movements were performed until the cross disappeared at t = 7 s. No feedback was performedin the experiments. The data recorded from first subject were 180 trials, while the data for last two subjects were 120 trials.

4.2. Comparisons with other systems in major features and mechanisms

Two systems are the most commonly used in BCI works. The first system recognizes EEG signals based on band powerestimates and hidden Markov models (Obermaier et al., 2001), whereas parametric modeling strategies in conjunction withlinear discriminant analysis are used for EEG discrimination in the second system (Burke et al., 2005). More specifically, thefirst system uses band power estimates and hidden Markov models to classify EEG patterns, which are combined based onthe separability of subsets of mental tasks. The second system explores the useful information and features in combinationwith parametric modeling strategies and linear discriminant analysis. The ensemble average evoked and event-relatedpotentials are used to estimate the models.

However, these two systems lack of the feature and mechanism of discriminative area selection. They would result intime-consuming computation and performance degradation at the same time, which greatly influences the real-time appli-cations of telemedicine in human–computer interaction. In addition, feature extractions used in these two systems are bandpower estimates and parametric modeling strategies, respectively. It has been proven that their performance is worse thanfractal-based features. Undoubtedly, they also perform worse in comparison with our proposed wavelet-fractal approach.Finally, these two systems use hidden Markov models and linear discriminant analysis for feature classification. In general,neural-network-based classifiers work better than they do in classification accuracy. Accordingly, the proposed system ispromising as compared to these two commonly used systems in all of the features and mechanisms, including discriminativearea selection, feature extraction and classification.

In summary, we propose a novel BCI system for the next frontier of telemedicine in HCI. It consists of discriminative areaselection, feature extraction and classification. Discriminative area selection detects the optimal discriminative area, whichcan effectively fulfill more efficient computation and higher accuracy, and this feature and mechanism is lacking in these twocommonly used systems. The features are extracted by means of wavelet-fractal approach, whose performance is better thanother two, band power estimates and parametric modeling strategies. Finally, the fuzzy Hopfield neural network having bet-ter performance than the two traditional classifiers, hidden Markov models and linear discriminant analysis, is used for fea-ture classification. The descriptions indicate that the proposed system is suitable for the applications of telemedicine in HCI.

4.3. Our proposed system

A flowchart of the novel BCI scheme is outlined in Fig. 4. It consists of three main processes, including discriminative areaselection, feature extraction, and FHNN classification. Discriminative area selection (DAS) has not previously been used fordetecting the location of a discriminative region by means of both continuous wavelet transform (CWT) and Pearson prod-uct-moment correlation coefficient (PPMCC). The WFFs are then extracted from the DWT data via the modified fractaldimension. Finally, FHNN is used to discriminate WFFs into two classes, left and right FL/MI, without supervision.

4.3.1. Discriminative area selectionGenerally, feature extraction and classification are the two main processes used in BCI systems over the past decade. Since

the success of EEG classification accuracy is greatly affected by feature extraction, before extracting important features, eachtrial usually contains 5–7 s event-related area. This, however, is too redundant to be effectively analyzed. Thus, in order toachieve efficient computation and increase classification accuracy, the length of event-related areas should be decreased byselecting only 1–2 s discriminative area, which is defined as the most discriminative area. The better the selected discrim-inative area, the higher the classification accuracy and computation speed.

Since CWT gives a highly complete representation of EEG signals in the time–frequency domain, it is used for the preciselocalization of ERP components (Hsu and Sun, 2009). The CWT has previously been applied for precise localization of the ERPcomponents (Hsu and Sun, 2009). Therefore, discriminative area selection based on the CWT and PPMCC is first adopted hereto obtain the optimal discriminative area from the time–frequency domain and event-related area, while remaining offline.

The CWTs of EEG signals for each trial, which are of either left or right FL/MI, are analyzed in the C3 and C4 channels,respectively,

Wðj; kÞ ¼Z

Rf ðxÞ 1ffiffi

jp w

x� kj

� �dx ð1Þ

where 1 ffijp wðx�k

j Þ are the dilated and translated versions of the wavelet function wðxÞ at scale j and shift k, and W(j, k) repre-

sents the CWT of the EEG signal f(x). W(j, k) is represented as a 2D time-scale plot that retains the scale separation of ERPcomponents.

However, the 2D time-scale plot W(j, k) generated from the CWT is so noisy that it is difficult to accurately locate thediscriminative area. In addition, the best discriminative area is not situated at a single point but lies over a range of timeand scale. Using a wider frequency range can achieve higher classification accuracy than using a narrower one from the

Fig. 4. Flowchart of the BCI scheme.

184 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

acquired EEG signals. Accordingly, we take the wider time and scale range into account by convoluting a larger Gaussian fil-ter, because the wider range contains all the mu and beta rhythmic components that are important for mental task classi-fication. Finally, we accumulate the power spectrum along the scale direction within the wide frequency range. The resultingprofile represents the brain ERP responses for left and right FL/MI.

PPMCC and discriminative area R between the left and right FL/MI are then evaluated from the training data set in both C3and C4 channels,

rðkÞ ¼PN

i¼1ðPiLðkÞ � PLðkÞÞðPi

RðkÞ � PRðkÞÞffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPNi¼1ðP

iLðkÞ � PLðkÞÞ

2q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPN

i¼1 PiRðkÞ � PRðkÞ

� �2r ð2Þ

RðkÞ ¼ 1� rðkÞ2

ð3Þ

where L and R represent left and right FL/MI, respectively, N denotes the number of trials, and PsðkÞ stands for the mean pro-file in state s from the training data set. R(k) with different time k can form a 1D profile with respect to time, but that wouldcontain different characteristics. The global peak in the profile R(k) implies that there is a maximal difference between leftand right FL/MI in the time-scale domain, indicating that the left and right FL/MI are best discriminated at this particulartime and scale range. The R(k) is used to select the most discriminative area. A segment of length 1–2 s is then selected, withits center being the peak after the C3 and C4 channels are concatenated.

4.3.2. Feature extractionPrior to classification, feature extraction is performed on the selected 1–2 s discriminative area rather than directly clas-

sifying the original EEG data without feature extraction. Feature extraction greatly affects the results of classification accu-racy, so better extracted features, can provide higher classification accuracy. In the present study, we first band-pass filteredthe discriminative areas to the wide frequency range containing all l and b rhythmic components using a Butterworth band-pass filter, and then performed the wavelet-fractal operation on the filtered discriminative areas.

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 185

Multiresolution analysis decomposes a signal into numerous details at various resolutions, where each resolution repre-sents a class of distinct physical characteristics within the signal. In other words, a signal is characterized with the formu-lation by decomposing it into sub-bands, and each sub-band can be treated individually based on its characteristics.Multiresolution representation of the filtered discriminative area is achieved by DWT. The discriminative area A for each trialis represented in terms of the DWT as

AðxÞ ¼X1

k¼�1SJðkÞ2J=2/ð2Jx� kÞ þ

XJ

j¼1

X1k¼�1

DjðkÞ2j=2wð2jx� kÞ ð4Þ

where the expansion coefficients are determined by

SJðkÞ ¼ hAðxÞ;2J=2/ð2Jx� kÞi;DjðkÞ ¼ hAðxÞ;2j=2wð2jx� kÞi ð5Þ

where SJ(k) and DJ(k) represent the approximation and detail spaces of A, respectively, and 2J=2/ð2Jx� kÞ and 2j=2wð2jx� kÞdenote the dilated and translated versions of the scaling function /ðxÞ and wavelet function wðxÞ, respectively. The discrim-inative area A is then decomposed into individual sub-bands SJ, DJ, . . ., and D1.

Fractal geometry (Mandelbrot, 1982) provides a proper mathematical model to describe complex shapes that exist in nat-ure with fractal features. The fractal dimension (FD) is one of the most commonly used fractal features, and we chose to usethe FD because it is relatively insensitive to signal scaling and shows a strong correlation with the human judgment of sur-face roughness (Pentland, 1984).

The differential box counting (DBC) method, which covers a wide dynamic range with a low computational complexity, isfrequently used. However, there are two significant faults in the original DBC method, which make its estimation of the frac-tal dimension inaccurate. The first author of the present study previously proposed a solution, referred to as the modifiedfractal dimension (Hsu, 2013), to resolve these issues. First, the original DBC method is based on the difference betweenthe minimum and maximum rectangle numbers, and is easily disturbed by noise. Since the standard deviation of the ampli-tude represents the dispersion of the signal, the present study uses the standard deviation to replace the difference in rect-angle numbers in the original DBC method. Second, DBC produces a stair-like function, which may result in underestimationof the fractal dimension. The present study overcomes this fault by means of the floating calculation of fractal dimension.

The WFFs are formed by concatenating fractal features at different scales, calculated from the discriminative area itselfand all of its sub-bands using the modified fractal dimension. The WFFs reflect the roughness of EEG data at multiresolution.

4.3.3. Fuzzy Hopfield neural network classificationFCM (Dunn, 1973) is a method of clustering, which discriminates the data into two or more classes by minimizing an

objective function in the least squared error sense (Bezdek, 1981). For class number cðc P 2Þ, sample number n and fuzzifi-cation parameter ðm1 6 m <1Þ, the algorithm chooses ui : X ! ½0;1� so that

Piui ¼ 1 and wj 2 Rd for j = 1, 2, . . ., c to min-

imize the objective function

JFCM ¼12

Xc

j¼1

Xn

i¼1

ðui;jÞm xi �wj

�� ��2 ð6Þ

where ui.j is the value of jth membership grade on ith sample xi. The cluster centroids w1, . . ., wj, . . ., wc can be regarded asprototypes for the clusters represented by the membership grades. For the purpose of minimizing the objective function, thecluster centroids and membership grades are chosen so that a high degree of membership occurs for samples close to thecorresponding cluster centroids.

The HNN is a form of recurrent artificial neural network with simple architecture and parallel potential. The present studycombines discrete HNN with FCM, namely FHNN, for feature classification. That is, the FHNN is used to classify the WFFsextracted from EEG data, and recognize complicated brain mental tasks, such as left and right FL/MI. For n training samplesand c classes, the FHNN consists of n � c neurons, which can be conceived as a two-dimensional array. Each sample is iter-atively trained to update the neurons’ weights by using the nearest neighbor rule.

Let ui,j be the fuzzy state of the (i, j)th neuron and Wi,j;k,j represents the interconnected weight between neuron (i, j) andneuron (k, j). A neuron (i, j) in the network receives weighted inputs Wi,j;k,j from each neuron (k, j) and a bias Ii,j from outside.The total input to neuron (i, j) is calculated as

Neti;j ¼ xi �Xn

k¼1

Wi;j;k;jðuk;jÞm�����

�����2

þ Ii;j ð7Þ

The modified Lyapunov energy function of the two-dimensional Hopfield neural network using FHNN strategy is given by

E ¼ 12

Xn

i¼1

Xc

j¼1

ðui;jÞm xi �Xn

k¼1

Wi;j;k;jðuk;jÞm�����

�����2

�Xn

i¼1

Xc

j¼1

Ii;jðui;jÞm ð8Þ

wherePn

k¼1Wi;j;k;j is the total weighted input received from neuron (k, j) in row j, ui,j is the output state at neuron (i, j), and mis the fuzzification parameter. Each column of this modified Hopfield network represents a class and each row represents a

186 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

training sample. The network reaches a stable state when the modified Lyapunov energy function is minimal. For example, aneuron (i, j) in a maximum membership state indicates that the training sample xi belongs to class j.

In order to generate an adequate classification with constraints, the objective function is defined in the following,

E ¼ A2

Xn

i¼1

Xc

j¼1

ðui;jÞmkxi �Xn

k¼1

1Pnh¼1ðuh;jÞm

xkðuk;jÞmk2 þ B2kXn

i¼1

Xc

j¼1

ui;j

!� nk2 ð9Þ

The first term in Eq. (9) is the within-class scatter energy that is the average distance between training samples to thecluster centroid over c clusters. The second term guarantees that the n training samples in X can only be distributed amongthese c classes. More specifically, the second term imposes constraints on the objective function and the first term minimizesthe intra-class Euclidean distance from training vectors to the cluster centroid in any given cluster.

Moreover, the quality of the classification results is very sensitive to the weighting factors A and B. Searching foroptimal values for these weighting factors are time-consuming and laborious. The FHNN can alleviate this problem sothat the constraint terms can be handled more efficiently. In FHNN, all the neurons in the same row compete withone another to determine which neuron has the maximum membership value belonging to class j. The summation ofthe membership grade of states in the same row equals 1, and the total membership states in all n rows equal n. Itis also ensured that all training vectors will be classified into these c classes. The FHNN enables the scatter energyfunction to converge rapidly to a minimum value. By using the FHNN strategy, the scatter energy of the FHNN canbe simplified as

E ¼ 12

Xn

i¼1

Xc

j¼1

ðui;jÞmkxi �Xn

k¼1

1Pnh¼1ðuh;jÞm

xkðuk;jÞmk2 ð10Þ

The minimization of energy E in Eq. (10) is greatly simplified since it contains only one term and hence the need to deter-mine the weighting factors A and B vanishes. Comparing Eq. (10) with the modified Lyapunov function of Eq. (8), the synapticinterconnection weights and the bias input can be obtained as

Wi;j;k;j ¼1Pn

h¼1ðuh;jÞmxk ð11Þ

and

Ii;j ¼ 0 ð12Þ

By introducing Eqs. (11) and (12) into Eq. (7), the input to neuron (i, j) can be expressed as

Neti;j ¼ xi �Xn

k¼1

1Pnh¼1ðuh;jÞm

xkðuk;jÞm�����

�����2

ð13Þ

Accordingly, the state (i.e., membership function) of the neuron at the (i, j)th row is given as

ui;j ¼Xc

‘¼1

Neti;j

Neti;‘

� �2=m� 1

" #�1

; for each j: ð14Þ

Using Eqs. (13) and (14), the FHNN can classify training samples into c classes in a parallel manner, through the followingsteps:

Step 1. Input a set of training samples X = {x1, x2, . . ., xn}, constant v (v > 0), fuzzification parameter m ð1 6 m <1Þ, thenumber of class c, and initialize the states for all neurons U = [ui,j] (membership matrix).

Step 2. Compute weighted matrix using Eq. (11).Step 3. Calculate the input to each neuron (i, j) by Eq. (13).Step 4. Apply Eq. (14) to update the neurons’ membership values in a synchronous manner.Step 5. Compute D ¼maxðjUðtþ1Þ � UðtÞjÞ. If D > e than go to Step 2; otherwise go to Step 6.Step 6. Find the cluster for the final membership matrix.

4.4. Data analysis

Analysis of variance (ANOVA) and multiple comparison tests are conducted on the dependent measure. All calculationswere made with the Statistical Products & Service Solutions (SPSS 13.0) software package.

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 187

5. Results and discussion

5.1. Analyses of discriminative area selection

Event-related area is reduced to 1–2 s areas, wherein it is the most discriminative, to make the results compute more effi-ciently and classify more accurately. In the present study, the CWT and PPMCC are proposed to achieve the aim.

To verify the efficiency of the proposed DAS, we compare the classification accuracy between without/with DAS on thethree datasets, with WFFs and FHNN used as features and a classifier, respectively. The features are extracted from the5 s event-related area for the case without DAS. In addition, discriminative areas in different time lengths, including 1,1.5 and 2 s, are further selected to obtain the appropriate length of time for each subject.

5.1.1. Results of discriminative area selection for the first datasetComparisons of classification accuracy for the first dataset are illustrated in Fig. 5. The results indicate that the average

classification accuracy without DAS is 76.2%, while that increases to 85.6, 84.3, and 84.6% when 1, 1.5, and 2 s DAS is per-formed, respectively. In addition, the classification accuracy of DAS with various lengths of time are all significantly betterthan without DAS for the four subjects. The FL EEG signals with 1 s DAS give superior results for all subjects, which improvethe classification accuracy by at least 8.1%.

The analysis indicates that the differences between with and without DAS are significant (p < 0.01). In addition, the testsalso indicate that there are insignificant differences among DASs with various time length (p > 0.05). Accordingly, the pro-posed DAS method can significantly improve the performance of FL EEG signal classification.

5.1.2. Results of discriminative area selection for the second datasetFig. 6 shows the classification accuracy for the second dataset, which is public from the BCI (2003) competition. Results

reveal that average classification accuracy is 71.4% for without DAS, whereas that increases to 83.1, 82.0, and 82.1% when 1,1.5, and 2 s DAS are performed, respectively. In addition, three kinds of DASs, including the lengths of 1, 1.5 and 2 s, are allsuperior to the case without DAS. After using DAS, imagery hand movement data are more easily discriminated for all sub-jects, which enhances the classification accuracy by more than 10.6%.

The analysis indicate that there are significant differences between with and without DAS (p < 0.01). In addition, the testsalso demonstrate that classification accuracy difference among DASs with various lengths of time are insignificant (p > 0.05).Hence, the classification accuracy of EEG signals associated with imagined hand movement data can be greatly improved bymeans of DAS.

5.1.3. Results of discriminative area selection for the third datasetFig. 7 shows the classification accuracy for the third dataset, which is public from the BCI (2005) competition. Results

show the average classification accuracy for without DAS is 71.6%, while that increases to 81.3, 79.6, and 79.8% if 1, 1.5,and 2 s DAS is performed, respectively. In addition, for all of the selected lengths of time, using DAS obtains better resultsthan without using DAS. The MI EEG signals with DAS achieve superior results for all subjects, with the classification accu-racy increasing by at least 8.0%.

The analysis indicates that the differences between without and with DAS are significant (p < 0.01). In addition, the testsalso denote that there are insignificant differences among DASs with various lengths of time (p > 0.05). This demonstratesthat the proposed area-selection approach can enhance the classification accuracy of MI EEG signals.

Fig. 5. Comparison of classification accuracy in discriminative area selection for the first dataset.

Fig. 6. Comparison of classification accuracy in discriminative area selection for the second dataset.

Fig. 7. Comparison of classification accuracy in discriminative area selection for the third dataset.

188 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

5.1.4. Summary of discriminative area selectionThe classification accuracy for the three datasets all indicated that results with DAS are significantly better than those

without DAS. This result might be due to decreasing the length of time for event-related area from 5–7 s to 1–2 s, whichcan enhance the computation speed and thus improve the classification accuracy of the EEG associated with FL or MI signals.Furthermore, the event-related area with 1 s DAS resulted in the best classification accuracy for the three datasets. Thisresult also indicated that the proposed algorithms in present study have high efficiency and validity.

5.2. Analyses of classifiers

FHNN is used for the classification of single-trial left and right FL/MI EEG signals. It is implemented to recognize left andright FL/MI data without any advance training. As FHNN is an unsupervised approach that partitions a collection of featurevectors into a number of subgroups based on minimizing the trace of a within-cluster scatter matrix, it is adopted to auto-matically classify WFFs into two clusters, left and right FL/MI. Moreover, the EEG data are non-stationary and their inherentcharacteristics vary with time. The classification of FL/MI EEG data using an unsupervised FHNN algorithm may result in abetter generalization performance than conventional supervised classifiers.

In addition, two widely-used classifiers, ANN and HNN, are implemented for comparison with FHNN. The first classifier isan ANN with one hidden layer and sigmoid activation functions. The optimal values for the weights and biases of ANNclassifier were estimated by applying the ten-fold cross validation technique on training and validation data. The secondclassifier is HNN, which is a recurrent neural network, synaptic connection pattern, whereby there is a Lyapunov functionfor the activity dynamics (Hopfield, 1982). In the original HNN model, the state of the system evolves from any initial stateto a final state in which there is a minimum of the Lyapunov function.

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 189

5.2.1. Results of classifiers for the first datasetFig. 8 compares the classification accuracy of the first dataset for each of the ANN, HNN, and FHNN with 1 s DAS. The aver-

age classification accuracy of the ANN, HNN, and FHNN are 81.2, 81.0, and 85.6%, respectively. Thus the classification accu-racy of FHNN is higher than the other two approaches for all subjects.

The analysis shows that the FHNN is significantly better than ANN and HNN (p < 0.01) for the classification of FL EEG sig-nals, whereas the differences between ANN and HNN are insignificant (p > 0.05).

5.2.2. Results of classifiers for the second datasetFig. 9 illustrates the classification accuracy comparisons of the second dataset among ANN, HNN and FHNN with 1 s DAS.

The average classification accuracy of the ANN, HNN, and FHNN are 80.0, 80.1, and 83.1%, respectively. FHNN leads to thebest average classification accuracy in this dataset, while the worst average results are obtained by using ANN, whereinANN is just slightly less than HNN by 0.1%. Furthermore, FHNN is superior to the other two approaches for all subjects inthe discrimination of imagery hand movement data.

The analysis demonstrates that the FHNN is significantly better than ANN and HNN (p < 0.01) for the classification of MIEEG signals, whereas the differences between ANN and HNN are insignificant (p > 0.05).

5.2.3. Results of classifiers for the third datasetComparisons of the classification accuracy for the third dataset are provided to assess the performance of classifiers with

1 s DAS, as illustrated in Fig. 10. The average classification accuracy of the ANN, HNN, and FHNN are 77.6, 77.5, and 81.3%,

Fig. 8. Comparison of classification accuracy in classifiers for the first dataset.

Fig. 9. Comparison of classification accuracy in classifiers for the second dataset.

Fig. 10. Comparison of classification accuracy in classifiers for the third dataset.

190 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

respectively. FHNN and HNN obtain the best and worst average classification accuracy, respectively, and the differencebetween them is 3.8%. Moreover, the FHNN performs better than the two others when classifying MI data for all subjects.

The analysis also indicates that the FHNN is significantly better than either the ANN or the HNN (p < 0.01) for the clas-sification of MI EEG signals, whereas the differences between ANN and HNN are insignificant (p > 0.05).

5.2.4. Effects of classifiersThe classification accuracy for the three datasets all indicated that FHNN are significantly better than ANN and HNN. The

average classification accuracies of the three datasets are 85.6, 83.1, and 81.3%, respectively. The average classification accu-racies are also greater than many previous studies (Furdea et al., 2012; Massari et al., 2013). In order to find a suitable clas-sifier for EEG data classification, Furdea et al. (2012) compared four different classifiers: step-wise linear discriminantanalysis (SWLDA), shrinkage linear discriminant analysis (SLDA), support vector machine with linear kernel (SVM-LIN),and radial basis function kernel support vector machine (RBF-SVM). Results showed the RBF-SVM had the highest single-trialclassification accuracy, of 68.8%. Massari et al. (2013) investigated three different classifiers (SWLDA; SVM-LIN; and Gaussiankernel) in order to differentiate between two cortical responses (participant code and session number) by means of EEGrecordings, and their average classification accuracy were all about less than 70%.

Several reasons might be offered to explain the results. First, if FHNN is used as a classifier, it is not necessary to label datain advance before training, i.e., we can discriminate the data without a priori knowledge (Halder et al., 2011). Second, FHNNis capable of making flexible partitions of a finite data set (Hossen et al., 2011). Third, the FHNN acts in combination with theFCM and HNN, so FHNN inherently has the advantages of FCM and HNN. Finally, FHNN is a robust approach suitable for theclassification of non-stationary biomedical signals, such as single-trial EEG data (Chen et al., 2013; Hillar et al., 2013).Accordingly, the classification of FL/MI EEG data with an unsupervised FHNN may lead to a better classification accuracythan can be obtained with conventional supervised classifiers (Hsu, 2012).

Furthermore, FHNN resulted in best classification accuracy for the three datasets with all subjects. This result also indi-cated that the proposed scheme in present study is very efficiency and robust.

6. Conclusion

The present study proposes a novel BCI scheme for single-trial classification for the next frontier of telemedicine in HCI. Inthis scheme, FHNN associated with DAS and WFFs are used for the implementation of BCI works. DAS effectively detects thelocation of discriminative area by taking the time–frequency information into account. This substantially reduces the origi-nal 5 s event-related area to that of length 1 s, which can also speed up feature extraction. Wavelet-fractal features are thenextracted from the DWT data by means of the proposed modified fractal dimension, which is beneficial for the recognition ofmental tasks. Finally, FHNN is used for classification. The proposed BCI system is robust and capable of processing non-sta-tionary biomedical signals, which is promising in the applications of BCI works. It indicates that the system has a substantialpotential for telemedicine in HCI applications.

6.1. Implications for clinical applications and benefits

Clinical applications and benefits of the proposed system are to provide an alternative channel for healthy and disabledusers as an assistive technology. This system can perform real-time processing of motor functions in healthy and disabled

W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192 191

subjects, such as reproducing the motor activities with a cursor or robotic arm (Wessberg et al., 2000). The technology can befurther expanded to neuro-motor prostheses use for humans with motor disabilities. The proposed EEG-based noninvasiveBCI system can be also investigated for various assistive applications, such as communication, which is one of the recentlypopular applications. One example is the applications of spelling train users to respond to symbols on a screen by focusing onspecific symbols. The technology can also be applied to the auditory domain, which is relevant to subjects with a progressiveneurodegenerative disease affecting vision in its later stages. In addition to communication applications, the proposed sys-tem is also suitable for prototypes of wheelchair control or 3D cursor control.

In addition, the use of the proposed BCI system for rehabilitation of gait after stroke is feasible. It detects imagined move-ment and then artificially moves users’ limbs by means of functional electrical stimulation of the muscles. Several studieshave illustrated a wide variety of potential applications in the treatment of pain, depression, schizophrenia, emotional dis-orders, and memory (Zotev et al., 2011).

6.2. Implications for game and entertainment applications and benefits

In addition to clinical applications, the proposed system can be also developed for the applications of games and enter-tainment (Nijholt et al., 2009). Since there is large number of potential users of such applications, there will likely be a sig-nificant increase in research and development resources. For example, the BCI painting applications indicated that evenentertainment-related BCI works may have a certain degree of clinical relevance in improving the users’ social and expres-sive potential.

6.3. Implications for ethical and social issues

In the future, the proposed system, as well as other BCI works, has the potential to influence not only individual subjectsbut also ethics and society (Vlek et al., 2012). Applications of future BCI research and development in BCI computer games,neuroprostheses, neuro-marketing, etc., inevitably raise ethical and social concerns, and public debates on appropriate rightsand restrictions of BCI will be expected. The results of ethical and social debates would substantially affect public acceptanceof BCI systems and related technologies.

Neuro-ethical debates have identified several important topics for BCI research and development (Tamburrini, 2009).Some of the ethical issues are relative to BCI research. They concern the differences between research and treatment inBCI works. Ethical issues may center on whether the goal is to apply approved technologies for treatment, or whether thegoal strictly involves the development of technologies. If the interests conflict, important ethical concerns will arise.

Acknowledgements

The author would like to express his sincere appreciation for grants partially from NSC102-2633-E-194-002 andMOST103-2410-H-194-070-MY2, Ministry of Science and Technology, Taiwan.

References

Aler, R., Galván, I.M., Valls, J.M., 2012. Applying evolution strategies to preprocessing EEG signals for brain–computer interfaces. Inf. Sci. 215, 53–66.Bezdek, J.C., 1981. Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York.Burke, D.P., Kelly, S.P., de Chazal, P., Reilly, R.B., Finucane, C., 2005. A parametric feature extraction and classification strategy for brain–computer

interfacing. IEEE Trans. Neural Sys. Rehabil. Eng. 13, 12–17.Channel, G., Kierkels, J.J.M., Soleymani, M., Pun, T., 2009. Short-term emotion assessment in a recall paradigm. Int. J. Hum.–Comp. Stud. 67, 607–627.Chen, W., Huang, L., Guo, Z., 2013. Finite time stability of periodic solution for Hopfield neural networks with discontinuous activations. Neurocomputing

103, 43–49.Doukas, C., Metsis, V., Becker, E., Le, Z., Makedon, F., Maglogiannis, I., 2011. Digital cities of the future: extending @home assistive technologies for the

elderly and the disabled. Telem. Inform. 28 (3), 176–190.Dunn, J.C., 1973. A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters. J. Cyber. 3, 32–57.Furdea, A., Ruf, C.A., Halder, S., De Massari, D., Bogdan, M., Rosenstiel, W., Matuz, T., Birbaumer, N., 2012. A new (semantic) reflexive brain–computer

interface: in search for a suitable classifier. J. Neurosci. Methods 203, 233–240.Graz Data Set and description for the BCI 2003 competition. [Online]. Available: http://www.bbci.de/competition/ii/.Graz Data Sets and description for the BCI 2005 competition. [Online]. Available: http://www.bbci.de/competition/iii/.Halder, A., Pramanik, S., Kar, A., 2011. Dynamic image segmentation using fuzzy C-means based genetic algorithm. Int. J. Comp. Appl. 28 (6), 15–20.Hillar, C., Sohl-Dickstein, J., Kilian, Koepsell K., 2013. Novel local learning rule for neural adaptation fits Hopfield memory networks efficiently and optimally.

BMC Neurosci. 14 (1), 215.Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. 79, 2554–2558.Hossen, J., Rahman, A., Sayeed, S., Samsuddin, K., Rokhani, F., 2011. A modified hybrid fuzzy clustering algorithm for data partitions. J. Appl. Sci. Res. 7 (8),

674.Hsu, W.Y., 2012. Fuzzy Hopfield neural network clustering for single-trial motor imagery EEG classification. Exp. Syst. Appl. 39, 1055–1061.Hsu, W.Y., 2013. Embedded prediction in feature extraction: application to single-trial EEG discrimination. Clin. EEG Neurosci. 44, 31–38.Hsu, W.Y., Sun, Y.N., 2009. EEG-based motor imagery analysis using weighted wavelet transform features. J. Neurosci. Methods 167, 310–318.Mandelbrot, B.B., 1982. Fractal Geometry of Nature. Freeman Press, San Francisco.Massari, D..De., Matuz, T., Furdea, A., Ruf, C.A., Halder, S., Birbaumer, N., 2013. Brain–computer interface and semantic classical conditioning if

communication in paralysis. Biol. Psychol. 92, 267–274.Nijholt, A., Bos, D.P.O., Reuderink, B., 2009. Turning shortcomings into challenges: Brain–computer interfaces for games. Entertainment Computing 1 (2),

85–94.

192 W.-Y. Hsu / Telematics and Informatics 32 (2015) 180–192

Obermaier, B., Neuper, C., Guger, C., Pfurtscheller, G., 2001. Information transfer rate in a five-classes brain–computer interface. IEEE Trans. Neural Sys.Rehabil. Eng. 9, 283–288.

Pentland, A.P., 1984. Fractal based description of natural scenes. IEEE Trans. Pattern Anal. Mach. Intell. 6, 661–674.Saprikis, V., 2013. Suppliers’ behavior on the post-adoption stage of business-to-business e-reverse auctions: an empirical study. Telem. Inform. 30 (2), 132–

143.Sim, J.J., Tan, W.H., Wong, C.J., Ooi, K.B., Hew, T.S., 2014. Understanding and predicting the motivators of mobile music acceptance – a multi-stage MRA-

artificial neural network approach. Telem. Inform. 31 (4), 569–584.Tamburrini, G., 2009. Brain to computer communication: ethical perspectives on interaction models. Neuroethics 2, 137–149.Telfer, B.A., Szu, H.H., Kiang, R.K., 1993. Classifying multispectral data by neural networks. Telem. Inform. 10 (3), 209–222.Vlek, R.J., Steines, D., Szibbo, D., Kübler, A., Schneider, M.J., Haselager, P., Nijboer, F., 2012. Ethical issues in brain–computer interface research, development,

and dissemination. J. Neurol. Phys. Ther. 36 (2), 94–99.Wessberg, J., Stambaugh, C.R., Kralik, J.D., Beck, P.D., Laubach, M., Chapin, J.K., Kim, J., Biggs, S.J., Srinivasan, M.A., Nicolelis, M.A., 2000. Real-time prediction

of hand trajectory by ensembles of cortical neurons in primates. Nature 408 (6810), 361–365.Williamson, J., Murray-Smith, R., Blankertz, B., Krauledat, M., Müller, K.R., 2009. Designing for uncertain, asymmetric control: Interaction design for brain–

computer interfaces. Int. J. Hum–Comp. Stud. 67, 827–841.Zotev, V., Krueger, F., Phillips, R., Alvarez, R.P., Simmons, W.K., Bellgowan, P., Bodurka, J., 2011. Self-regulation of amygdala activation using real-time fMRI

neurofeedback. PLoS ONE 6 (9), e24522.