Embed Size (px)
Citation preview
Touchless Interaction with a Smartphone: EMG-BasedSleeve for Cyclists
Cristina MatonteIT University of Copenhagen
Tomas MiseikisIT University of Copenhagen
Kyra TijhuisIT University of Copenhagen
ABSTRACTIn this paper we present the development of a system that al-lows cyclists to control the music player on their smartphonein a touchless manner. A custom built electromyography(EMG) based sleeve is used to recognize 6 hand gestures thatcorrespond to specific commands as input to the music player.The sleeve uses two channels to measure electrical potentialgenerated in the Extensor Digitorum and Flexor Carpi Ulnarismuscles when performing gestures. To increase the accuracyof classifier a number of features were used, including Min-imum value, Mean, Median, Modified Mean Absolute Value2, Zero Crossing, and Successive Difference Sum. The per-formance of the system was evaluated using 6 subjects, whowere using exercise bike while performing the various ges-tures. Overall, the highest accuracy of 68.00% was achievedusing Neural Network classifier with a Low-Pass Filter fordata smoothing.
Author KeywordsElectromyography (EMG) Sensors; Touchless Interaction;Machine Learning; Signal Processing; Gesture Recognition.
ACM Classification KeywordsH.5.2. Information Interfaces and Presentation (e.g. HCI):User Interfaces
INTRODUCTIONTraditionally we interact with computing devices in a touch-based manner. Computers are supplemented with mice, key-boards, and touch-sensitive surfaces, which all require phys-ical contact to use them. This heavily limits the way we in-teract with technology. Not only do we need to use our handsfor a prolonged period of time, but our eyes must also focuson the computing device itself. However, in some situationspeople might be unable to use their devices this way. For ex-ample, people who are driving, cycling, or in general “on themove”, might not be able to focus on their devices for a longperiod of time. Furthermore, people with various disabilitiesand injuries that affect their fingers might be unable to usethese devices at all.
Paper submitted for evaluation in the Pervasive Computing Course, Autumn 2014. TheIT University of Copenhagen. Copyright remains with the authors.
Existing TechnologiesOne way to tackle the problems described above is to relyon touchless interaction. The touchless interaction paradigmaims at allowing users to interact with technology throughtouchless interaction and body movements [20]. There aremany different ways in which this can be achieved. A pop-ular approach is eye tracking, where eye movement is mon-itored and used as an input mechanism to a computing de-vice [5]. Using human voice is another way of touchlesslyinteracting with a computing device, where different featuresof voice, such as pitch and volume, can be used as an in-put [15, 22]. Using wireless technology approaches, it is pos-sible to extract gesture information from ambient RF signals(e.g., TV transmission) and use it to control your device [17].Proximity sensors [3] and ultrasound chips [21] provide a ba-sic way to touchless interaction, where users can move theirhands towards and away from the device to execute a specificcommand. Approaches based on cameras use image process-ing techniques to track movements and gestures, and translatethem to specific instructions as input to a device [25, 23, 31].Finally, various body sensors can be utilized to track arm,feet, or head movements, and these can be used as input [4,42, 10]. One of the devices that uses these sensors is theMyo, a bracelet that is worn around the thickest part of theforearm. The Myo has several EMG sensors, a gyroscope,an accelerometer and a magnetometer. These sensors enablethe Myo to detect both hand gestures and the position of theforearm [36].
LimitationsThe problem with these approaches is that most of themstill require physical contact with the device, as well as eyefocus. For example, if we consider a smartphone as ourmain device, then the mentioned proximity, ultrasound, andcamera-based techniques would require users to hold theirsmartphone in one hand and perform a gesture with theother hand. Eye tracking approaches present an even morechallenging situation, since the device must be held close tothe face, and the users eyes must focus only on the device.Voice-based techniques are not a very good solution if thedevice will be used in public places, since it might presenta privacy risk, as well as disturb other people. For thesereasons, in this project we are focusing on body sensors,and more specifically sensors that measure hand movements.Since hands are one of the most flexible parts of our bodieswe can consider a large variety of different gestures.
1
EMG SensorsOne of the approaches in recognizing hand gestures is to useelectromyography (EMG) sensors. EMG sensors measureelectrical potential that is generated in a muscle during itscontraction. Researchers who seek to utilize the EMG tech-nology have been mostly focused on the medical field. Ma-jority of the applications focus on serving the needs of peoplewith injuries or various disabilities, since EMG can help in di-agnosing muscular disorders [14], controlling prosthetics [18,30], and the measuring of mental stress [40].
The Need for Touchless InteractionAlthough a lot of research has been conducted and applica-tions have been developed based on EMG, there are still manyareas of our life that could benefit from this technology. Theidea of using EMG technology for enabling touchless interac-tion provides a perfect foundation for applications that allowusers to use their computing devices, even if their hands areotherwise occupied. This is a quite common scenario for peo-ple who drive or cycle to school or work, and at the same timetry to use their smartphone. Naturally, this can be dangerous,since using a smartphone while cycling is extremely distract-ing. Furthermore, cyclists have less control of their bicyclewhen they only have one hand on the handlebars for a pro-longed period of time. According to the Cycling Embassy ofDenmark, each year 17.500 cyclists are treated at the hospi-tal for cycle-related injuries [7]. A number of these injuriesare due to distractions while accepting a phone call or try-ing to change a song on the music player. In a report the U.S.Department of Transportation presents the results of an exten-sive research on “The Impact of Hand-Held And Hands-FreeCell Phone Use on Driving Performance and Safety-CriticalEvent Risk” [11]. Their research shows that the odds ratioof having an accident while trying to locate or answer yourphone is very high, namely 3.65. Furthermore, looking atthe phone while driving increases your chance of an accidentby 1.73. In contrast, using hands-free equipment reduces thechances of getting into an accident to 0.79. Therefore we canconclude that, while driving or cycling, it is not the using ofthe smartphone that is most dangerous. It is the looking at orholding of a smartphone that increases the risk of an accidentsignificantly.
Our SolutionIn this work we present an EMG-based arm sleeve prototypethat supports touchless interaction with a smartphone whilethe user is cycling. The sleeve has 6 electrode connectorsthat target muscles in both the anterior and the posterior com-partment of the forearm. These muscles are monitored bytwo EMG sensors, one for each compartment of the forearm.With this sleeve the electrical activity of muscles can be mea-sured in real time, and the data is then sent via Bluetooth tothe smartphone, where the recognition of the hand gesturesis performed. Using this sleeve we are able to recognize fivehand gestures and one activation gesture. Our chosen sce-nario deals with controlling a music player on a smartphonewhile cycling, so each different gesture corresponds to a spe-cific command, namely play/pause song, switch to next song,switch to previous song, volume up, and volume down. The
activation gesture is there to prevent unintentional gesturesbeing acted upon by the system.
Firstly, we present a short overview on other work that is insome way related to our solution. In later parts, the details ofthe developed system are provided, followed by an evaluationand a discussion of the system.
RELATED WORKIn this section we present related work that involves a shortbackground on electromyography (EMG), and discussions onelectrodes, classification, and pattern recognition.
ElectromyographyElectromyography measures muscle activity as electrical po-tential between a ground electrode and a sensor electrode.It is possible to recognize diverse gestures, since differentgestures produce distinct EMG signals. EMG can measuresignals either inside the muscle (invasive EMG) or from theskin (surface EMG). For more detailed information on elec-tromyography, see Merletti et al. [19]. Invasive EMG is notusable in this case, since it requires the electrodes to be in-serted into the muscle. Surface EMG, even though less accu-rate, is better suited for our project.
Electrodes and Their PlacementThere are mainly two types of surface electrodes used whenmeasuring EMG signals, namely wet and dry. Wet electrodesin general provide a better signal-to-noise ratio and accu-racy [26] than dry electrodes. This is due to the fact thatwet electrodes are sticky and ensure better contact with theskin. However, there are studies showing that dry electrodescan perform as good as [12] and sometimes even better thanwet electrodes [28]. In aforementioned paper a quantitativecomparison is presented, which shows that interference expe-rienced by dry electrodes can be 40dB less than that experi-enced by wet electrodes.
One of the disadvantages of wet electrodes is that they mayinduce some discomfort when applying to and removing fromthe skin. Furthermore, traditional wet electrodes are supposedto be used only once, so after every use they are meant to bethrown away and replaced with a new set. The main issuewith dry electrodes is conductivity problems. The impedancebetween a dry electrode and the skin can be affected dueto hairs lifting the sensor or differing moisture levels in theskin [39]. Furthermore, it is harder to keep dry electrodes in afixed position, especially when on the move. The electrodeswould need to be used together with some kind of stretchableand compressive fabric that would help to keep them in thesame place.
In general, it is challenging to apply electrodes on the exactsame place on the body, therefore the position of the elec-trodes may vary greatly from one day to the next. If the elec-trode placement differs a lot from the initial placement (whenthe classifier model was built), this can affect the accuracy ofgesture recognition. There have been experiments [39] thatmeasures the effects of minor (1-3 mm) and major (1-2 cm)variations in the placement of electrodes. Findings showed
2
that minor variations had no impact; however, major displace-ments required re-training of the recognition models.
The electrodes themselves can be made from different typesof material. Silver and copper are commonly used [9]. Adifferent possibility is using conductive fabric, which is madeby plating nylon fabric with silver [34].
Besides the mentioned problems with wet and dry electrodesthere are other factors that may affect the quality of the EMGsignal. Individual physiological differences in people, suchas differences in arm lengths and widths, can make it chal-lenging to put electrodes at the proper positions.
Concluding, dry electrodes seem to be most fitting in ourcase, since we want to be able to use the electrode multi-ple times. To ensure conductivity between the electrodes andskin, and to make sure that the electrode placement does notdiffer too much between usages, we use a compressive sleeveto which we attach the electrodes. When we combine conduc-tive fabric with a compressive sleeve [2] the resulting devicesatisfies all our requirements.
Classification and Pattern RecognitionPrevious work on classifying patterns based on EMG inputhas been done using different machine learning techniquessuch as Neural Network and Support Vector Machine.
An experiment using a Selective Desensitization Neural Net-work (SDNN) [16] tries to overcomes some limitations of themultilayer perceptron and is able to approximate some func-tions by using only a few training data samples. In this exper-iment, test subjects were asked to execute each of the avail-able gestures for 2 seconds, then repeat this nine times. After-wards a cross validation was performed over the total numberof data samples, in order to calculate the classification rate.The total average classification rate over the subjects and theavailable gestures was 95.26%.
A different experiment using Support Vector Machine [27]has set a 3-phases laboratory experiment: Hands-free Fin-ger Gestures, Hands Busy Finger Gestures and Controlling aPortable Music Player Application. The first two phases wereused for training and testing of the predictive model. Thethird phase used this training data for real-time control andclassification. The real-time data-processing consisted of abasic signal processing, the generation of feature vectors andfurther classification using a Sequential Minimal Optimiza-tion of Support Vector Machines. The results of the experi-ment varied from an accuracy of 65% to 91%, depending onwhether the user received some immediate feedback to cor-rect their gestures or not. The best results were achieved onphase 2, when the user got some feedback; phase 3 got anaverage accuracy of 86%.
However, there are also many other classification algorithmsin use [4, 14, 39]. Therefore we will compare different clas-sification algorithms, in order to find out which algorithmworks best for our solution.
THE SYSTEMIn this section we present the design of our system.
Figure 1. System overview
Figure 1 displays a high-level overview of the system. Theuser wears the sleeve that is connected to the Arduino board.The Android phone connects to the Arduino board via Blue-tooth and waits for input data. After the user performs anactivation gesture, the Arduino sends the input data. The An-droid device reads and preprocesses this data, extracts a num-ber of features and runs it through the classifier. Based on theresult of the classification, the device will perform the corre-sponding action on the media player. When training data isobtained the setup varies a little. The input data is sent to acomputer instead of a smartphone, and the activation gestureis not required. Instead the user uses a button on the computerto start a recording.
Wearable Device and ArduinoThe wearable device is made by sewing strips of conductivefabric onto a compressive arm sleeve. The strips of conduc-tive fabric are the electrodes that measure the muscle activ-ity. The compressive arm sleeve ensures that the electrodesmake contact with the skin at all times and also keeps themin place. To connect the electrodes to the sensor we addedsnap buttons to the sleeve. These metallic snap buttons con-nect the electrodes on the inside with the cable leads on theoutside. Figure 2 shows the wearable device with the cablesconnected.
Figure 2. Sleeve with the cable leads connecting the sensor to the elec-trodes
On the inside of the sleeve there are five strips of conductivefabric. In Figure 3 four of the five electrodes can be seen. PerEMG sensor we need 3 electrodes; one electrode on the mid-dle of the muscle, one electrode on the end of the muscle, andone electrode on a bony part of the arm, for grounding [1]. We
3
connect both grounding cables to the same electrode, there-fore we only need 5 instead of 6 strips of conductive fabric.Two of the strips are supposed to target the Extensor Digi-torum muscle in the posterior compartment of the forearm.This muscle is used to extend the fingers (from a curled to astreched position). However, since not all arms are the same,it might occur that other muscles are targeted as well. Themuscle that is targeted by the other EMG sensor is the FlexorCarpi Ulnaris, which flexes and adducts the hand [35]. Again,it might be possible that other muscles are targeted as well.
Figure 3. Sleeve worn inside out. The upper electrodes target the Ex-tensor Digitorum muscle, the lower electrodes target the Flexor CarpiUlnaris
The electrodes have a width (the direction perpendicular tothe muscle fibres) of 4.5 cm and a height of 1 cm. Withthese dimensions they follow the recommendations made bythe SENIAM initiative [8]. To ensure conductivity betweenthe skin and electrode, it is important to wet the electrodesbefore usage, otherwise the readings are not usable. It is alsoimportant to make sure that the cable leads connecting theelectrodes to the sensors are in a fixed position, since move-ment of the cables will increase the signal-to-noise ratio.
Figure 4. Arduino board with all connected components
The rest of the device-side components can be seen in Fig-ure 4. The red, blue and black cables connect the electrodesto the EMG sensors. These sensors then amplify, rectify andsmooth the data, so it can be immediately put through to theArduino RedBoard [33]. The data is sent wirelessly to thesmartphone using a Bluetooth module [32].
GesturesInitially, 16 different gestures were tried out; however, someof them were complex to perform or were not easily recog-nizable by the system. For the purpose of this project a min-imum of five gestures is required, so that each gesture corre-sponds to a particular action in the media player. The original16 gestures were grouped into smaller sets to try out whichcombination provided the best accuracy. Also, a few bikerswere asked about the gestures’ intuitiveness and ease of usewhile cycling. Based on these two sets of information, sixgestures were chosen to be used in the system. Five of themwill be used to control the media player, namely Snap, SideLeft, Gun Right, Wrist Down, and Wrist Up; while the sixthgesture, Fist, is used as an activation gesture. The main goalof having an activation gesture is to avoid interfering with theactivities the user is performing. Fist was chosen as an ac-tivation gesture, since it is easy to perform it for a period oftime. The minimum period of time that the activation gestureneeds to be performed is 150ms. During this period the val-ues need to be above 150mV for the muscle in the posteriorcompartment, and above 50mV for the muscle in the anteriorcompartment.
The selected gestures can be seen in Figure 5.
(a) Fist (b) Snap (c) Side Left
(d) Gun Right (e) Wrist Down (f) Wrist Up
Figure 5. Gestures
The graph in Figure 6 shows that the different gestures havedistinct forms.
Training DataIn order to capture the training data to be used for buildingthe classifier, we invited 7 people to put the sleeve on andperform the gestures illustrated in Figure 5. Each person per-formed every gesture at least 20 times. The recording fre-quency was set to 30 samples per second and each recordingwas performed for 1 second. A snapshot of the recorded datais presented in Figure 7. The first number is the measurementfrom the electrodes on the posterior compartment of the fore-arm, whereas the second number is the measurement from the
4
Figure 6. Graphs of measurements of the six different gestures, withtime (in s) on the x-axis and electrical activity of the muscle (in mV) onthe y-axis. The red line shows the measurements from the electrodes onthe posterior part of the forearm, the blue line from the electrodes onthe anterior part of the forearm.
anterior compartment of the forearm. These numbers showthe electrical potential in mV and can be between 0 and 1024.The third attribute is the timestamp of the measurement innanoseconds. The last attribute describes the gesture and thefirst name letter of the person that is associated with the read-ing.
1 200,28,1417521377251202000,wristUp_C2 304,46,1417521377275042000,wristUp_C3 391,43,1417521377314587000,wristUp_C4 451,45,1417521377358898000,wristUp_C5 473,39,1417521377358944000,wristUp_C
Figure 7. Snapshot of the recorded data
Feature ExtractionFor each recording we generate a set of features. First of all,we convert each recording from a sequential learning probleminto a traditional classification problem by turning the electri-cal potential measured by both sensors into vectors of size30. This is represented as (x1, ..., x30, y1, ..., y30), where x isdata received from the first sensor and y is data received fromthe second sensor. Additionally, we generate 12 more fea-tures, which were chosen based on prior work and researchby people that dealt with EMG sensors and applications [24,6]. The chosen features are Maximum Value, Minimum Value,Median, Mean, Variance, Standard Deviation, Root MeanSquare, Modified Mean Absolute Value 1, Modified Mean Ab-solute Value 2, Wilson Amplitude (WAMP), Zero Crossing,and Successive Difference Sum. These features are generated
for the measurements from both sensors, therefore in the endthe total number of features is 24. The features are describedmore into detail below.
• Maximum Value (MAX). Gives us the highest electrical po-tential read by the sensors in the given recording.
• Minimum Value (MIN). Gives us the lowest electrical po-tential read by the sensors in the given recording.
• Median. Median gives us the middle value of the EMGsignal within a recording and is calculated by sorting thenumbers in increasing order and simply taking the middlenumber.
• Mean (MEAN). The mean is an easy way to see the aver-age muscle contraction levels for different gestures. It isdefined as
MEAN = ~x =1
N
N∑n=1
xn. (1)
• Variance (VAR). The variance allows us to measure thepower of the signal and is calculated as
V AR =1
N − 1
N∑n=1
(xn − ~x)2, (2)
where xn is the nth sample, N is the total number of sam-ples, and ~x is the mean of the samples.
• Standard Deviation (SD). SD allows us to see how spreadout the numbers of the EMG signal are. SD is calculatedby taking the square root of variance
SD =√V AR. (3)
• Root Mean Square (RMS). In the context of EMG, rootmean square is related to the constant force and non-fatiguing contraction [24] and can be expressed as
RMS =
√√√√ 1
N
N∑n=1
x2n. (4)
• Modified Mean Absolute Value 1 (MMAV1). MMAV1 ex-tends calculation of the mean by applying weighting win-dow function wn. It is defined as
MMAV 1 =1
N
N∑n=1
wnxn, (5)
where
wn =
{1, if0.25N ≤ n ≤ 0.75N0.5, otherwise.
• Modified Mean Absolute Value 2 (MMAV2). MMAV2 issimilar to MMAV1, however cotinuous weighting windowfunction is used to improve smooth window. MMAV2 iscalculated as
MMAV 2 =1
N
N∑n=1
wnxn, (6)
5
where
wn =
1, if0.25N ≤ n ≤ 0.75N4nN , if0.25N > n
4(n−N)N , if0.75N < n.
• Wilson Amplitude (WAMP). WAMP calculates the amountof times that the change in EMG signal amplitude betweentwo adjacent readings exceeds the given threshold, 50µVin our case. It is related to the muscle contraction level andis defined as
WAMP =
N−1∑n=1
f(|xn − xn+1|), (7)
where
f(x) =
{1, ifx ≥ threshold0, otherwise.
• Zero Crossing (ZC). ZC is the amount of times the sig-nal passes the zero amplitude axes. In our case, the sig-nal received from EMG sensors is amplified, rectified, andsmoothed, thus ZC can be easily computed as
ZC =
N∑n=1
f(xn), (8)
where
f(x) =
{1, ifx = 00, otherwise.
• Successive Difference Sum (SDS). Calculates the summeddifference between two adjacent EMG signal readings in agiven recording. It is defined as
SDS =
N−1∑n=1
(xn − xn+1). (9)
Building the ClassifierBefore starting to build the classifier, some considerationswere made regarding the features. As it was described in pre-vious section, we use two vectors of size 30 for the data re-ceived from the sensors and we generate 12 extra features forboth sensors. Therefore, if we consider using all the features,the built classifier will end up with 84 attributes. However,using all the features does not necessary produce the highestaccuracy. The complicated part regarding feature extractionis to figure out which combination of features works best andproduces the classifier with the highest accuracy. In order tofind that out, we decided to generate all possible combina-tions of the features and build a classifier for every combina-tion.
For building classifiers we used Weka [38], which containsmachine learning algorithms for data mining tasks. We tested5 different classifiers, namely K*, KNN, Neural Network,C4.5, and Bayes. All the classifiers were trained using 10-fold cross-validation. Furthermore, default Weka parameterswere used to train the classifiers, with exception of NeuralNetwork classifier, where training time was increased from
Classifier AccuracyKNN 97.56%
Neural Network 97.15%K* 96.74%
C4.5 90.02%Bayes 89.00%
Table 1. Built classifiers and their Accuracy
Gesture ActionSnap Play/Pause
Wrist Up Volume UpWrist Down Volume Down
Side Left Previous SongGun Right Next Song
Table 2. The gestures and actions they correspond to on the music player
500 milliseconds to 1 second. After building classifiers forall the possible feature combinations, we found that 6 par-ticular features produce the highest accuracy. These featuresare Minimum Value, Mean, Median, Modified Mean Abso-lute Value 2, Zero Crossing, and Successive Difference Sum.The built classifiers and their accuracy using these 6 featurescan be seen in Table 1. The three most accurate classifiers,namely KNN, Neural Network, and K*, were chosen to beused for evaluation.
Android ApplicationAn Android Application that communicates with the Arduinoboard was developed. It consists of a simple user interfacethat allows users to connect to the Arduino board and can beseen in Figure 8. The application will establish communica-tion with the Arduino board through a Bluetooth socket andthen create an Asynchronous Task that listens for incomingdata. When the user performs the activation gesture, the Ar-duino will send a ’start’ message to the Android device andit will play a short notification sound. The user performs theaction after the notification sound has played and the data issent to the Android device for approx. 1s. The received datais preprocessed, the features discussed in subsection FeatureExtraction are extracted, and an arff file [37] is created. Thisfile is then passed to the Weka library to be evaluated withthe trained model using the KNN classifier. Weka proceedsto classify the data received into a gesture mapped to an ac-tion to control the media player, which will immediately beexecuted. See Table 2 for a relation between the gesture per-formed and the action executed by the Media Player on theAndroid device.
As it can be seen in Figure 8, a field for the participant namehas been added for evaluation purposes. The participantname should consist of the action that is intended to beperformed and the subject name. This information is laterused to store files on the device containing the preprocesseddata, so it can be reused later to evaluate the performance ofdifferent classifiers. Also a visual notification is displayedproviding feedback on what gesture and action was inter-preted by the device. This can be seen at the bottom part ofthe figure.
6
Figure 8. Android app screenshot
EVALUATION AND RESULTS
Experimental SettingThe perfect setting for evaluating the system would involvesubjects cycling on their chosen route while trying to interactwith the system. This, however, was difficult to achieve withthe current system setup, because it would be hard to carry theArduino board with all the connected components while cy-cling. Instead, we decided to use stationary exercise bikes forevaluation, as they are the closest alternative that resembles areal cycling scenario.
In total 6 subjects, 3 female and 3 male, were asked to par-ticipate in the evaluation. Overall, the experiment took 30minutes per subject, where 20 minutes were given to famil-iarize the subject with the sleeve, gestures, and the commandsthe gestures correspond to (see Table 2), and the last 10 min-utes were dedicated to the actual control of the music playerby using hand gestures. In these last 10 minutes subjectswere asked to perform every gesture exactly 10 times. Thefirst 3 subjects received visual and audio feedback after ev-ery gesture during the entire experiment, so they would knowwhether their performed gesture was interpreted correctly ornot. This allowed them to adjust the way they performed thegesture if the system did not interpret it correctly. The last 3subjects were only given audio feedback, making it harder forthem to know which action was performed.
Furthermore, the generated test samples for gestures weresaved locally on the smartphone. These were later used totest the accuracy of different classifiers.
ResultsIn this section we present the results of our experiment.
Table 3 shows the general accuracy for each test subject us-ing different classifiers without data smoothing, as well as theaverage accuracy for all of the subjects. The first 3 subjects,who were given audio and visual feedback, have a better ac-curacy than the ones who only got audio feedback. From
KNN K* Neural NetworkSubject 1 74.00% 74.00% 68.00%Subject 2 84.00% 86.00% 84.00%Subject 3 62.00% 68.00% 64.00%Subject 4 54.00% 54.00% 46.00%Subject 5 50.00% 62.00% 46.00%Subject 6 60.00% 44.00% 60.00%AVG. 64.00% 64.67% 61.33%
Table 3. Subjects and their corresponding accuracies with different clas-sifiers without data smoothing
GR SL Snap WD WUGR 26.67% 30.00% 3.33% 16.67% 23.33%SL 0.00% 95.00% 3.33% 1.67% 0.00%Snap 0.00% 8.33% 73.33% 18.33% 0.00%WD 0.00% 3.33% 0.00% 96.67% 0.00%WU 58.33% 13.33% 0.00% 0.00% 28.33%
Table 4. Confusion matrix using KNN classifier
this we can argue that the more familiarized the user is withthe correct execution of the gestures, the better the accuracywill be. On average K* performs the best, closely followedby KNN. Neural Network seems to perform the worst. Thisdiffers from the results that were achieved during the trainingphase (see Table 1), where KNN had the best accuracy. Thesefindings suggest that the model built in the training phase doesnot necessary perform the best in a real scenario.
After noticing that the model used was not having such a highaccuracy as in the training phase, we decided to check whichgestures were being classified incorrectly. As it can be seenin Table 4, Side Left, Snap and Wrist Down are quite accu-rate. However, Gun Right tends to be confused with all othergestures, and Wrist Up is most of the times confused as GunRight. Looking at Figure 6, we can see that the pattern forWrist Up and Gun Right are slightly similar, which couldjustify such confusion. However, based on the patterns theredoes not seem to be a clear relation between Gun Right andthe other gestures.
Table 5 shows the averaged accuracies of different classifierswhen applying various smoothing techniques. We can seethat using Weighted Low-Pass Differential (WLPD) [41] witha window width of 5 decreases the accuracy of all three classi-fiers. Using Moving Average (MA) [29] with window size of30 decreases the accuracy for KNN and K* classifiers; how-ever, it increases the accuracy of Neural Network classifierby 6%. We can notice a similar trend when using a Low-PassFilter (LPF) [13] (with smoothing = 17). Here, the accu-racy of KNN and K* classifiers decreases, however, the accu-racy of Neural Network classifier increases by 6.67%. In theend, it seems like using Neural Network classifier with LPFto smooth the data allows us to achieve the highest accuracy.
Table 6 shows the average gesture accuracies when using dif-ferent classifiers with Low-Pass Filter for data smoothing. Wecan notice that the Side Left and Wrist Down gestures havethe highest classification accuracy out of all gestures. Thisindicates that these particular gestures have unique charac-teristics and are easily distinguishable in the context of other
7
KNN K* Neural NetworkAVG. WLPD 62.33% 59.33% 57.00%AVG. MA 63.33% 62.00% 67.33%AVG. LPF 61.00% 62.33% 68.00%
Table 5. Averaged accuracies of classifiers when applying differentsmoothing techniques
KNN K* Neural NetworkSnap 63.33% 83.33% 83.33%Side Left 80.00% 81.67% 85.00%Gun Right 45.00% 45.00% 55.00%Wrist Down 86.67% 65.00% 95.00%Wrist Up 30.00% 31.67% 21.67%
Table 6. Gesture accuracies with different classifiers using LPF for datasmoothing
gestures. The Snap gesture in general has an above aver-age accuracy of being classified correctly. We can see thatthe K* and Neural Network classifiers have accuracies above83%, whereas KNN can classify this gesture 63.33% of thetime. The Gun Right and Wrist Up gestures seem to havethe lowest chance of being classified correctly. We can arguethat these two gestures are too similar for most of the sub-jects and therefore get mixed up quite a lot. This suspicionis verified by the confusion matrix in Table 7. We can noticethat the Wrist Up gesture is interpreted as Gun Right ges-ture 76.67% of the time. The Gun Right gesture, on the otherhand, is often interpreted as three other gestures, namely SideLeft, Wrist Down, and Wrist Up.
When comparing the confusion matrix using KNN (Table 4)with the confusion matrix using Neural Network and LPF(Table 7), we can see a significant improvement when clas-sifying Gun Right. Accuracy when classifying Snap also in-creased. However, the accuracy for Side Left, Wrist Down,and Wrist Up decreased. Nevertheless, the general accuracyincreased after smoothing the data.
DISCUSSIONDuring the development of the system we encountered manychallenges. Firstly, the performance of the built sleeve var-ied greatly with different people. For some, the sleeveworked immediately after being put on, for others it was non-responsive. We soon found out that wetting the conductivestrips (with water) would increase conductivity between skinand electrode immensely. With this trick we could makethe sleeve work for everybody, for a short period of time atleast. Because after 15-20 minutes the electrodes would dry,making the sleeve non-responsive again. Since compressivesleeves are meant to keep skin warm and dry (from Sub Sports
GR SL Snap WD WUGR 55.00% 10.00% 6.67% 11.67% 16.67%SL 5.00% 85.00% 8.33% 1.67% 0.00%Snap 0.00% 0.00% 83.33% 16.67% 0.00%WD 1.67% 0.00% 3.33% 95.00% 0.00%WU 76.67% 0.00% 1.67% 0.00% 21.67%
Table 7. Confusion matrix using Neural Network and LPF for datasmoothing
Amazon page: “...Sub RX Arm Sleeves are highly breathableand excellent at wicking moisture away from the skin...”), themoisture we put on the electrodes would be quickly removedfrom the skin by the compressive sleeve. A possible solutionwould be to put a waterproof backing between the electrodeand the compressive sleeve. This would help to keep the elec-trode moisturized. As for the difference between people, wepostulate that, since some people have dry skin, they need towet the electrodes before usage, while others, whose skin hashigher moisture levels, can use the sleeve immediately. Fur-thermore, people with hairier forearms seemed to have moreproblems, most likely due to the hairs pushing the conduc-tive strips off the skin. Lastly, our built sleeve became lesscompressive over the time, due to constant application and re-moval of the sleeve. This seemed to affect the readings fromthe sensors, especially for people with smaller and slimmerforearms, since the compression of the sleeve was not enoughto ensure contact between skin and electrodes.
The wires connecting the sensors and electrodes presentedanother considerable issue. Even a slight movement of thewires seemed to greatly affect the readings from the sensors.In order to avoid this interference wires had to be placed in afixed position, which seemed to help slightly. Naturally, in areal cycling scenario a person is constantly moving, makingthe received data very noisy and most likely unreliable.
For gathering the training and test data, as well as evaluationof the system, the number of subjects was small. This is dueto the fact that it takes quite some time to prepare the system,introduce the subject to the sleeve, describe the different ges-tures, and do the gestures a number of times, while makingsure the sleeve is still wet enough. More subjects would beneeded to build better classifier and perform a more extensiveevaluation.
Moreover, our evaluation was very constrained and limited,since exercise bikes were used to simulate cycling on theroad. However, our system targets cyclists that use real bi-cycles and are often exposed to different conditions. Becauseof this, a better evaluation of the system is needed that in-volves subjects using real bicycles and cycling through dif-ferent paths.
The results we got from our evaluation were not as good as weinitially hoped for. Our highest achieved accuracy of 68.00%for classifying gestures shows that there is still much workto be done. It seems necessary to find alternative gestures toGun Right and Wrist Up, as they have the lowest accuracyof being classified correctly.
Regarding future work, there are a number of changes thatcould be made to the current setup of our system. Insteadof using an activation gesture, we could use a fixed lengthsliding window. This way the users would not need to worryabout performing the activation gesture before performing theactual gesture. Nevertheless, this approach would present dif-ferent problems, for example related to the battery consump-tion, since the data would be continuously streamed to thesmartphone for processing.
8
Furthermore, the Android application could support a wayfor users to customize which gesture corresponds to whichparticular command. This way, different profiles could be setup for different applications, allowing users to use the sleevefor more than just media playback control.
CONCLUSIONThe objective of this project was to develop EMG-enabledsleeve for cyclists, which would allow them to control themusic player on their smartphone using hand gestures. Oneactivation gesture and five control gestures were chosen tobe used in the system. The evaluation of the system showedthat by using a Neural Network classifier with a low-pass fil-ter for smoothing, it is possible to achieve 68.00% accuracyin correctly classifying different gestures. The average accu-racy of the classifier was primarily reduced by two gestures,which were classified correctly only 55.00% and 21.67% ofthe time, respectively. The other three gestures had a muchhigher accuracy, namely 85.00%, 83.33%, and 95.00%.
These results show that it is possible to recognize some ges-tures using simple EMG sensors and a custom built com-pressive sleeve with conductive fabric strips stitched to it.Nevertheless, more research and work needs to be done tofind a better combination of gestures, so that the accuracy ofthe classifier can be increased. Furthermore, different typesand combinations of sensors, conductive fabrics, compressivesleeves, features, classifiers, and preprocessing algorithmsneed to be tested to find the best solution.
REFERENCES1. Advancer Technologies. Advancer TechnologiesTM:
Muscle Sensor v3.http://www.advancertechnologies.com/p/muscle-sensor-v3.html. (Visited on 11/30/2014).
2. Advancer Technologies Support. AdvancerTechnologiesTM: [DIY] Conductive Fabric Electrodes.http://www.advancertechnologies.com/2013/03/diy-conductive-fabric-electrodes.html, March2013. (Visited on 11/27/2014).
3. Ahmed, S. VCNL4020 Proximity Sensor - GestureControl Sensor Board, 2013.
4. Assad, C., Wolf, M., Theodoridis, T., Glette, K., andStoica, A. BioSleeve: A Natural EMG-based Interfacefor HRI. In Proceedings of the 8th ACM/IEEEInternational Conference on Human-robot Interaction,HRI ’13, IEEE Press (Piscataway, NJ, USA, 2013),69–70.
5. Boczon, P. State of the art: Eye tracking technology andapplications. In Proceedings of the 6th Seminar onResearch Trends in Media Informatics (February 2014),1–8.
6. Boostani, R., and Moradi, M. H. Evaluation of theforearm emg signal features for the control of aprosthetic hand. Physiological Measurement 24, 2(2003), 309.
7. Cycling Embassy of Denmark. Bicycle statistics fromdenmark, 2010.
8. Day, S. Important factors in surface emg measurement.Calgary: Bortech Biomedical Ltd (2002).
9. Engineering360, I. Electrodes and electrode materialsinformation on globalspec. http://www.globalspec.com/learnmore/materials_chemicals_adhesives/electrical_optical_specialty_materials/electrical_contact_electrode_materials/electrical_contact_electrode_materials. (Visitedon 12/14/2014).
10. Fels, S., and Hinton, G. Glove-talkii: An adaptivegesture-to-formant interface. In Proceedings of theSIGCHI Conference on Human Factors in ComputingSystems, CHI ’95, ACM Press/Addison-WesleyPublishing Co. (New York, NY, USA, 1995), 456–463.
11. Fitch, G. M., Soccolich, S. A., Guo, F., McClafferty, J.,Fang, Y., Olson, R. L., Perez, M. A., Hanowski, R. J.,Hankey, J. M., and Dingus, T. A. The impact ofhand-held and hands-free cell phone use on drivingperformance and safety-critical event risk. Tech. rep.,2013.
12. Fuhrhop, S., Lamparth, S., Kirst, M., Wagner, G., andOttenbacher, J. Ambulant ecg recording with wet anddry electrodes: A direct comparison of two systems. InWorld Congress on Medical Physics and BiomedicalEngineering, September 7 - 12, 2009, Munich, Germany,O. Dssel and W. Schlegel, Eds., vol. 25/5 of IFMBEProceedings. Springer Berlin Heidelberg, 2009,305–307.
13. Gavin Kistner. Frame rate-independent low-pass filter.http://phrogz.net/js/framerate-independent-low-pass-filter.html,2011. (Visited on 11/25/2014).
14. Gokgoz, E., and Subasi, A. Effect of multiscale PCADe-noising on EMG signal classification for diagnosisof neuromuscular disorders. J. Med. Syst. 38, 4 (Apr.2014), 1–10.
15. Igarashi, T., and Hughes, J. F. Voice as sound: Usingnon-verbal voice input for interactive control. InProceedings of the 14th Annual ACM Symposium onUser Interface Software and Technology, UIST ’01,ACM (New York, NY, USA, 2001), 155–156.
16. Kawata, H., Tanaka, F., Suemitsu, A., and Morita, M.Practical surface emg pattern classification by using aselective desensitization neural network. In ICONIP (2),K. W. Wong, B. S. U. Mendis, and A. Bouzerdoum,Eds., vol. 6444 of Lecture Notes in Computer Science,Springer (2010), 42–49.
17. Kellogg, B., Talla, V., and Gollakota, S. Bringing gesturerecognition to all devices. In Proceedings of the 11thUSENIX Conference on Networked Systems Design andImplementation, USENIX Association (2014), 303–316.
18. Lee, S. Intelligent Control of a Prosthetic Arm by EMGPattern Recognition. PhD thesis, West Lafayette, IN,USA, 1982. AAI8310766.
9
19. Merletti, R., and Parker, P. A. Electromyography:physiology, engineering, and non-invasive applications,vol. 11. John Wiley & Sons, 2004.
20. O’Hara, K., Harper, R., Mentis, H., Sellen, A., andTaylor, A. On the naturalness of touchless: putting the“interaction” back into nui. ACM Transactions onComputer-Human Interaction (2013).
21. Owano, N. Elliptic labs powers up ultrasound fortouchless gesturing, 2014.
22. Perera, D., Eales, R. T. J., and Blashki, K. Voice art:Investigating paralinguistic voice as a mode ofinteraction to create visual art. In Proceedings of the21st British HCI Group Annual Conference on Peopleand Computers: HCI...But Not As We Know It - Volume2, BCS-HCI ’07, British Computer Society (Swinton,UK, UK, 2007), 87–90.
23. Pheatt, C., and Wayman, A. Using the XboxKinect&Trade; sensor for gesture recognition. J.Comput. Sci. Coll. 28, 5 (May 2013), 226–227.
24. Phinyomark, A., Limsakul, C., and Phukpattaranont, P.A novel feature extraction for robust emg patternrecognition. CoRR (2009), 1–10.
25. Ren, Z., Yuan, J., and Zhang, Z. Robust hand gesturerecognition based on finger-earth mover’s distance witha commodity depth camera. In Proceedings of the 19thACM International Conference on Multimedia, MM ’11,ACM (New York, NY, USA, 2011), 1093–1096.
26. Saab, J., Battes, B., and Grosse-Wentrup, M.Simultaneous EEG recordings with dry and wetelectrodes in motor-imagery. Tech. rep., Max PlanckInstitute for Intelligent Systems, Dept. EmpiricalInference, Tubingen, Germany, 2011.
27. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R.,Turner, J., and Landay, J. A. Enabling always-availableinput with muscle-computer interfaces. In Proceedingsof the 22Nd Annual ACM Symposium on User InterfaceSoftware and Technology, UIST ’09, ACM (New York,NY, USA, 2009), 167–176.
28. Searle, A., and Kirkup, L. A direct comparison of wet,dry and insulating bioelectric recording electrodes.Physiological Measurement 21, 2 (2000), 271.
29. Smith, S. Digital Signal Processing: A Practical Guidefor Engineers and Scientists. Demystifying technologyseries. Newnes, 2003. Chapter 15, 277-284.
30. Soares, A., Andrade, A., Lamounier, E., and Carrijo, R.The development of a virtual myoelectric prosthesiscontrolled by an EMG pattern recognition system basedon neural networks. J. Intell. Inf. Syst. 21, 2 (Sept.2003), 127–141.
31. Song, J., Soros, G., Pece, F., Fanello, S. R., Izadi, S.,Keskin, C., and Hilliges, O. In-air gestures aroundunmodified mobile devices. In Proceedings of the 27thAnnual ACM Symposium on User Interface Softwareand Technology, UIST ’14, ACM (New York, NY, USA,2014), 319–329.
32. SparkFun. SparkFun Bluetooth Mate Gold -WRL-12580 - SparkFun Electronics.https://www.sparkfun.com/products/12580. (Visitedon 11/30/2014).
33. SparkFun. SparkFun RedBoard - Programmed withArduino - DEV-12757 - SparkFun Electronics.https://www.sparkfun.com/products/12757. (Visitedon 11/30/2014).
34. SparkFun Electronics. Conductive fabric - 12”x13”medtex130 - dev-10070.https://www.sparkfun.com/products/10070. (Visitedon 12/14/2014).
35. Standring, S., et al. Grays anatomy, vol. 39. Elsevier,2008.
36. Thalmic Labs. Myo - gesture control armband bythalmic labs.https://www.thalmic.com/en/myo/techspecs, 2013.(Visited on 12/13/2014).
37. The University of Waikato. Department of ComputerScience. Attribute-relation file format (arff).http://www.cs.waikato.ac.nz/ml/weka/arff.html.(Visited on 11/30/2014).
38. The University of Waikato. Department of ComputerScience. Weka 3: Data mining software in java.http://www.cs.waikato.ac.nz/ml/weka/. (Visited on12/07/2014).
39. Wheeler, K. Device control using gestures sensed fromemg. In Soft Computing in Industrial Applications,2003. SMCia/03. Proceedings of the 2003 IEEEInternational Workshop on (June 2003), 21–26.
40. Wijsman, J., Grundlehner, B., Penders, J., and Hermens,H. Trapezius muscle EMG as predictor of mental stress.In Wireless Health 2010, WH ’10, ACM (New York,NY, USA, 2010), 155–163.
41. Xu, Z., and Xiao, S. Digital filter design for peakdetection of surface {EMG}. Journal ofElectromyography and Kinesiology 10, 4 (2000), 275 –281.
42. Zhang, X., Chen, X., Wang, W.-h., Yang, J.-h., Lantz, V.,and Wang, K.-q. Hand gesture recognition and virtualgame control based on 3D Accelerometer and EMGSensors. In Proceedings of the 14th InternationalConference on Intelligent User Interfaces, IUI ’09,ACM (New York, NY, USA, 2009), 401–406.
10
