Assessing Brain-Computer Interfaces for Controlling Serious Games

Fotis Liarokapis1,2, Athanasios Vourvopoulos1, Alina Ene1, Panagiotis Petridis2 1Interactive Worlds Applied Research Group (iWARG)

2Serious Games Institute (SGI) Coventry University

Coventry, UK F.Liarokapis@coventry.ac.uk

Abstract— This paper aims at examining how to fully interact with serious games in noisy environments using only non-invasive EEG-based information. Two different EEG-based BCI devices were used, one which requires no calibration, and another one that needs some sort of calibration to create a user profile. User testing was performed using both types of BCIs with 61 participants. Results indicate that although BCI devices are still in their infancy, they offer the potential of being used as alternative game interfaces prior to some familiarisation with the device and in several cases a certain degree of calibration.

Keywords – serious games, brain-computer interfaces,

human-machine interaction.

I. INTRODUCTION Human-computer interaction techniques for

computer games are one of the hottest topics in terms of research and development. Although a lot of different interaction approaches exist (e. g. mouse, keyboard, joystic, Nintento Wii controller, Microsoft Kinect, Guitar Hero, etc) they require a lot of physical effort. This restricts user’s expressive capabilities as well as the information transfered from the user to the computer [1]. During the past few years, non-invasive brain-computer interfaces (BCIs) seem to be getting a lot of attention as alternative human-computer interaction devices for games and virtual environments [2], [3].

Computer gamers, who represent a big proportion of modern society, are looking for new and more intuitive ways of interacting with video games more effectively. Although non-invasive BCI technologies seem to have the potential of providing an interactive environment where “thoughts are not constrained by what is physically possible” [4], they are still not ready for commercial use.

The aim of this research is to examine how to play effectively a three-dimensional serious game using only non-invasive EEG-based information. The objectives of the research are twofold: (a) to fully control an avatar of a serious game in real-time performance using only EEG data and (b) to examine the reaction of users while playing the game.

To achieve that, two different EEG-based BCI devices were used, one which requires no calibration and another one that needs some calibration. The user is visually stimulated by fully controlling an avatar in the Roma Nova serious game. Two different types of EEG-

based BCIs were used: the Neurosky Mindset and the Emotiv EPOC. All tests (61 participants in total) were done using the same serious game, which was integrated with the devices (31 participants for the Neurosky Mindset and 30 for the Emotiv EPOC).

II. ROMA NOVA GAME Rome Reborn project created highly realistic 3D

representations illustrating the urban development of ancient Rome from the first settlement in the late Bronze Age (ca. 1000 B.C.) to the depopulation of the city in the early Middle Ages (ca. A.D. 550) [5]. Rome Reborn includes hundreds of buildings, thirty two of which are highly detailed monuments reconstructed on the basis of reliable archaeological evidence. The rest of the 25 to 30 square kilometres model is filled with procedurally-generated buildings based on accurate historical knowledge. Figure 1 illustrates the western plaza of the Flavian Amphitheatre (right) with the Arch of Constantine (center), and Temple of Venus and Rome (left). Behind them, the arch looms and the bronze Colossus of the Sun are shown.

Figure 1 Rome Reborn Reconstruction [5]

The interactive game is built upon Rome Reborn and it is called the Roma Nova project. It builds on previous work at Coventry University [6] and it is a serious game that aims at teaching history to young children (11 to 14 years old). The game allows for exploratory learning by immersing the learner/player inside a virtual heritage environment where they learn different aspects of history through their interactions with a crowd of virtual authentic Roman avatars. The game was designed based on the ‘Unity 3D’ game engine using parts of the realistic reconstruction of ancient Rome. The aim of the game is to navigate an avatar inside virtual Rome and interact with intelligent agents while learning at the same

978-1-4799-0965-0/13/$31.00 ©2013 IEEE

time. Both navigation and interaction are performed using brain-wave technology from the two headsets (Neurosky and Emotiv).

The implementation of Roma Nova that was integrated and tested with the two BCI systems includes: (a) a crowd of Roman characters in the Forum and (b) a highly detailed set of buildings that belong to the Rome Reborn model. Intelligent agents are wandering in the gaming environment between predefined points of interest. , whereas the avatar is controlled by the BCI devices (see sections III and IV).

III. NEUROSKY MINDSET INTERACTION The Neurosky Mindset device was used, which is a

complete headset with speakers and microphone transmitting data on Bluetooth. It allows extracting the ‘Attention’ and ‘Meditation’ levels of the user. The headset is calculating the Raw EEG signals to produce the “eSense Meters” [7]. The patterns of the electrical activity are analyzed with the help of feature extraction and classification algorithms by converting the EEG signals into control commands. The Neurosky headset is using a single dry sensor attached to the forehead outside the cerebral cortex in the frontal lobe of the brain being responsible for the attention level and short-term memory tasks [8]. As soon as a connection between the RomaNova and the the headset is established, it initiates a connection with the avatar and the simulation to send movement instructions.

In terms of hardware configuration for the interaction of the game as well as the evaluation, off-the-shelf hardware components have been used—with the main component being the NeuroSky MindSet neuro-headset (i.e. one electrode at the FP1 position). A laptop with a 64ibt Intel(R) Core(TM) 2 Duo processor T6600 at 2.2GHz and 4 GB of memory was used for the evaluation. The laptop is equipped with an NVIDIA GeForce GT 240M graphics card. Standard laptop display technology, such as a 16’ inch wide LCD has been used in order to display the 3D content of the application.

A. Methodology Since no direct training is required, as soon as the

Roma Nova serious game establishes connection with the headset (through Bluetooth) it initiates the simulation. Both ends of the connection can ‘open’ input and output streams and read/write data. The computer sends two integer values (in the range 0 to 100) to the game depending on the attention and meditation levels of the user. Table 1 and Table 2 illustrate how these values were mapped to represent direction and speed of the avatar.

Attention (0-100) Direction 0-40 Turn Left 40-75 Go straight ahead 75-100 Turn right Table 1 Controlling the direction of the player

Table 1 illustrates the values that were assigned for controlling the direction of the players whereas table 2 presents the values that were assigned for controlling the speed of the players.

Meditation (0-100) Speed 0-30 Stand still 30-50 Go backwards 50-70 Walk 70-100 Run Table 2 Controlling the speed of the player

As soon as the RomaNova serious game starts, the player controls the avatar by changing cognitive states such as meditation and attention. To take a right turn, they will try to concentrate as hard as possible, while in order to take a left turn, users have to defocus their attention. Going straight ahead is possible only by maintaining a balance between the two states. High levels of meditation will prompt the avatar to run, medium levels will cause it to walk, low levels will make it go backwards and extremely low levels of meditation will cause it to stagnate.

Users were allowed five minutes to accommodate with controlling the avatar and around three minutes to complete the task by arriving at a particular destination. The prototype was tested on 31 users, in various environments, at different times of the day, within a span of 2 months. In particular, it was tested at Coventry University computer games laboratory, at the third Phoenix Partner Annual Conference, which took place at Coventry University as well as at an international conference (Archeovirtual 2012) which took place at Paestum, Italy, 15-18 November 2012 (Figure 2).

Figure 2 User testing at Archaeovirtual 2012

All users were given a participant information leaflet to read beforehand and a consent form to fill in and sign. The user survey was designed to gather qualitative data.

B. Results With regard to the overall experience, the following

positive observations were made. Users were drawn by the novelty of the system and were keen on testing the

control of a virtual character via brain technology. They viewed the concept as an interesting approach to a gaming scenario, categorising the experience as challenging, enjoyable, and engaging. Some testers also reported that they would participate in further research studies related to the current interface and would want to be informed about the outcome of the project. Even though the gameplay was generally seen as “fun and fascinating”, some users were dissatisfied with the avatar movement accuracy. They were able to recognise some movement types better than others. Turning left and right (switching from a strong state of concentration to a more relaxed state) was reported as the most difficult part to manage. Also, they reported a lag around 2-3 seconds between their intentions and the actual output.

In terms of possible improvements users mentioned: (a) the need for an initial training, (b) user-profiling period, and (c) in-game guide. They found it easy to use the device but much harder to adapt to it and felt it was hard to ‘train’ their brains to concentrate and meditate. Going backwards was not seen as a popular movement type amongst players and its removal was advocated. Seeing relevant feedback on the screen, apart from the actual character movement, was suggested as being highly beneficial to the experience. Moreover, participants wanted the interface to show the actual measured data, to get real-time feedback and know what to do in order to attempt self-regulating the attention/meditation levels. They proposed the introduction of hotkeys that should facilitate what attention can measure. Last but not least, the implementation of the concept using a different game was advised.

IV. EMOTIV HEADSET INTERACTION The Emotiv Headset is a neuro-signal acquisition

and processing wireless neuro-headset with 14 wet sensors (and 2 reference sensors) which is capable of detecting brain signals as well as user’s facial expressions and head position [9]. This requires a unique user profile to be trained to map users’ brain-patters. The Emotiv Development Kit was used connecting the Emotiv Epoc headset to the Emotiv control panel to create and train a new user profile. The user is visually stimulated by controlling an avatar in the serious game (RomaNova). The raw data is calculated on the headset’s chip and sent to the dedicated computer. Afterwards EmoKey was used, a program that generates keystroke events for moving the player based on the users’ brain-activity and facial expressions within the virtual world.

The prototype system is fully operational and it is s using a combination of Cognitive and Facial/Muscular functions. Off-the-shelf hardware components have been used—with the main component being the Emotiv Epoc neuro-headset (e.g. 14 electrodes and a gyroscope). A laptop with a 64ibt Intel(R) Core(TM) 2 Duo processor T7700 at 2.4GHz and 4 GB of memory was used for the evaluation. The laptop is equipped with an NVIDIA GeForce 8700M GT graphics card. Standard laptop display technology, such as a 17’ inch wide LCD has

been used in order to display the 3D content of the application. It was ensured that each participant was comfortable and at ease prior to the start of the experiment. The testing was performed in open-space laboratory conditions.

A. Methodology Training an effective new user profile takes

approximately 30 to 60 minutes depending on the adaptability of the user and the classification score. However, training the profile is not an easy task and requires practice and familiarization, especially when the user needs to train more than two actions as it is easy to get distracted from outside stimuli and ‘confuse’ the training process of the users real ‘intentions’. In a training session no more than 1 hour, user’s skills can be increased up to 65% for the forward & backward moves using the Emotiv control panel [10]. As soon as the profile is created, then the combination of Cognitive and Facial/Muscular functions can be used to control an avatar in a computer game.

Figure 3 Ability to control events

New players (Figure 3) can gain control over a single action quite quickly. Learning to control multiple actions typically requires practice and becomes progressively harder for the classifier as additional actions are added. As players learn to train reproducible mental states for each action, the detection becomes increasingly precise. Most players typically achieve their best results after training each action several times. Overtraining can sometimes produce a decrease in accuracy – although this may also indicate a lack of consistency and mental fatigue. Practice and experience will help to determine the ideal amount of training required for each individual user to successfully interact with the serious game [10].

B. Results Thirty users were asked to provide comments on a

questionnaire anonymously after playing the serious game. All users had been asked to provide comments on the questionnaire anonymously. Moreover, an unstructured short interview took place in a light mood after the trial. These comments constitute a very helpful contribution towards the improvement of the system,

giving feedback that an ordinary questionnaire cannot capture. A very useful suggestion was about the Graphical User Interface (GUI) of the Game. User’s found easier to focus on GUI components instead of the virtual character in order to perform the required action. This might be a result of the training trial, in which the users had to push/pull a virtual cube and when entering the virtual environment they had to re-adapt to the new elements. This can be confirmed by reports that it was easier to move the avatar by thinking the cube from the training trial, that actually visualising the character movement through space. This is a clear indication that it would be better for the training trial to include the components from the game so as for the user to get familiarised with it.

Alternatively, assistive GUI components might be a useful addition. Overall the experience was reported as quite engaging and interesting regardless certain issues of response time and accuracy that other Natural User Interfaces (NUI’s) might have. Finally, it was reported that people with more experience in computer games will have an easier time learning to use the interface due to the simulation and interaction required for a computer game. That experience makes it much easier to learn how to operate the interface. The only negative reporting had to do with the tiredness that the system was triggering after a few minutes of interaction.

V. CONCLUSIONS AND FUTURE WORK This paper presented two different ways of fully

interacting with the same serious game under noisy environments using non-invasive BCIs. Two different EEG-based BCI devices were used, one which requires no calibration and another one that needs some sort of calibration to create a user profile. Overall the results indicate that both BCI technologies offer the potential of being used as alternative game interfaces prior to some familiarisation with the device and in some cases some sort of calibration.

As far as the qualitative feedback is concerned, both categories of participants enjoyed the experience. Neurosky users found it easier to use the device but much harder to adapt to it. They felt it was difficult to achieve the desired levels of attention and meditation on a first time basis. However, after some self-training it is possible to improve a lot. On the other hand, the Emotiv device was easier to perform the training procedure and control more accurately the avatar but takes a lot of time. Also, setup is much more complicated compared to the Neurosky one.

Further prototype developments could also include an analysis into how certain audio tracks can stimulate concentration/attention and inherently affect gameplay. Additional recommendations comprise of the incorporation of more sensors and maybe eye tracking technology to enhance movement accuracy. Also, more sophisticated non-invasive BCI devices equipped with more electrodes and sensors will be used.

ACKNOWLEDGEMENTS

The authors would like to thank the Interactive Worlds Applied Research Group (iWARG) as well as the Serious Games Institute (SGI) members for their support and inspiration. Two videos that demonstrate both systems in operation can be found at: http://www.youtube.com/watch?v=L6t4Ji5yu7k&feature=youtu.be and http://www.youtube.com/watch?v=5Y_clGGoO4Y.

REFERENCES

[1] Gürkök, H., Hakvoort, G., Poel, M. “Modality Switching and Performance in a Thought and Speech Controlled Computer Game”, Proc. of the 13th Int’l conference on multimodal interfaces, ACM Press, 41-48, (2011).

[2] Lécuyer, A., Lotte, F., Reilly, R.B., Leeb, R., Hirose, M., Slater, M. “Brain-Computer Interfaces, Virtual Reality and Video Games”, IEEE Computer, 41(10): 66-72, (2008).

[3] Nijholt, A., Reuderink, B., Plass-Oude Bos, D. “Turning Shortcomings into Challenges: Brain-Computer Interfaces for Games”, Entertainment Computing, 1(2): 85-94, (2009).

[4] Bos, D.P.O., Reuderink, B., et al. “Human-Computer Interaction for BCI Games: Usability and User Experience”, Proc. of the 2010 Int’l Conference on Cyberworlds, IEEE Computer Society, 277-281, (2010).

[5] Rome Reborn, Available at: [http://www.romereborn.virginia.edu/], Accessed at: 17/03/2013.

[6] Panzoli, D., Peters, C. et al. “A Level of Interaction Framework for Exploratory Learning with Characters in Virtual Environments”, Intelligent Computer Graphics, Studies in Computational Intelligence, Springer-Verlag Berlin Heidelberg, 321:123-143, (2010).

[7] NeuroSky’s eSense™ Meters and Detection of Mental State, Available at: [http://company.neurosky.com/files/neurosky_esense_whitepaper.pdf], Accessed at: 17/03/2013.

[8] Vourvopoulos, A., Liarokapis, F. “Robot Navigation using Brain-Computer Interfaces”, Proc. of the 11th Int’l Conference on Ubiquitous Computingand Communications (IUCC-2012), IEEE Computer Society, Liverpool, UK, 1785-1792, (2012).

[9] Emotiv EPOC Software Development Kit, Available at: [http://www.emotiv.com/store/hardware/299/], Accessed at: 17/03/2013.

[10] Vourvopoulos, A., Liarokapis, F. Petridis, P. “Brain-Controlled Serious Games for Cultural Heritage”, Proc. of the 18th Int’l Conference on Virtual Systems and Multimedia, Virtual Systems in the Information Society, IEEE Computer Society, Milan, Italy, 291-298, (2012).