Usability of Serious Games for Health

Grassyara Pinho Tolentino

Departamento de Educação Física e Desporto

Universidade Estadual de Montes Claros

Montes Claros, Brazil

grassyara@yahoo.com.br

Alam Carlos Ventura Pereira

Departamento de Ciências da Computação e Sistema de

Informação, Universidade Católica de Brasília

Brasilia, Brazil

aventura@ucb.br

Marilene Gesiane Mendes de Paula

Departamento de Educação Física e Desporto

Universidade Estadual de Montes Claros

Montes Claros, Brazil

marilene.mendes@yahoo.com.br

Claudio Battaglini

Exercise Physiology - Dept.

The University of North Carolina at Chapel Hill Chapel

Hill, USA

claudio@email.unc.edu

Ricardo Jacó de Oliveria

Faculdade de Educação Física

Universidade de Brasília

Brasilia, Brazil

rjaco@terra.com.br

Abstract — This study aimes to measure the interface usability in

a Serious Game for Health and compare it to a game for

entertainment. The sample is composed of players with extensive

experience in online games. The tool used to measure the

subjective satisfaction of the users is the System Usability Scale

(SUS). The results indicate significantly higher usability scores in

the game for entertainment when compared to the Serious

Games. There is a positive correlation between in-game

experience and user satisfaction. However, both games presented

satisfactory usability, above the average range, indicating that

usability would not be a negative aspect in the games analyzed.

Keywords- usability; serious games; human-computer interaction

I. INTRODUCTION

The potential appeal of virtual games is attracting the interest of a lot of social segments, including education. Serious games is a terminology used to describe virtual games where the goals go beyond the playfulness [1]. Serious games for health are those who seek to modify the behavior of people to improve and expand health care [2] . They may be directed to diseases prevention or management, acquisition of healthy habits, among others things [3].

However, serious games face some obstacles in the affirmation process as a pedagogical tool. In addition to questions about the educational potential of these games, they are still qualify as boring, without challenges and with limited resources [4]. What often moves users away, preventing even a first contact. Furthermore, in games where the pedagogical focus supplants technical features of games for entertainment, the gameplay is threatened and they quality questioned by a direct comparison with games for entertainment [1].

Going beyond issues related to fun and challenge, the software interface usability is another aspect that may be detrimental to the human-machine interaction and prevent that the proposed targets for serious games are complete. Difficulties or excessive facilities in commands, little intuitiveness, inappropriate response time, excessive or insufficient amount of information, faulty navigation are examples of spurious errors in software that affect user satisfaction [5].

Usability can be understood as the quality of man-machine interaction, either on a website, software or any other technological devices. It is composed of attributes such as: ease of learning, efficiency, easy to be remembered, low error rate, and ability to generate satisfaction [6]. Because software has constant interaction, usability problems can pose a significant obstacle to the consume, gameplay of games, mainly focusing on the individual's satisfaction with the product [7].

To ignore the usability of the system is at odds with the rules of ergonomics and quality of softwares [8]. Although usability is not a determining factor for success and engagement in the use of games, it can provide clues to limitations in the system that could undermine the human-machine interaction.

Difficulties in interaction with the interface prevents the player progress through the game. When the player does not succeed against the offered challenges he cannot absorb the information, and often loses he game. Consequently, feelings of frustration, defeat and disability manifest. It can be catastrophic in games for educational purpose

2011 Third International Conference on Games and Virtual Worlds for Serious Applications

978-0-7695-4419-9/11 $26.00 © 2011 IEEE

DOI 10.1109/VS-GAMES.2011.33

172

Serious games for healthcare brings strong representations about the clash between health and disease,they are intended for people with some risk or physiological impairment, and always has the function of transmitting a positive message of optimism and perseverance to users. Therefore, to identify aspects of usability of software that can focus on the player's success in dealing with the interface seems to be an important goal.

Assuming that usability is an important aspect in the evaluation of virtual games, and in view of the scant literature on this topic, it justifies this study that aimed to measure the subjective satisfaction of the user interface to a serious game applied to health, and compare it to a game of entertainment commercial success

II. METHOD

Due to the limited number of studies found that measured

the satisfaction of the final user of serious games applied to health, it’s an exploratory study with a small sample of experienced users.

A. Sample

The study sample consisted of 23 experienced users of

online gaming, with a mean age of 23.61 ± 4.57 and residents

of several satellite cities of Brasilia, Brazil. The criteria for

inclusion in the study were: age over 18 years, does not make

a diagnosis of chronic degenerative condition, be skilled in

MMOG's besides using a virtual game at least 3 hours per

week. The exclusion criteria for the study were: a

manifestation of some chronic health condition, failure to

attend at least one feedback sessions, length of experience

with virtual games less than a year. No aspect of ethno-

cultural or socioeconomic status was considered in the process

of sample selection. All players were invited to participate in

the survey in order to play a network game and serious games

applied to healthcare. After selection, only 13 individuals

completed all the research stages, and two were females.

B. Products Studied

The games employed in this study were:

Re-MissionTM: (HopeLab Foundation, Redwood City, CA,

USA) is an example of serious games for young and adult people with cancer.

Counter-Strike: SourceTM

: (Valve Corporation, Bellevue, WA, USA) is known worldwide as one of the most popular First Person Shooter (FPS) games.

C. Metrics

The instrument used to diagnose the user's subjective

satisfaction was the System Usability Scale (SUS). This is a

general usability scale developed by John Brooke [21],

comprising 10 items that look for measure the subjective

perception of the individual to interact with software without

considering the personal taste. In the used cheklist the answers

are provided through a Likert scale of 5 points. After the

calculations, the scores can range from 0 to 100, that the

higher the value the more favorable usability as perceived by

the user [9].

D. Procedure

Data collection was divided into three phases. At first, all procedures and the research objectives were explained to the volunteers. They were asked to sign the Deed of Consent and complete a questionnaire about the time using that technology. Next, we made a setting of the players to the collection room, and to two different games.

At the second visit to the lab, the volunteers were invited to play CS. Because it’s a multiplayer game this stage of collection proceeded simultaneously with all the volunteers divided into two "teams". The duration of the match was 25 minutes. Immediately after the end of the game the volunteers had to fill a Usability questionnaire. After 5 minutes of rest, all volunteers underwent a re-test to identify variations in usability generated by the initial stress of the game. There were no significant effects identified in the variables between the two sessions. At the last phase, the volunteers went individually to the laboratory and followed the same protocols performed before the game Re-MissionTM (RM).

E. Statistical Analysis

The data were processed using descriptive statistics on average, deviation, and standard percentage. A comparison of usability between groups was performed using Test T paread. In order to measure the impact of discrete variables on usability, we used analysis of covariance ANCOVA. The correlation was assessed using the Spearman ρ. The statistical software used was Statistical Package for Social Sciences (SPSS

®) version 14 for Windows

®. It was adopted a

significance level of p ≤ 0.05.

III. DISCUSSION OF RESULTS

The final sample indicated that the volunteers had an

average of 12.53 ± 4.57 years of experience in virtual games.

Where 100% were older than 11 years of formal education,

and 53.85% over 10 years of knowledge into information

technology and communication tools. Regarding the

experience of the studied games 69.2% of respondents

reported being experienced in CS games and only 32.0% in

RM, as expected because of the specificity of RM and its non-

commercial purpose.

The average usability values of RM were 58.46 ± 14.87

and 82.61 ± 3.81 in CS. The scores of the two games were

above the midpoint of the scale, indicating that they had a

favorable levels of subjective satisfaction with the interfaces.

A score of 50, or midpoint of the table, would be enough to

classify the software as usable. The results classify the RM

usability as satisfying and the CS usability as excellent [10]. There are questions about the applicability of the

requirements of the traditional usability in virtual games [12,13], based generally on the argument that games are not software, so they do not have a defined function theselves [14]. However, others argue that the diversity and complexity

173

of today's online games require user-centered tests in all production phases as well as in e after-market. These tests seek to improve the game, better understand the produced emotions and cognitive processes performed by the user while using the game [15,16].

It is believed that the user satisfaction with the interface in health games can aid in symbolic interaction among player, software and with the health and illness processes. Insofar as the user learns to cope with the game controls, understands the mechanics and objectives, he or she would be able to continue in the game and eventually beat the disease. However, if the individual is constantly losing, or are unable to continue in missions, the usability can become a barrier to learning, effectiveness and fun [62].

In Serious Games the usability takes a special meaning: it can also be correlated with the learning function [17]. Interactive processes between users and non-friendly interfaces that annoy or fatigue the user tend to restrict their efficiency, reduce the concentration on the task, and therefore increase the frequency of errors and thus learning [18]. Therefore, maintaining a high standard of usability in these systems is a prerequisite for the performance of its serious purpose.

The comparison between the two games showed that CS had an average user satisfaction statistically significant (p = 0.001) compared to RM. Unfortunately no studies were found that compared the usability of Serious Games with games for entertainment.

The identified works that measured the usability of games were mostly directed to Massively Multi-player Online Role-playing Games (MMORPGs) [19,20], using heuristic methods [17, 21, 22]. Furthermore, usability tests in games are usually performed by experts [23] and aimed at the development stage of the game [24, 25]. There are few studies applied to the final product on the market and are directed at the analysis the user's perception [25,26].

It is hypothesized that characteristics of the FPS such as high graphics and real time communication channels [28,29] the presence of several players simultaneously in a consistent virtual environment that a real-life scenario. The non-linearity of the game [30] and expression of real and virtual groups with conflicting or collaborative goals [31], may have raised the players' immersion, and enhanced the usability of the CS sample. There is scientific evidence that interactive interfaces and greater degree of immersion, such as online multiple player games or games with motion sensitive controls would present a greater usability compared to traditional systems such as computer, keyboard and mouse [32].

When searching for clues that could explain some previous results some statistically significant correlations were highlighted. Confidence in using the system was positively associated with age (r = 0.618, p = 0.024) and with users’ degree of education (r = 0.650, p = 0.016) in RM, and linked also to the users’ time in virtual games in CS (r = 0.591, p = 0.043). Having higher levels of education and being older were factors correlated to confidence in using RM, and being a

user of digital games for a long time favored confidence in CS.

These factors are supported in a study of serious games where usability problems were detected in individuals considered inexperienced in virtual games [18] and also in those with poorer academic performance [15].

These correlations match the findings of the sample profile of this study. The average age found (23.61 ± 4.57 years), the experience time in vitual games (12.53 ± 4.57 years), and higher levels of education (building on the average schooling of Brazil) characterized the subjects as digital natives, with access to Information and Communication skills that have grown and evolved together with the games. If in the 80s they were playing Pac-Man

TM (Atari

®) and Mario Bros

(Nintendo®) they currently interact with systems much more complex as HaloTM (Microsoft

® Game Studios) and World of

WarcraftTM (BlizzardTM

) [33]. These sample characteristics probably give knowledge and sufficient stability to users so they feel confident in using the software.

The usability analysis revealed a general positive association between the CS and the experience time with this game (r = 0.591, p = 0.043). These data are consistent with a study of children who used virtual games [26]. The results showed that those more experienced in virtual games reported fewer usability problems and considered the game more enjoyable when compared to the first experiment. A survey with 10 MMORPGs users revealed that individuals who were less experienced presented more usability problems when compared to experienced individuals [19]. The experience was also a crucial aspect for the evaluation of World of Warcraft, and experienced individuals have also made a more positive assessment of this MMORPG [20, 34]. The results showed that those more experienced in virtual games reported fewer usability problems and considered the game more enjoyable when compared to the first experience [26].

IV. CONCLUDING REMARKS

The collected data through this study indicate that Re-mission, a Serious Game Applied to Health has a satisfactory usability, but lower than the usability of an intertainment game. We also conclude that in a sample of users with extensive knowledge in virtual games user experience may be an aspect associated with the user’s satisfaction with the game.

The limitations of this study are the small number of volunteers, making it difficult to generalize the results and the lack of data that could indicate more precisely the aspects related to reduced usability of the Re-mission.

Future works should look to analyze the effect of experience on usability and measure this variable in a target audience of Serious Games Applied to Health.

It is also suggested that new studies attempt to develop questionaires on usability specific for games, which contain a greater amount of information and may be divided into classes, as occurs in heuristic evaluations.

174

ACKNOWLEDGMENT

The authors thank the Digital Equipment Co. Ltd. (DEC),

Reading (UK) for kindly giving the System Usability

Scale,the HopeLab for allowing to conduct research with

software re-missionTM. And the Prof. Dr. Herbert Simões for

borrowing the installations of Leef-UCB for the realization of

this study.

REFERENCES

[1] D. Charsky, “From Edutainment to Serious Games: A Changein the Use

of Game Characteristics” Games and Culture, vol. 5, no. 2, pp. 177-198, 2010.

[2] D. Thompson, T. Baranowski, R. Buday, J. Baranowski, V. Thompson, R. Jago, and M. J. Griffith ”Serious Video Games for Health: How Behavioral Science Guided the Development of a Serious Video Game” Simulation & Gaming, vol. 41, no. 4, pp. 587–606, 2010.

[3] T. Baranowski, R. Buday, D. I. Thompson, and J. Baranowski, “Playing for Real Video Games and Stories for Health-Related Behavior Change” Am J Prev Med, vol. 34, no.1, pp. 74–82, 2008.

[4] R.Van Eck, “Digital Game-Based Learning: It's Not Just the Digital Natives Who Are Restless”. Educause Review, vol. 41, no. 2 , pp.16–30, 2006.

[5] D. Pinelle, N. Wong, and T. Stach, “Heuristic Evaluation for Games: Usability Principles for Video Game Design.” pp: 1453-1462. April, 2008. Proceeding of the twentysixth annual SIGCHI conference on Human factors in computing systems CHI, Florence, Italy, 2008 [doi>10.1145/1357054.1357282]

[6] J. Nielsen, and R.L. Mack, (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, NY, 1994.

[7] D. Pinelle, N.Wong, T. Stach, and C. Gutwin, “Usability Heuristic for Networked Multiplayer Games”. GROUP '09 Proceedings of the ACM 2009 International conference on Supporting group work.[ doi>10.1145/1531674.1531700]

[8] ISONORM 9241 Part 1-17: “Ergonomic Requirements for Office Work with visual display terminals”. Standard, 2008.

[9] J. Brooke, SUS: A “quick and dirty” usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A.,McClelland (eds.) Usability Evaluation in Industry, pp. 189-94. Taylor & Francis, London, UK, 1996.

[10] A. Bangor, P. Kortum, and J. Mille,r “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale”. Journal of Usability Studies, vol. 4, Issue 3, pp. 114-123, 2009.

[11] T. Colombi, F. Louis-Fernand, A. Chevalier, and L. Synyukov ,“How to improve video game menus usability? A specific methodology”. Available from: http://www.ludo-tic.com/dev/downloads/publications/HCII07_Colombi%20et%20al%20How%20to%20improve%20videogame%20menus.pdf

[12] R. Pagulayan and K. Steury, “Beyond Usability in Games” ACM. Interactions/ Sep./Oct, 2004.

[13] L. Nacke “From Playability to a Hierarchical Game Usability Model” FuturePlay @ GDC Canada, May 12-13, Vancouver, Canada, 2009.

[14] H. M. Omar, and A. Jaafar, “Heuristics evaluation in computer games.” International Conference on Information Retrieval Knowledge

Management CAMP, 188-193. 2010. IEEE Computer Society Press. Retrieved from http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5466921

[15] S. Papaloukas, C. Stoli, K. Patriarcheas, and M. Xenos,“A Survey on How Online Games Usability Affects the Retention of Information”. In 14th Panhellenic Conference on Informatics (IEEE), Greece, pp. 209-213, 2010.

[16] A. Drachen, and A. Canossa, “Towards Gameplay Analysis via Gameplay Metrics” In MindTrek 2009, September 30th-October 2nd, Tampere, FINLAND, pp. 202-209, 2009.

[17] M. Xenos, S. Papaloukas, and N. Kostaras ,“Games’ Usability And Learning – The Civilization Iv Paradigm”. IADIS GET, Game and Entertainment Technologies, pp. 3-10, Algarve, Portugal, 2009.

[18] M. Virvou, G. Katsionis, and K. Manos, “Combining Software Games with Education: Evaluation of its Educational Effectiveness.” Educational Technology & Society, vol. 8, no. 2, pp. 54-65, 2005

[19] S. Cornett, “The Usability of Massively Multiplayer Online Roleplaying Games: Designing for New Users. Conference on Human Factors in Computing Systems” . Proceedings of the SIGCHI conference on Human factors in computing systems. Vienna, Austria, pp. 703 – 710, 2004

[20] S. Song, J.Lee, and I.Hwang, “A New Framework of Usability Evaluation for Massively Multi-player Online Game: Case Study of “World of Warcraft” Game”. J. Jacko (Ed.): Human-Computer Interaction, Part IV, HCII 2007, pp. 341–350, 2007.

[21] M.A. Federoff, Heuristics And Usability Guidelines For The Creation And Evaluation Of Fun In Video Games. Dissertação (mestrado), Department of Telecommunications of Indiana University.2002, 52 p.

[22] A. Febretti, and F. Garzotto, “Usability, Playability, and Long-Term Engagement in Computer Games.” CHI 2009 ~ Spotlight on Works in Progress ~ Session 1 April 4-9, ~ Boston, MA, USA, 2009.

[23] A.R.M. Cuperschmid, “Heurísticas de Jogabilidade para Jogos de Computador.” Dissertação (Mestrado), Universidade Estadual de Campinas. Campinas, SP. 2008, 200p.

[24] C. Gresse von Wangenheim, M. Thiry, and D. Kochanski, “Empirical evaluation of an educational game on software measurement” Empir Software Eng, vol. 14, pp. 418–452, 2009.

[25] A. S. Gomes and S. Padovani, “Usabilidade no ciclo de desenvolvimento de software educativo.” Simpósio Brasileiro de Informática na Educação SBIE´2005, Juiz de Fora (MG). Anais do Simpósio Brasileiro de Informática na Educação SBIE vol. 1, 2005.

[26] W. Barendregt, M. M. Bekker, D. G. Bouwhuis, E. Baauw, “Identifying usability and fun problems in a computer game during first use and after some practice.” International Journal of Human-Computer Studies, vol. 64, no. 9, pp. 830-846, 2006.

[27] F.B. Breyer, D. Moura, S. Padovani, and A. Neves, “Avaliação De Usabilidade No Processo De Desenvolvimento De Jogos: Definição de métodos de acompanhamento de qualidade para Game.” Design. V Brazilian Symposium On Computer Games And Digital Entertainment. 8 - 10 November, Recife – Brazil, 2006.

[28] R.T.A. Wood, “ The Structural Characteristics of Video Games: A Psycho-Structural Analysis.” Cyberpsychology & Behavior, v. 7, n.1, p: 1-11, 2004.

[29] F. Trinta, F. R. Cecin “ Jogos Multiusuário Distribuídos: Modelos, Suporte, Variações e Estado da Arte”. VI Brazilian Symposium on Games and Digital Entertainment, 2007, São Leopoldo. Anais Eletrônicos… VI Brazilian Symposium on Games and Digital Entertainment, 2007

[30] T. Manninen, “Virtual Team Interactions in Networked Multimedia Games - Case: “Counter-Strike” – Multi-player 3D Action Game.” Proceedings of Presence Conference, Philadelphia, USA, p: 1-9. 2001

[31] D. Clarke, P.R. Duimering, “How Computer Gamers Experience the Game Situation: A Behavioral Study”. ACM Computers in Entertainment. v. 4, n. 3, p:1-23, 2006.

[32] N. Adamo-Villani, R. Wilbur, and M. Wasburn, “Gender differences in usability and enjoyment of VR educational games: a study of SMILE™” International Conference Visualisation, 2008.

[33] E. M. Glinert, “The Human Controller: Usability and Accessibility in Video Game Interfaces”. In Massachusetts Institute of Technology, 2008.

[34] J.W. Yoon, S.H. Jang, and S.B. Cho, “ Enhanced User Immersive Experience with a Virtual Reality based FPS Game Interface.” Computational Intelligence and Games (CIG), 2010 IEEE Symposium on . pp: 69 - 74 18-21 Aug. 2010 [DOI> 10.1109/ITW.2010.5593369 ]

175