Embed Size (px)
Citation preview
Evaluating Efficacy and Validating Games for Health
Pamela M. Kato, EdM, PhD
Abstract
The field of games for health is growing rapidly, but many games for health are not validated for their use asa tool to improve outcomes. The few research studies that do exist are often poorly designed, and theirconclusions cannot be considered valid evidence to support or refute efficacy. Based on lessons learned fromthe field of digital game-based learning in education and recent efforts to systematically review games forhealth in meta-analyses, guidelines are suggested for conducting high-quality efficacy studies on gamesfor health.
The Problem
The field of games for health is growing dramatically.Major funding agencies are supporting the development
of innovative games to train, educate, and even enhancetreatments for patients. Hospitals and medical schools aredeveloping their own computer simulations and seriousgames to train healthcare professionals. Conferences thatspecifically address games for health are being organized andattended in impressive numbers. The number of blogs, relatedLinkedIn groups, and the publication of a peer-reviewedjournal suggest that the interest and activity in this area haveattained ‘‘critical mass.’’
An Internet search combining the name of any of the topdiseases with ‘‘game’’ will turn up several examples. How-ever, only a handful have undergone a rigorous scientificevaluation to validate their use as tools that contribute tohealth and well-being.1 Of those that have been evaluated,inferences regarding their effectiveness are virtually impos-sible to make given the weakness of the study designs.2–5
Exceptions are seen in evaluations of games used to promotephysical activity, physiotherapy, and healthy eating.6–8
A Foreshadowing of Things to Come?
If we fail to conduct rigorous research on games for health,we may be destined to experience the same situation facingthe proliferation of serious games in education. This field,known as digital game-based learning (DGBL), was enthu-siastically embraced to address the problem of decreases instudents’ motivation to pursue academics, deterioratingstudy habits, and drops in academic performance.9 A recentarticle in the New York Times10 reported that the marketingclaims of educational software companies were not consistentwith research evaluating the efficacy of these programs con-
ducted by the What Works Clearinghouse (WWC), an ini-tiative of the U.S. Department of Education’s Institute ofEducation Sciences. WWC investigations of math softwarefound inadequate standards of evidence to make any con-clusions about the effectiveness of most math educationalsoftware they reviewed.11–13
The reports on the DGBL programs available on the WWCWebsite revealed that the WWC did not conclude that thesoftware had no impact on learning. Rather, they concludedthat the studies conducted on the DGBL programs were sopoorly designed and conducted that they could not be re-viewed no matter what the findings of the individual studieswere. As such, there is inadequate evidence to draw anyconclusions regarding the impact of those educational soft-ware programs on outcomes in the schools.
In addition, the conclusions of the WWC reports are eerilysimilar to the conclusions of recent reviews of games forhealth applications. A recent meta-analysis on games to trainhealth professionals concluded that the lack of high-qualityresearch neither confirms nor refutes their efficacy.3 Althoughmeta-analyses on interactive technologies for health that in-clude games do show areas of effectiveness,14,15 they still statethat conclusions are limited because of poor research designand reporting of studies.
Guidelines for Future Research
Future research evaluating the effectiveness of gamesfor health to improve outcomes should be driven by thefollowing guidelines. These guidelines are based on lessonslearned from educational software and ongoing efforts tointegrate quantitative research on games for health andare intended to create confidence, validity, and integrity intheir results.
Senior Researcher, University Medical Center Utrecht, Utrecht, The Netherlands.
GAMES FOR HEALTH JOURNAL: Research, Development, and Clinical ApplicationsVolume 1, Number 1, 2012ª Mary Ann Liebert, Inc.DOI: 10.1089/g4h.2012.1017
74
Ground your game in theory
The first step to conducting a good validation trial beginsduring the design phase.16 Several researchers in DGBL havebeen calling for more rigorous research in order to developguidelines for making effective digital games for learning.17–
19 They argue that the lack of coherent theories and models inthe field of DGBL is the root cause of the lack of clear andestablished guidelines for making effective DGBL. Similarly,a theoretical approach guiding the development of games forhealth provides a good basis for testing hypotheses based onthose theories in subsequent research evaluations.
Conduct a randomized trial
In 2011, there was an impressive and sharp increase in thepublication of randomized trials on games for health com-pared with previous years.20–22 This trend must continue.Randomized trials are often very time consuming and ex-pensive to conduct, but they are considered the gold standardfor research aimed at identifying the causal relationship be-tween an intervention and outcomes.
Include adequate control groups
Implicit in conducting a randomized trial is inclusion of acontrol or comparison group to which participants are ran-domly assigned to which researchers are blinded. A recentstudy compared people who cycled to music or to an inter-active cycling game found that cycling to music was equal toor better than cycling to a videogame and cost much lessmoney.23 This is very valuable information that could onlyhave been revealed by comparing the game with anothermotivating but less interactive activity. Blinding of treatmentassignment is often difficult for participants in these types oftrials but quite necessary for objectivity.
Recruit an adequate number of participants
Conduct a power analysis when you design your study. Itis advisable to conduct power analyses even before designingyour game to ensure that the game is focusing on a problemthat the game can adequately address and that you can re-cruit adequate numbers of participants in your research trialto assess the game’s impact.16
Include objective measures of health
Incorporate standardized measures in your research tofacilitate comparisons across studies and combine self-report with objective measures. Most evaluations of gamesfor health assess cognitive, affective, behavioral, and biolog-ical functioning via self-report.14 Ideally, self-report should becombined with objective measures of health outcomes.24 Anexcellent example of this is a randomized trial of game fordiabetes control published 15 years ago.25 Researchers foundthat although children who played the diabetes game did notshow increases in knowledge about diabetes or glycemiccontrol, their communication about the disease increased,their self-care behaviors improved, and, most impressively,they showed a significant decrease in urgent care visitscompared with children who did not play the game duringthe year of follow-up, an objective indicator of the game’simpact on health.
Monitor and report potential negative side effects
Evaluations of games for health should attempt to monitorthe use of their game for negative side effects. Possible neg-ative side effects of playing games, although rare, can includeseizures due to photosensitivity and tendinitis.26 Anotherpotential side effect is excessive and addictive use of games.Research investigating the consequences of excessive gamingand playing violent videogames27,28 requires us to monitorthe safety of these interventions.
Publish even null results
The problem with studies with small sample sizes is thatthey often lack power to show an effect even if one really doesexist. Thus, they are often rejected for publication. This phe-nomenon of null findings (or even negative effects) not beingpublished is known as the ‘‘file drawer’’ problem.29 If we putour null findings in a file drawer, future meta-analyses ofgames for health research will be biased to show positiveeffects. Existing journals often recognize this need and willpublish null findings.
Conclusions
Our standards for validation of the growing number ofgames for health need to be raised. Studies that show dramaticand impressive effects of games for health will be overlookedand disregarded if they are not conducted according to theguidelines above and others that reflect basic standards forproviding valid research evidence. The guidelines are pro-vided here in a very simple form. They are not exhaustive anddo not adequately acknowledge the complexity of carryingthem out in the real world. It is a difficult task, but the futureof our endeavors depends on high-quality research to drivethe current enthusiasm and activity in this field.
Acknowledgments
The author would like to thank John P. Pinto, PhD for hisvaluable comments and technical advice on this article.
Author Disclosure Statement
No competing financial interests exist.
References
1. Kato PM. Video games in health care: Closing the gap. RevGen Psychol 2010; 14:113–121.
2. Baranowski T, Buday R, Thompson DI, Baranowski J.Playing for real: Video games and stories for health-relatedbehavior change. Am J Prev Med 2008; 34:74–82.
3. Akl EA, Pretorius RW, Sackett K, et al. The effect of edu-cational games on medical students’ learning outcomes: Asystematic review: BEME Guide No 14. Med Teach 2010; 32:16–27.
4. Blakely G, Skirton HG, Cooper S, et al. Educational gamingin the health sciences: Systematic review. J Adv Nurs 2009;65:259–269.
5. Laver K, George S, Thomas S, et al. Virtual reality forstroke rehabilitation. Cochrane Database Syst Rev 2011: 9:CD008349.
6. Peng W, Lin J-H, Crouse J. Is playing exergames really ex-ercising? A meta-analysis of energy expenditure in active
EVALUATION AND VALIDATION 75
video games. Cyberpsychol Behav Soc Netw 2011; 14:681–688.
7. Guy S, Ratzki-Leewing A, Gwadry-Sridhar F. Moving be-yond the stigma: Systematic review of video games and theirpotential to combat obesity. Int J Hypertens 2011; 2011:doi: 10.4061/2011/179124, 13 pp.
8. Mark R, Rhodes R, Warburton DER, Bredin S. Interactivevideo games and physical activity: A review of the literatureand future directions. Health Fitness J Canada 2008; 1:14–24.
9. Beaton AE, Mullis IVS, Martin MO, et al. MathematicsAchievement in the Middle School Years. Boston: AAA, BostonCollege; 1996.
10. Gabriel T, Richtel M. Grading the digital school: Inflating thesoftware report card. New York Times October 8, 2011.
11. What Works Clearinghouse. Carnegie Learning Curricula andCognitive Tutor� Software. Washington, DC: Institute ofEducation Sciences, U.S. Department of Education; 2010.
12. What Works Clearinghouse. Cognitive Tutor� Algebra I.Washington, DC: Institute of Education Sciences, U.S. De-partment of Education; 2011.
13. What Works Clearinghouse. Accelerated Math�. Wa-shington, DC: Institute of Education Sciences, U.S. Depart-ment of Education; 2011.
14. Bailey JV, Murray E, Rait G, et al. Cochrane Review: Inter-active computer-based interventions for sexual health pro-motion. Evidence-Based Child Health: A Cochrane ReviewJournal 2011; 6:2192–2268.
15. Portnoy DB, Scott-Sheldon LAJ, Johnson BT, Carey MP.Computer-delivered interventions for health promotion andbehavioral risk reduction: A meta-analysis of 75 randomizedcontrolled trials, 1988–2007. Prev Med 2008; 47:3–16.
16. Kato PM. The role of the researcher in making effective se-rious games for health. In: Arnab S, Dunwell I, Debattista K,eds. Serious Games for Healthcare: Applications and Implications.Hershey, PA: IGI Global; in press.
17. Van Eck R. Six ideas in search of a discipline. In: Shelton BE,Wiley D, eds. The Design and Use of Simulation ComputerGames in Education. Rotterdam: Sense Publishers; 2007: 31–60.
18. Van Eck R. Digital game-based learning: It’s not just thedigital natives who are restless. EDUCAUSE Rev 2006;41(2):16–30.
19. Gunter G, Kenny RF, Vick EH. A case for a formal designparadigm for serious games. J Int Digital Media Arts Assoc2006; 3:93–105.
20. Szturm T, Betker AL, Moussavi Z, et al. Effects of an interac-tive computer game exercise regimen on balance impairmentin frail community-dwelling older adults: A randomizedcontrolled trial. Phys Ther 2011; 91:1449–1462.
21. Kowalczewski J, Chong SL, Galea M, Prochazka A. In-hometele-rehabilitation improves tetraplegic hand function. Neu-rorehabil Neural Repair 2011; 25:412–422.
22. Baranowski T, Baranowski J, Thompson D, et al. Videogame play, child diet, and physical activity behaviorchange: A randomized clinical trial. Am J Prev Med 2011;40:33–38.
23. Adamo KB, Rutherford JA, Goldfield GS. Effects of interac-tive video game cycling on overweight and obese adolescenthealth. Appl Physiol Nutr Metab 2010; 35:805–815.
24. Gorber SC, Tremblay M, Moher D, Gorber B. A comparisonof direct vs. self-report measures for assessing height, weightand body mass index: A systematic review. Obes Rev 2007;8:307–326.
25. Brown SJ, Lieberman DA, Gemeny BA, et al. Educationalvideo game for juvenile diabetes: Results of a controlledtrial. Inform Health Soc Care 1997; 22:77–89.
26. Emes CE. Is Mr Pac Man eating our children? A review ofthe effect of video games on children. Can J Psychiatry 1997;42:409–414.
27. Griffiths M. Video games and health. BMJ 2005; 331:122–123.28. Anderson CA. An update on the effects of playing violent
video games. J Adolesc 2004; 27:113–122.29. Rosenthal R. The file drawer problem and tolerance for null
results. Psychol Bull 1979; 86:638–641.
Address correspondence to:Pamela M. Kato, EdM, PhD
Dorpsstraat 143749 AD Lage Vuursche
The Netherlands
E-mail: [email protected]
76 KATO
This article has been cited by:
1. Matt Simkins, Irina Fedulow, Hyunchul Kim, Gary Abrams, Nancy Byl, Jacob Rosen. 2012. Robotic Rehabilitation Game Designfor Chronic Stroke. Games for Health Journal 1:6, 422-430. [Abstract] [Full Text HTML] [Full Text PDF] [Full Text PDFwith Links]
2. Bill Ferguson. 2012. Games for Wellness—Impacting the Lives of Employees and the Profits of Employers. Games for HealthJournal 1:3, 177-179. [Citation] [Full Text HTML] [Full Text PDF] [Full Text PDF with Links]
