Brain Injury, December 2012; 26(13–14): 1564–1573

ORIGINAL ARTICLE

Virtual reality as a screening tool for sports concussion inadolescents

PIERRE NOLIN1, ANNIE STIPANICIC1, MYLENE HENRY1, CHRISTIAN C. JOYAL1,& PHILIPPE ALLAIN1,2

1Laboratoire de Recherche Interdisciplinaire en Realite Virtuelle (LARI-RV), Departement de psychologie, Universite du

Quebec a Trois-Rivieres, Quebec, Canada, and 2Laboratoire de Psychologie ‘Processus de pensee et interventions’,

Universite d’Angers, Angers, France

(Received 9 September 2011; revised 8 May 2012; accepted 23 May 2012)

AbstractPrimary objective: There is controversy surrounding the cognitive effects of sports concussion. This study aimed to verifywhether the technique of virtual reality could aid in the identification of attention and inhibition deficits in adolescents.Study design: A prospective design was used to assess 25 sports-concussed and 25 non-sports-concussed adolescentsenrolled in a sport and education programme.Methods and procedures: Participants were evaluated in immersive virtual reality via ClinicaVR: Classroom-CPT and in real lifevia the traditional VIGIL-CPT.Main outcomes and results: The neuropsychological assessment using virtual reality showed greater sensitivity to the subtleeffects of sports concussion compared to the traditional test, which showed no difference between groups. The results alsodemonstrated that the sports concussion group reported more symptoms of cybersickness and more intense cybersicknessthan the control group.Conclusions: Sports concussion was associated with subtle deficits in attention and inhibition. However, further studies areneeded to support these results.

Keywords: Virtual reality, virtual classroom, ClinicaVR: Classroom, sports concussion, mTBI, mild traumatic brain injury,adolescents

Introduction

The term sports concussion refers to a case of mildtraumatic brain injury (mTBI) resulting from a sportaccident. Sports concussion is defined as change inmental status resulting from trauma or rotationalforces to the brain, with or without loss of con-sciousness [1]. Concussions represent 8.9% of allhigh school athletic injuries and 5.8% of all collegiateathletic injuries and they occur in sports such ashockey and football, as well as soccer, basketball,lacrosse and wrestling [2]. According to Sosinet al.(1996) approximately 300,000 sport-related

concussion occur annually in the United States [3].Given that the direct and indirect costs of sportsconcussion are estimated at nearly US$17 billion[4], continued research in this area is a worthwhileendeavour.

Research on mTBI, both sports-related and non-sports-related, has been gaining prominence in theliterature. Empirical research conducted in the1980s by Bart and colleagues [5, 6] has served asthe foundation for subsequent studies. While certainpeople experience at least some of the cognitiveeffects of mTBI for the rest of their lives, others

Correspondence: Pierre Nolin, PhD, Laboratoire de Recherche Interdisciplinaire en Realite Virtuelle, Departement de psychologie, Universite du Quebec aTrois-Rivieres, C.P. 500, Trois-Rivieres, Quebec, G9A 5H7, Canada. Tel: (819) 376-5011 ext. 3544. Fax: (819) 376-5195. E-mail: pierre.nolin@uqtr.ca

ISSN 0269–9052 print/ISSN 1362–301X online � 2012 Informa UK Ltd.DOI: 10.3109/02699052.2012.698359

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

recover from it; the nature and the duration ofthe post-acute recovery process are still a matterof debate [7]. While most people with non-sports-related mTBI tend to show strong recovery over the3 months after their accident [8–14], some experi-ence persistent symptoms beyond this period. Theprevalence of post-acute symptoms is highly variablefrom study-to-study: Binder et al. (1997) [15]reported post-acute symptoms in 7–8% of cases,while Rimel et al. [6] reported 33%. These symp-toms have been known to last several months or evenyears [8, 16–20].

Two meta-analyses conducted by Belanger andVanderploeg [7] and Broglio and Puetz (2008) [21]on studies dealing with sports concussion identifiedthe impact of sports concussion on a number ofneurocognitive functions in the initial hours, daysand, in some cases, weeks post-injury. The studiespointed out deficits in four areas: (1) attention,(2) memory and learning, (3) overall cognitivefunction and (4) executive function. Generally,most participants appear to recover fully during themonths following the concussion. Some individuals,however, experience long-term effects and/or slowerrecovery. While most of the existing research hasfocused on the acute phase post-concussion, recentstudies conducted using neuroimaging techniquessupport the idea that certain athletes experience animpact over the long-term.

One of the key considerations in studying sportsconcussion is the tool used to assess the impairedcognitive functions. Standard neuropsychologicalassessment is considered sensitive to the immediateeffects of concussion in certain athletes [22]; it isrecognized as being more sensitive than regularneurological or radiological testing [23]. If neuro-psychological assessment is to detect the effectsof sports concussion, it must be sensitive to thesymptoms of mild brain injury [24–26]. Recently,Broglio et al. [27] noted that event-related potentials(ERPs) were useful in identifying areas of cognitionthat remained dysfunctional for an extended periodof time post-concussion and that would otherwise goundetected by standard neuropsychological tests.

The short- and long-term effects of sports con-cussion have been observed in several related studiesthat have been conducted using sophisticated neu-roimaging techniques such as functional magneticresonance imaging (fMRI) [28–31] and ERPs[27, 32–34]. While these technologies are essentialin demonstrating the effects of sports concussion inthe brain, they are difficult to use and are not readilytransported into ‘everyday’ environments, such asschools with sports and education programmes(SEPs). There is still a need, therefore, to furtherdevelop tools for assessing the cognitive effects of

sports concussion that are both user-friendly andsufficiently sensitive.

Virtual reality

By means of immersive virtual reality, users caninteract in real time with a 3-dimensional, computer-simulated environment that mirrors daily life[35–38]. This makes it a revolutionary tool. Usersenter the immersion by wearing a head-mounteddisplay (HMD) onto which the virtual environmentis projected. The user’s head movements are sensedby a detection system integrated into the HMD andconverted into computerized data.

Studies using immersive neuropsychologicalassessments to study people with mTBI are few.The advantages offered by virtual reality, such asgreater sensitivity and ecological validity, makeresearch in this area a promising endeavour.Testing in virtual reality gauges the true functioningof subjects in environments where they are left totheir own devices and allowed to behave naturally.Additionally, automatic scoring reduces the numberof errors that might skew the results. By its verynature, testing in virtual reality detects subtle defi-cits, which are often imperceptible in traditionalassessments [36, 37, 39–43]. The use of virtualreality techniques in attention and inhibition testingis therefore a promising avenue for improved iden-tification of cognitive deficits following sportsconcussion.

Only recently has virtual reality become a tool forstudying people with traumatic brain injury (TBI).Most studies on this topic have been conducted onadults and most have focused on a limited numberof cognitive domains, such as motor and balancerehabilitation [44–51], social problem-solving [52],community living skills [53] and car driving [54].These studies have demonstrated the advantages ofvirtual reality on adults with moderate or severe TBI.

As for studies on children with TBI, those usingvirtual reality as an assessment tool are relativelyscarce [55, 56]. Some studies have demonstrated theutility of virtual reality in cases of moderate or severeTBI [39, 40], but not the mild type (mTBI). (Theonly two studies to have used virtual reality to assesspeople with mTBI were both conducted on adults.).Slobounov et al. (2006) [57] showed that advancedvirtual reality technology, used in combination withbalance and motion tracking technologies, was ableto detect residual symptoms of concussion symp-toms in university athletes. More recently the sameresearch team (Slobounov et al. 2010) [58] studiedthe effects of concussion using fMRI in combinationwith a virtual reality paradigm in adult athletes.In summary, while virtual reality has met withsuccess in detecting subtle cognitive deficits resulting

ClinicaVR: Classroom-CPT and sports concussion 1565

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

from sports concussion, further studies are needed,both on adults and on children. To the best of theauthors’ knowledge, the present study is the first tofocus on adolescents.

The objective of this study is to compare per-formance in attention and inhibition tests (bothtraditional and virtual) between two groups ofadolescents enrolled in an SEP: those who havesustained a concussion and those who have not.

Method

Participants

Group 1 (sports concussion) was composed of25 adolescents enrolled in an SEP at the Academieles Estacades in Trois-Rivieres, Quebec, Canada.All participants suffered a concussion while playingtheir sport at some time in the 2 years precedingthe study. The time between concussion andevaluation ranged from 1–24 months(mean¼ 11.71 months, SD¼ 8.80 months). Themost common sports in which the participants hadbeen injured were hockey, basketball and soccer.The sports concussion group was made up of 15boys and 10 girls, with a mean age of 13.64 years(SD¼ 1.11 years). All participants in the groupsuffered a Grade 1 concussion (Cantu ConcussionGrading Guideline, 2001 [59]). By this definition,they experienced no loss of consciousness (LOC)due to the injury and any post-traumatic amnesialasted under 30 minutes. Concussions were of thesimple type, which is defined by the Summary andAgreement Statement of the 2nd InternationalConference on Concussion in Sport (McCroryet al. 2005 [60]) as a concussion for whichassociated symptoms resolved within 7–10 dayswithout complications. Inclusion criteria alsoincluded commonly accepted clinical symptoms ofmTBI taken from SCAT2 (http://www.cces.ca/files/pdfs/SCAT2[1].pdf) such as headache, ‘pressure inhead’, neck pain, nausea or vomiting, dizziness,difficulty concentrating, irritability, etc. [14].

Group 2 (control) consisted of 25 adolescents whowere also enrolled in the SEP but who did notsustain a sports concussion in the 2 years precedingthe study. This group was also made up of 15 boysand 10 girls, this time with a mean age of 13.76 years(SD¼ 1.13 years).

All participants followed the normal SEP curric-ulum and had no history of neurological, psycho-logical or learning disabilities. The two groups wereequivalent with respect to age [t(48)¼�0.38,p< 0.05] and gender. All participants in the studywere francophone.

Measures

Sense of presence and cybersickness questionnaires. Inorder to verify the quality of the immersion duringthe virtual neuropsychological test, we administeredtwo questionnaires to each participant. The ques-tionnaires covered two important factors in virtual-reality studies: sense of presence and cybersickness.Sense of presence refers to the propensity to respondto virtually generated sensory data as if they werereal [61]. Cybersickness denotes symptoms that maybe felt during or after the participant’s experience invirtual reality, such as nausea or eye strain.

The first questionnaire, the PresenceQuestionnaire, was developed by Witmer andSinger (1998) [62] and revised by the UQOCyberpsychology Laboratory, 2004 [63]. It consistsof 18 questions answered on a Likert-type scaleranging from 0 (‘not at all’) to 7 (‘completely’). Forthe purposes of this study, a customized version ofthe questionnaire was used with eight items expresslyadapted for this adolescent study population. Thepsychometric validity of the questionnaire is sup-ported by the Cronbach’s alpha of 0.78 for this studysample.

The second questionnaire, the Post-ExposureSymptom Checklist, was developed by Kennedyet al. (1993) [64] and later translated into Frenchand revised by the UQO CyberpsychologyLaboratory (2002) [65]. It consists of 16 itemsanswered on a Likert-type scale ranging from 0(‘none’) to 3 (‘severe’). The variables measured bythis questionnaire are the number of cybersicknessand the mean intensity of cybersickness as experi-enced by participants. The psychometric validity ofthe questionnaire is supported by a Cronbach’salpha of 0.87 for this study sample. The complete listof symptoms can be found in Table I.

Neuropsychological test: Traditional

To test attention and inhibition functions in atraditional setting, the VIGIL-CPT (VigilContinuous Performance Test, Cegalis & Bowlin,1991 [66]) was used. In this computerized test,letters appear one at a time in the centre of a screen,changing at an interval that is kept constantthroughout the test. The participant is required toclick the mouse each time the letter K appears afterbeing immediately preceded by the letter A. The 6-minute test presents a total of 300 stimuli, 60 ofwhich require a response. In clinic and in research,the VIGIL-CPT is a recognized measure of sus-tained attention, vigilance, impulsivity and reactiontime [67]. The three variables measured in thetraditional test were (1) the number of omissions(i.e. failure to respond to the letter K when imme-diately preceded by the letter A), (2) the number

1566 P. Nolin et al.

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

of commissions (i.e. responding to the letter K whennot preceded by the letter A or responding toanother letter) and (3) the mean reaction time inmilliseconds. For each of the three variables, perfor-mance can be tracked over time by splitting the6-minute test into three blocks of 2 minutes each. Acomparison of the scores in the three blocks serves asa measure of the effects of fatigue as the testprogresses.

Neuropsychological test: Virtual

The immersive test of attention and inhibition wasfirst developed by Rizzo et al. [39]. It was revised bythe Digital MediaWorks team (http://www.dmw.ca/)under the name ClinicaVR: Classroom-CPT. The testis identical to the traditional VIGIL-CPT exceptfor the environment in which it is administered:instead of being presented on a computer screen,

the stimuli appear on a whiteboard situated in avirtual classroom. The virtual classroom featuresobjects and people commonly found in real class-rooms, such as the blackboard, desks, a teacher andstudents. Participants were immersed in the virtualenvironment by wearing an Emagin Z800 HMDwith the ability to monitor the wearer’s head move-ments. Participants were able to look 360� aroundthemselves as well as up and down in the virtualenvironment. Typical classroom sounds were playedto the participant through headphones integratedinto the HMD. Throughout the duration of thevirtual test, the wearer experienced auditory andvisual distractions typical of a real classroom, such asa knock at the door, a bell announcing the end ofclass, children laughing outside and a visit from theprincipal. The four variables measured in the virtualtest were (1) the number of omissions, (2) thenumber of commissions, (3) the mean reaction time

Table I. Frequency of cybersickness symptoms experienced during the ClinicaVR: Classroom test and resultsfrom group comparisons.

Symptoms

Frequency

0 1 2 3None Slight Moderate Severe Chi square

General discomfort Gr 1Gr 2

12 10 3 0 7.22*21 3 1 0

Fatigue Gr 1Gr 2

4 13 6 2 8.18*10 14 1 0

Headache Gr 1Gr 2

14 6 3 2 5.5818 7 0 0

Eye strain Gr 1Gr 2

4 10 8 3 3.858 12 4 1

Difficulty focusing Gr 1Gr 2

12 8 2 3 5.8617 8 0 0

Increased salivation Gr 1Gr 2

14 9 2 0 6.78*22 3 0 0

Sweating Gr 1Gr 2

24 1 0 0 1.0922 3 0 0

Nausea Gr 1Gr 2

20 4 1 0 5.5625 0 0 0

Difficulty concentrating Gr 1Gr 2

11 11 3 0 9.39**21 4 0 0

‘Fullness of the head’ Gr 1Gr 2

10 8 7 0 8.44*16 9 0 0

Blurred vision Gr 1Gr 2

15 5 3 2 5.7120 5 0 0

Dizziness with eyes open Gr 1Gr 2

18 5 1 1 5.5224 1 0 0

Dizziness with eyes closed Gr 1Gr 2

17 4 2 2 4.7822 3 0 0

Vertigo Gr 1Gr 2

22 3 0 0 1.0924 1 0 0

‘Stomach awareness’ Gr 1Gr 2

23 2 0 0 0.2222 3 0 0

Burping Gr 1Gr 2

24 1 0 0 1.0225 0 0 0

* p< 0.05; ** p<0.01.

ClinicaVR: Classroom-CPT and sports concussion 1567

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

in milliseconds and (4) the number of left–right headmovements (i.e. those made along the horizontalaxis). As in the traditional VIGIL-CPT, it waspossible to obtain performance scores for the6-minute test in three blocks of 2 minutes, therebytracking performance over the course of the test.

Procedure

Students participated individually in testing sessionsduring regular class hours. The order of the tradi-tional and virtual tests was counterbalanced acrossparticipants to prevent skewing of the results due topractice or fatigue effects. The data were collectedover a period of 5 weeks. The study procedure wasapproved by the Human Research Ethics Committeeof the University of Quebec at Trois-Rivieres.

Results

Sense of presence and cybersickness questionnaires

This section presents the results for sense of pres-ence and cybersickness experienced by participantsin the virtual test.

On average, individuals in the sports concussiongroup experienced roughly the same sense of pres-ence as those in the control group (see Table II); theANOVA showed no difference between the groupsfor this variable [F(1,48)¼1.61, p>0.05]. All par-ticipants felt ‘moderately’ present during the virtualtest. Since there was no correlation between sense ofpresence score and the virtual test scores (omissionsr¼ 0.03, p> 0.05; commissions r¼�0.04, p> 0.05;reaction time r¼ 0.06, p> 0.05; left–right headmovements, r¼�0.06, p> 0.05), this study wasable to compare performance between the twogroups for the immersive test.

There was no correlation between cybersicknessscore and virtual test scores (omissions r¼ 0.14,p> 0.05; commissions r¼�0.15, p>0.05; reactiontime r¼0.22, p> 0.05; left–right head movements,r¼0.08, p>0.05). The cybersickness intensityscores showed that all participants, on average,experienced ‘mild’ cybersickness. Table I shows thefrequency of reported cybersickness by group. Thesports concussion group reported more symptomsof cybersickness [F(1,48)¼12.73, p< 0.001] thanthe control group (see Table II). The intensity ofthe cybersickness was higher for the sports concus-sion group [F(1,48)¼23.50, p<0.001] than forthe control group (see Table II). The score from thecybersickness questionnaire was then co-varied incomparative analysis of the groups for the virtual testscores.

Group comparisons: Traditional vs immersive

neuropsychological test

This section presents the results of group compar-isons performed on the traditional and virtualneuropsychological test scores. First, results aregiven for ANOVAs performed on the three variablesmeasured in the traditional VIGIL-CPT: omissions,commissions and reaction time. Secondly, results aregiven for the ANCOVAs performed on the fourvariables measured in the virtual version of the CPT:omissions, commissions, reaction time and left–righthead movements; scores from the cybersicknessquestionnaire were co-varied in these analyses.Table II shows the means and standard deviationsfor all variables as well as the results from thestatistical analysis.

Table II shows that there was no significantdifference between the two groups for any of thethree variables in the traditional VIGIL-CPT.

Table II. Scores for traditional VIGIL-CPT and ClinicaVR: Classroom-CPT and results from group comparisons.

Group 1Sports concussion

(n¼ 251)

Group 2Control(n¼ 25)

M SD M SDF Co-variable

(Cybersickness) F

Partial Etasquared

Sense of presence 3.44 0.94 3.42 0.73 — 0.01 0.00Cybersickness (number of symptoms) 6.24 3.63 3.32 1.89 12.73*** 0.21Cybersickness (intensity) 1.57 0.44 1.23 0.14 — 13.79*** 0.22Traditional VIGIL-CPT Omissions 4.79 7.52 3.12 3.26 — 1.04 0.02Traditional VIGIL-CPT Commissions 4.54 3.59 6.40 4.53 — 2.52 0.05Traditional VIGIL-CPT Reaction time 324.49 58.38 320.33 39.00 — 0.09 0.00ClinicaVR: Classroom Omissions 3.36 3.26 1.96 2.17 0.02 2.23 0.05ClinicaVR: Classroom Commissions 5.96 6.47 3.88 2.65 4.10* 5.34* 0.10ClinicaVR: Classroom Reaction Time 377.23 42.54 380.08 46.67 3.64 t 1.22 0.03ClinicaVR: Classroom Left–right head movements 72.28 69.98 35.04 21.29 0.53 6.64** 0.12

*p< 0.05; **p< 0.01; ***p< 0.001; t¼ statistical tendency between 0.06–0.09.

1568 P. Nolin et al.

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

However, significant differences did exist betweenthe groups for two of the four variables in thevirtual version of the CPT: number of commissionerrors and number of left–right head movements.(These differences were observed after the effect ofcybersickness was removed.)

Since significant differences were observed forcommission errors and head movements in thevirtual test, supplementary analyses were conductedon the respective groups’ test scores for thesevariables over the three 2-minute blocks. (Recallthat scores from the VIGIL-CPT, in both thetraditional and the immersive versions, can beexamined by dividing the 6-minute test into threeblocks of 2 minutes each, thus obtaining a measureof performance over time.)

Figure 1 shows the number of commission errorsby group for the three time blocks. Since there werenot enough participants for a multivariate analysis,an ANOVA was used to compare performance

between the groups in each time block. ABonferroni correction was applied with a significancethreshold of 0.017 to avoid Type I errors.The analyses showed no significant differencebetween the two groups; the significance thresholdof 0.04 was reached only in Block 3. In summary,the following trend emerged from the statisticalanalysis: the control group tended to make fewercommission errors over time, whereas the sportsconcussion group made the same number of errorsall throughout the test.

Figure 2 shows the number of left-right headmovements for the three time blocks by group, usingthe same statistical approach as above. The analysesshowed a significant difference between the groupsfor left–right head movements in Block 1[F(1,48)¼4.27, p¼ 0.01] and in Block 3[F(1,48)¼8.84, p¼0.019] and a statistical trendwas observed in Block 2 [F(1,48)¼ , p¼ 0.06]. It canbe concluded that the sports concussion participants

18.56

23.43

30.24

9.811.08

14.16

0

5

10

15

20

25

30

35

Block 1 Block 2 Block 3

Num

ber o

f le

ft-rig

ht h

ead

mov

em

ent

s

Group 1

Group 2

Figure 2. Number of left–right head movements in ClinicaVR: Classroom-CPT by time block.

2.212.05

1.89

1.57

1.38

0.71

0

0.5

1

1.5

2

2.5

Num

ber

of c

omm

issi

on e

rror

s

Group 1

Block 1 Block 2 Block 3

Group 2

Figure 1. Number of commission errors in ClinicaVR: Classroom-CPT by time block.

ClinicaVR: Classroom-CPT and sports concussion 1569

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

made more head movements than control partici-pants throughout the duration of the test.

Discussion

The goal of the present study was to determine if anattention and inhibition test administered in virtualreality (ClinicaVR: Classroom-CPT) was sensitiveenough to detect the subtle effects of sports concus-sion in a group of adolescents enrolled in an SEP.The virtual environment represents a revolutionaryassessment tool for neuropsychological research, yetit remains under-exploited, especially in studies onchildren with mTBI. These results support thehypothesis that ClinicaVR: Classroom-CPT is sensi-tive enough to detect subtle effects in the sportsconcussion group when compared to the controlgroup.

The virtual test specifically revealed a differencebetween the groups in the number of commissionerrors and the number of left–right head movements;both these variables may be interpreted as deficitsof inhibition. The data are consistent with previ-ous studies that have demonstrated a link betweensports concussion and cognitive deficits of thistype [26, 68–70].

Given that subtle cognitive deficits typicallyimprove within 3–6 months of the accident [7, 71,72], it is somewhat surprising that they weredetected in these sports concussion participants.Few studies have dealt with the long-term effects ofsports concussion. Further research into this topicis needed and the sensitivity of the tool will be a keyconsideration in this. A handful of studies usingfMRI [30] and event-related potentials [27, 34]point to the existence of long-term cognitive effectsof sport concussion in adults.

Sports-concussion group experienced more cyber-sickness than the control group, both in the numberof cybersickness and in the intensity thereof.Specifically, the sports concussion group reportedthe following five symptoms more frequently: gen-eral discomfort, fatigue, increased salivation, diffi-culty concentrating and ‘fullness of the head’.Generally speaking, the nature of these subjectivelyreported symptoms is consistent with the kind ofcognitive deficits detected by the virtual test.

Attention and inhibition problems among adoles-cents were found to be greater in the virtual CPTthan in the traditional version; in fact, the traditionalversion showed no difference between the groups forthese deficits. The immersive assessment approachthus appears more sensitive to the effects of sportsconcussion than the traditional approach. It ispossible that the superior sensitivity of ClinicaVR:

Classroom-CPT may be due to the ‘ecological’ nature

of the tool; this means that it closely represents real-life experiences, thus placing greater demands on theparticipant’s capacities of attention and inhibition.The question of whether the ecological representa-tiveness of the tool accounts for these results is onethat must be addressed by future research. Suchresearch should illustrate, first, the relationshipbetween the two assessment approaches (traditionaland virtual) and, secondly, the relationships betweenthese approaches and existing ecological measuressuch as the Behavior Rating Inventory of ExecutiveFunction [73] and observed behaviours in real-lifeclassrooms. Perhaps it is not the case that the virtualCPT is more ecological than the traditional version,but that it is simply more complex given that itrequires participants to process more information.At the very least, it can be said that ClinicaVR:

Classroom-CPT seems to respond well to sensitivitycriteria for detecting the subtle effects of sportsconcussion.

The results contribute to a body of existingliterature that has demonstrated the utility of thevirtual classroom in studying children and adoles-cents with ADHD [74–79]. Importantly, the virtualclassroom also has the benefit of being enjoyable toparticipants [78, 80]. The use of ClinicaVR:

Classroom-CPT need not be limited to the populationin our study: it would be equally advantageous infuture clinical neuropsychological research on otherpopulations.

The results generated in the present study couldserve as the basis for a subsequent prospective study.The study would be designed in such a way that thesensitivity of ClinicaVR: Classroom-CPT in detectingthe subtle effects of sports concussion would beverified by assessing, at the beginning of the schoolyear, basic functions in all students enrolled in theSEP, and by subsequently performing short- andlong-term reassessments on those students whosustain concussions in the future. This proposedstudy design is consistent with recommendationsmade in related studies [23] and could aid inpreventing faulty attribution, i.e. the phenomenonof injured students being viewed as havingother diagnosis such as hyperactivity or conductdisorders [81].

Conclusion

Neuropsychological assessment, used as a tool fordetecting cognitive deficits resulting from sportsconcussion, is recommdended by experts in thefield as part of a sports concussion protocol [82–84].Assessments based on computerized tests are rec-ommended over traditional ones (e.g. paper andpencil tests) because they present a number of

1570 P. Nolin et al.

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

advantages [23, 85]). One such advantage, easyreproducibility, makes it possible to conduct multi-ple studies at the same time or in different locations;another is the fact that no specialized training isrequired to operate the equipment. ClinicaVR:

Classroom-CPT meets all these criteria and is alsomuch more affordable and user-friendly than neu-roradiological technologies. For all these reasons,ClinicaVR: Classroom-CPT is a compelling choice forthe study of sports concussion.

Acknowledgements

The authors would like to thank Albert Rizzo, fromUniversity of Southern California - Institute forcreative technologies, and Roman Mitura and DeanKlimchuk, from Digital MediaWorks, for allowingthem to use the Virtual Classroom and ClinicaVR:Classroom. They would also like to thank thefollowing people for their valuable contributions tothis research project: the staff of Academie lesEstacades de Trois-Rivieres, particularly RosemarieBoucher, Luce Mongrain and Michel Boutin; DrDavid Fecteau of the Centre Hospitalier Regional deTrois-Rivieres; Fernand Bouchard of the St.Maurice Physiotherapy Clinic; and all researchassistants from our laboratory (Julie Baillargeon,Caroline Cliche, Rosalie Coderre Sevigny, JessicaCouture, Anne Drouin-Germain, Andree-AnneDurocher, Isabelle Fournier, Nicolas Leblond,Valerie Noel, Roxan Noel, Pacifiee Rodrigue,Mikako Tsuchigahata, Emanuelle Valliomee).

Decleration of Interest: The authors report noconflicts of interest. This research was made possiblethanks to the Fond de Developpement Academiquedu Reseau de l’Universite du Quebec (FODAR) andFonds Institutionnel de Recherche (FIR) del’Universite du Quebec a Trois-Rivieres (UQTR).

References

1. American Academy of Neurology Practice Parameter. Themanagement of concussion in sports (summary statement).Report of the Quality Standards Subcommittee. Neurology1997;48:581–585.

2. Gessel LM, Fields SK, Collins CL, Dick RW, Comstock RD.Concussions among United States High School and Collegiateathletes. Journal of Athletic Training Med Sports Systems2007;42:495–503.

3. Sosin DM, Sniezek JE, Thurman DJ. Incidence of mild andmoderate brain injury in the United States, 1991. Brain Injury1996;10:47–54.

4. Thurman DJ, Alverson C, Dunn KA, Guerrero J, Sniezek JE.Traumatic brain injury in the United States: A public healthperspective. Journal of Head Trauma Rehabilitation 1999;14:602–615.

5. Barth JM, Stephen N, Giordani B, Rimel RN, Jane JA,Boll TJ. Neuropsychological sequelae of minor head injury.Neurosurgery 1983;13:529–533.

6. Rimel RW, Giordani B, Bruno Barth JT, Boll TJ, Jane JA,Disability caused by minor head injury. Neurosurgery1981;9:221–228.

7. Belanger HG, Vanderploeg RD. The neuropsychologicalimpact of sports-related concussion: A meta-analysis. Journalof the International Neuropsychological Society 2005;11:345–357.

8. Dikmen S, McLean A, Temkin N. Neuropsychological andpsychosocial consequences of minor head injury. Journal ofNeurology, Neurosurgery & Psychiatry 1986;49:1227–1232.

9. Dikmen SS, Machamer JE, Winn HR, Temkin NR.Neuropsychological outcome at 1-year post head injury.Neuropsychology 1995;9:80–90.

10. Dikmen SS, Ross BL, Machamer JE, Temkin NR. One yearpsychosocial outcome in head injury. Journal of theInternational Neuropsychological Society 1995;1:67–77.

11. Gentilini M. Neuropsychological evaluation of mild headinjury. Journal of Neurology, Neurosurgery & Psychiatry1985;48:137–140.

12. Gronwall DM, Sampson H. The psychological effects ofconcussion. Oxford, England: Auckland U Press; 1974.

13. Levin HS, Gary Jr HE, High Jr WM, Mattis S, Ruff RM,Eisenberg HM, Marshall LF, Tabaddor K. Minor headinjury and the postconcussional syndrome: Methodologicalissues in outcome studies. New York: Oxford UniversityPress; 1987.

14. Levin HS, Grafman J, Eisenberg HM. Neurobehavioralrecovery from head injury. New York, NY: OxfordUniversity Press; 1987.

15. Binder LM, Rohling ML, Larrabee GJ. A review of mild headtrauma. Part I: Meta-analytic review of neuropsychologicalstudies. Journal of Clinical and ExperimentalNeuropsychology 1997;19:421–431.

16. Alves W, Macciocchi SN, Barth JT. Postconcussive symp-toms after uncomplicated mild head injury. The Journal ofHead Trauma Rehabilitation 1993;3:48–59.

17. Hartlage LC, Durant-Wilson D, Patch PC. Persistentneurobehavioral problems following mild traumatic braininjury. Archives of Clinical Neuropsychology 2001;16:561–570.

18. Powell TJ, Collin C, Sutton K. A follow-up study of patientshospitalized after minor head injury. Disability andRehabilitation: An International, Multidisciplinary Journal1996;18:231–237.

19. Alexander GE. Factors related to patient report of cognitivedifficulties following head injury. Dissertation AbstractsInternational 1992;52(12-B, Pt1):6651.

20. Deb S, Lyons I, Koutzoukis C. Neurobehavioural symptomsone year after a head injury. British Journal of Psychiatry1999;174:360–365.

21. Broglio SP, Puetz TW. The effect of sport concussion onneurocognitive function, self-report symptoms and posturalcontrol: A meta-analysis. Sports Medicine, 2008;38:53–67.

22. Iverson GL. Evidence-based neuropsychological assessmentin sport-related concussion. In: FM Webbe, editor. TheHandbook of Sport Neuropsychology (pp. 131–153)Springer Publishing Co.; 2011.

23. Schatz P. Computerized neuropsychological assessment insport. In: FM Webbe, editor. The Handbook of SportNeuropsychology (pp. 173–186). Springer Publishing Co.;2011.

24. Gronwall D, Wrightson P. Delayed recovery of intellectualfunction after minor head injury. The Lancet1974;14:605–609.

ClinicaVR: Classroom-CPT and sports concussion 1571

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

25. Middleton J. Annotation: Thinking about head injury inchildren. Journal of Child Psychology and Psychiatry1989;30:663–670.

26. Nolin P, Mathieu F. L’importance de la sensibilite desmesures neuropsychologiques dans l’identification des defi-cits de l’attention chez des enfants ayant subi un traumatismecraniocerebral leger. Revue de Neuropsychologie 2001;11:23–38.

27. Broglio SP, Moore RD, Hillman CH. A history of sport-related concussion on event-related brain potential correlatesof cognition. International Journal of Psychophysiology2011;82:16–23.

28. McAllister TW, Saykin AJ, Flashman LA, Sparling MB,Johnson SC, Guerin SJ, Mamourian AC, Weaver JB,Yanofsky N. Brain activation during working memory1 month after mild traumatic brain injury: A functionalMRI study. Neurology 1999;53:1300–1308.

29. McAllister TW, Sparling MB, Flashman LA, Saykin AJ.Neuroimaging findings in mild traumatic brain injury.Journal of Clinical and Experimental Neuropsychology2001;23:775–791.

30. Chen CJ, Toh SC, Fauzy WM. The theoretical frameworkfor designing desktop virtual reality-based learning environ-ments. Journal of Interactive Learning Research 2004;15:147–176.

31. Ptito A, Chen J-K, Johnston KM. Contributions of functionalmagnetic resonance imaging (fMRI) to sport concussionevaluation. NeuroRehabilitation 2007;22:217–227.

32. Dupuis F, Johnston KM, Lavoie M, Lepore F, Lassonde M.Concussions in athletes produce brain dysfunction asrevealed by event-related potentials. NeuroReport: ForRapid Communication of Neuroscience Research 2000;11:4087–4092.

33. Leblanc J, De Guise E, Gosselin N, Feyz M. Comparison offunctional outcome following acute care in young, middle-aged and elderly patients with traumatic brain injury. BrainInjury 2006;20:779–790.

34. De Beaumont L. Long-term effects of sports concussion.Psychologie - recherche et intervention. Montreal: Universitede Montreal; 2010.

35. Pratt DR, Zyda M, Kelleher K. Virtual reality: In the mind ofthe beholder. IEEE Computer Society Press 1995;28:17–19.

36. Tarr MJ, Warren WH. Virtual reality in behavioral neuros-ciences and beyond. Nature Neuroscience 2002;5:1089–092.

37. Trepagnier CG. Virtual environments for the investigationand rehabilitation of cognitive and perceptual impairments.NeuroRehabilitation 1999;12:63–72.

38. Zhang J, Sun Y, Wang X, Wu G. How do learners utilize thecourse package and learning support services in distancelearning: A survey on the learning processes. InternationalJournal on E-Learning 2003;2:17–23.

39. Rizzo AA, Buckwalter JG, Bowerly T, van der Zaag C,Humphrey L, Neumann U, Chua C, Kyriakakis C, vanRooyen A, Sisemore D. The virtual classroom: A virtualreality environment for the assessment and rehabilitation ofattention deficits. CyberPsychology & Behavior 2000;3:483–499.

40. Rizzo AA, Schultheis M, Kerns KA, Mateer C. Analysis ofassets for virtual reality applications in neuropsychology.Neuropsychological Rehabilitation 2004;14:207–239.

41. Theriault M, De Beaumont L, Tremblay S, Lassonde M,Jolicoeur P. Cumulative effects of concussion in athletesrevealed by electrophysiological abnormalities on visualworking memory. Journal of Clinical and ExperimentalNeuropsychology 2011;33:30–41.

42. Nolin P, Martin C, Bouchard S. Assessment of inhibitiondeficits with the virtual classroom in children with traumatic

brain injury: A pilot-study. Annual Review of CyberTherapyand Telemedicine 2009;7:240–242.

43. Martin C, Nolin P. La realite virtuelle comme nouvelleapproche evaluative en neuropsychologie: L’exemple de laclasse virtuelle avec des enfants ayant subi un traumatismecranio-cerebral. A.N.A.E. Approche Neuropsychologiquedes Apprentissages chez l’Enfant 2009;21:28–32.

44. Rose FD, Brooks BM, Rizzo AA. Virtual reality in braindamage rehabilitation: Review. CyberPsychology & Behavior2005;8:241–262.

45. Larson EB, Ramaiya M, Zollman FS, Pacini S, Hsu N,Patton JL, Dvorkin AY. Tolerance of a virtual realityintervention for attention remediation in persons withsevere TBI. Brain Injury 2011;25:274–281.

46. Levin MF. Can virtual reality offer enriched environments forrehabilitation? Expert Review of Neurotherapeutics 2011;11:153–155.

47. Mumford N, Wilson PH. Virtual reality in acquired braininjury upper limb rehabilitation: Evidence-based evaluationof clinical research. Brain Injury 2009;23:179–191.

48. Mumford N, Duckworth J, Thomas PR, Shum D,Williams G, Wilson PH. Upper limb virtual rehabilitationfor traumatic brain injury: Initial evaluation of the elementssystem. Brain Injury 2010;24:780–791.

49. Sveistrup H, McComas J, Thornton M, Marshall S,Finestone H, McCormick A, Babulic K, Mayhew A.Experimental studies of virtual reality-delivered comparedto conventional exercise programs for rehabilitation.CyberPsychology & Behavior 2003;6:245–249.

50. Thornton KE, Carmody DP. Electroencephalogram biofeed-back for reading disability and traumatic brain injury. Childand Adolescent Psychiatric Clinics of North America2005;14:137–162.

51. Thornton M, Marshall S, McComas J, Finestone H,McCormick A, Sveistrup H. Benefits of activity and virtualreality based balance exercise programmes for adults withtraumatic brain injury: Perceptions of participants and theircaregivers. Brain Injury 2005;989–1000.

52. Hanten G, Cook L, Orsten K, Chapman SB, Li X, Wilde EA,Schnelle KP, Levin HS. Effects of traumatic brain injury on avirtual reality social problem solving task and relationsto cortical thickness in adolescence. Neuropsychologia2011;49:486–497.

53. Yip BCB, Man DWK. Virtual reality (VR)-based communityliving skills training for people with acquired brain injury: Apilot study. Brain Injury 2009;23:1017–1026.

54. Wald JL, Liu L, Reil S. Concurrent validity of a virtual realitydriving assessment for persons with brain injury.CyberPsychology & Behavior 2000;3:643–654.

55. Penn PR, Rose FD, Johnson DA. Virtual enriched environ-ments in paediatric neuropsychological rehabilitation follow-ing traumatic brain injury: Feasibility, benefits andchallenges. Developmental Neurorehabilitation 2009;12:32–43.

56. Yen Hwee-Ling WJT. Rehabilitation for traumatic braininjury in children and adolescents. Annals Academy ofMedicine 2007;36:62–66.

57. Slobounov S, Slobounov E, Newell K. Application of virtualreality graphics in assessment of concussion.CyberPsychology & Behavior 2006;9:188–191.

58. Slobounov SM, Zhang KD, Pennell D, Ray W, Johnson P,Sebatianelli W. Functional abnormalities in normally appear-ing athletes following mild traumatic brain injury: A func-tional MRI study. Experimental Brain Research2010;202:341–354.

59. Cantu RC. Posttraumatic retrograde and anterograde amne-sia: pathophysiology and implications in grading and safereturn to play. Journal of Athletic Training 2001;36:244–248.

1572 P. Nolin et al.

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.

60. McCrory P, Johnston K, Meevwisse W, Aubry M, Cantu R,Dvorak J, et al. Summary and agreement statement of the2nd international conference on concussion in sport, Prague2004. Clinical Journal of Sport Medicine 2005;15:48–55.

61. Sanchez-Vives MV, Slater M. From presence to conscious-ness through virtual reality. Nature Reviews Neuroscience2005;6:332–339.

62. Witmer BG, Singer MJ. Measuring presence in virtual environ-ments: A presence questionnaire. Presence 1998;7:225–240.

63. UQO Cyberpsychology Laboratory. Traduction du PresenceQuestionnaire de Witmer & Singer. Gatineau :Universite duQuebec en Outaouais; 2004.

64. Kenedy RS, Lane NE, Berbaum KS, Lilienthal MG.Simulator sickness questionnaire : An enhaced method forquantifying simulator sickness. International Journal ofAviation Psychology 1993;3:203–220.

65. UQO Cyberpsychology Laboratory. Traduction du Post-Exposure Symptom Checklist. de Kenedy, et al. Gatineau:Universite du Quebec en Outaouais; 2002.

66. Cegalis J, Bowlin J. Vigil : software for the assessment ofattention. Nashua, NH : Forthought; 1991.

67. Egeland J, Kovalik-Gran I. Measuring several aspects ofattention in one test: The factor structure of Conners’sContinuous performance test. Journal of Attention Disorders2010;13:339–346.

68. Fazio VC, Lovell MR, Pardini JE, Collins MW. The relationbetween post concussion symptoms and neurocognitiveperformance in concussed athletes. NeuroRehabilitation2007;22:207–216.

69. McAllister WR, McAllister DE. Recovery of conditioned fearby a single postextinction shock: Effect of similarity of shockcontexts and of time following extinction. Learning &Behavior 2006;34:44–49.

70. Belanger HG, Vanderploeg RD, Curtiss G, Warden DL.Recent neuroimaging techniques in mild traumatic braininjury. The Journal of Neuropsychiatry and ClinicalNeurosciences 2007;19:5–20.

71. Schretlen DJ, Shapiro AM. A quantitative review of theeffects of traumatic brain injury on cognitive functioning.International Review of Psychiatry 2003;15:341–349.

72. Belanger HG, Curtiss G, Demery JA, Lebowitz BK,Vanderploeg RD. Factors moderating neuropsychologicaloutcomes following mild traumatic brain injury: A meta-analysis. Journal of the International NeuropsychologicalSociety 2005;11:215–227.

73. Gioia GA, Isquith PK. Ecological assessment of executivefunction in traumatic brain injury. DevelopmentalNeuropsychology 2004;25:135–158.

74. Adams R, Finn P, Moes E, Flannery K, Rizzo AS.Distractibility in attention/deficit/hyperactivity disorder(ADHD): The virtual reality classroom. ChildNeuropsychology 2009;15:120–135.

75. Parsons TD, Bowerly T, Buckwalter JG, Rizzo AA.A controlled clinical comparison of attention performancein children with ADHA in a virtual reality classroomcompared to standard neuropsychological methods. ChildNeuropsychology 2007;13:363–381.

76. Moreau G, Guay MC, Achim A, Rizzo A, Lageix P. Thevirtual classroom: An ecological version of the ContinuousPerformance Test–A pilot study. Annual Review ofCyberTherapy and Telemedicine 2006;4:59–66.

77. Parsons TD, Bowerly T, Buckwalter JG, Rizzo AA. Acontrolled clinical comparison of attention performancein children with ADHD in a virtual reality classroomcompared to standard neuropsychological methods. ChildNeuropsychology 2007;13:363–381.

78. Pollak Y, Shomaly HB, Weiss PL, Rizzo AA, Gross-Tsur V.Methylphenidate effect in children with ADHD can bemeasured by an ecologically valid continuous performancetest embedded in virtual reality. CNS Spectrums 2010;15:125–130.

79. Rizzo AA, Bowerly T, Buckwalter JG, Klimchuk D,Mitura R, Parsons TD. A virtual reality scenario for allseasons: The virtual classroom. CNS Spectrums 2006;11:35–44.

80. Pollak Y, Weiss PL, Rizzo AA, Weizer M, Shriki L,Shalev RS, Gross-Tsur V. The utility of a continuousperformance test embedded in virtual reality in measuringADHD-related deficits. Journal of Developmental andBehavioral Pediatrics 2009;30:2–6.

81. Mittenberg W, Henry G, Heilbronner R, Enders C,Stanczak S. Comparison of the Lees-Haley Fake Bad Scale,Henry-Heilbronner Index, and Restructured Clinical Scale 1in identifying noncredible symptom reporting. The ClinicalNeuropsychologist Neuropsychology, Development andCognition 2007;22:919–929.

82. Aubry M, Cantu R, Dvorak J, Graf-Baumann T,Johnston KM, Kelly J, Lovell M, McCrory P,Meeuwisse WH, Schamasch P. Summary and agreementstatement of the 1st International Symposium on Concussionin Sport, Vienna, 2001. Clinical Journal of Sport Medicine:Statement 2002;12:6–11.

83. McCrory PM. Preparticipation assessment for head injury.Clinical Journal of Sport Medicine 2004;14:139–144.

84. De Brito SA, Hodgins S, McCrory EJP, Mechelli A, Wilke M,Jones AP, Viding E. Structural neuroimaging and theantisocial brain: Main findings and methodological chal-lenges. Criminal Justice and Behavior 2009;36:1173–1186.

85. Segalowitz SJ, Mahaney P, Santesso DL, MacGregor L,Dywan J, Willer B. Retest reliability in adolescents of acomputerized neuropsychological battery used to assessrecovery from concussion. NeuroRehabilitation 2007;22:243–251.

ClinicaVR: Classroom-CPT and sports concussion 1573

Bra

in I

nj D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y M

cgill

Uni

vers

ity o

n 10

/27/

14Fo

r pe

rson

al u

se o

nly.