Designing a proficienvalidated virtual reallaparoscopic colorec
e explorecn a
which they thought that a particular task should be included in a final technical skills curriculum.
s a was 0.715; after the second round, consensus was
SURGERally useto trainrect sucreaseregardtime oogy sufeasibla portiroom
thatment,ealityent in, theulatorperat-tagesabilitytheiraram-idualsey areculty
rtingtrain-f theed on
Reprint600 Uniada. E-m0039-6060/$ - see front matterloped 2012doi:10.1reached at 0.865. Consensus was reached for 7 basic tasks and 1 advanced suturing task. Medianexpert time and economy of movement scores were defined as benchmarks for all curricular tasks.Conclusion. This study used Delphi consensus methodology to create a curriculum for an advancedlaparoscopic procedure that is reflective of current clinical practice on an international level andconforms to current educational standards of proficiency-based training. (Surgery 2012;151:391-7.)
From the University of Toronto,a Toronto, Ontario, Canada; the Department of Surgery,b Academic MedicalCenter Amsterdam, Amsterdam, The Netherlands; and the Department of Surgery,c St. Michaels Hospital,Toronto, Ontario, Canada
Y RESIDENCY TRAINING PROGRAMS have tradition-d the operating room to teach surgical skillsees through graded responsibility under di-pervision. Because of the mandated de-in resident work hours, ethical concernsing trainees learning procedures for the firstn patients, and the advent of new technol-ch as laparoscopy, this strategy is no longere. As such, it has become necessary to shifton of residency training from the operatingto the surgical skills laboratory. Several
recent systematic reviews have demonstratedtechnical skills training in an ex vivo environwhether on a bench-top model, or a virtual r(VR) simulator, translates into an improvemoperating room performance.1-4 Moreovertechnical improvements seen on a VR simhave shown to persist for $10 cases in the oing room.5 VR simulators have several advanover bench-top simulators, including theirto simulate complications, such as bleeding,ability to automatically generate assessment peters allowing for comparison between indivand performances, as well as the fact that thable to simulate tasks at varying levels of diffiallowing for a natural gradation of training.6
With the strong body of evidence suppothe role of VR simulation in technical skillsing, it is somewhat surprising that outside orealm of research studies, few curricula bastraining using VR simulation have been deve
by the Royal College Fellowship in Education.
d for publication August 4, 2011.
requests: Vanessa N. Palter, MD, University of Toronto,versity Ave, Rm. 440, Toronto, Ontario M5G 1X5 Can-ail: Vanessa.email@example.com.
Mosby, Inc. All rights reserved.Results. In the first round of the survey, CronbachResults of the survey were sent back to participants until consensus (Cronbachs a >0.8) was reached. Across-sectional design was utilized to define the benchmark scores for the identified tasks. Nine expertsurgeons completed all identified tasks on the easy, medium, and hard settings of the simulator.Delphi approachVanessa N. Palter, MD,a Maurits Graafland, MD,b
and Teodor P. Grantcharov, MD, PhD,c Toronto, O
Background. Although task training on virtual reaoperating room, to date no VR curricula have beenpurpose of this study was to develop a proficiency-bcolorectal surgery.Methods. The Delphi method was used to determinsimulator) are relevant to teaching laparoscopic cointernational experts rated all the LapSim tasks o016/j.surg.2011.08.005cy-based, contentity curriculum fortal surgery: A
lies P. Schijven, MD, PhD, MHSc,b
o, Canada, and Amsterdam, the Netherlands
VR) simulators has been shown to transfer to theribed for advanced laparoscopic procedures. TheVR technical skills curriculum for laparoscopic
ert consensus on which VR tasks (on the LapSimtal surgery. To accomplish this task, 19Likert scale (15) with respect to the degree tofor minimally invasive procedures. Several groups
Surgery392 Palter et alhave described curricula for basic laparoscopy,including curricula for basic minimally invasivetasks, or for less complex procedures such aslaparoscopic cholecystectomy.7-9 Although thesecurricula represent an important step in definingVR curricula for minimally invasive procedures,and largely conform to current educational theo-ries regarding proficiency-based learning and dis-tributed practice, as a group, they have beendeveloped largely using local expertise. Specifi-cally, experts at 1 institution determine which tasksor components are included in the final curricu-lum. To ensure the applicability of the developedcurricula, it is essential that the final educationalproduct be reflective of practice across diverse in-stitutions. In addition, to our knowledge, no tech-nical skills curricula using VR simulation havebeen described for advanced minimally invasiveprocedures.
Laparoscopic colorectal surgery is consideredan advanced, minimally invasive procedure. Per-forming this procedure successfully involves ligat-ing large blood vessels, working in multiplequadrants of the abdomen, and creating a viableanastomosis.10 A long, variable learning curve11
has been described for laparoscopic colorectal sur-gery, which underscores the necessity of develop-ing a technical skills curriculum for learning thisprocedure, ideally in a simulated environment.The purpose of this study is 2-fold. Our first aimwas to use consensus methodology to develop atechnical skills curriculum based on VR for laparo-scopic colorectal surgery. Our second aim was todefine expert benchmarks of proficiency for thiscurriculum. We hypothesized that the Delphimethod of consensus would be a feasible meansof developing a technical skills curriculum forthis advanced minimally invasive procedure thatis reflective of international practice.
Study design. This study used Delphi methodol-ogy to obtain consensus on the essential compo-nents of a VR curriculum for laparoscopiccolorectal surgery. In addition, a cross-sectionaldesign was utilized to determine expert levels ofproficiency for the defined curriculum. The studywas approved by the local institutional reviewboard.
Participants for Delphi consensus. Participantsfor the Delphi consensus portion of the study wererequired to be leaders in their clinical field asevidenced by their role as opinion leaders within
organizations such as The American Society ofColon and Rectal Surgeons, the Society ofGastrointestinal and Endoscopic Surgeons, orother national surgery societies. Furthermore,they were required to be familiar with the VRsystem used for the curriculum. Finally, the expertswere required to be practicing surgeons who wereinvolved in training laparoscopic colon and rectalsurgery both at the resident and continuing pro-fessional development level. Twenty experts wererecruited by e-mail to respond to an on-line survey.The experts were intentionally selected to repre-sent a wide geographic area. In North America, 11experts were contacted, and in Europe, 9 expertswere contacted with the assistance of the DutchSociety for Endoscopic Surgery. Membership ofthe expert panel was not revealed to the surveyparticipants.
On-line survey. The VR system that was utilizedfor the technical skills training portion of thecurriculum was the LapSim laparoscopy trainer(Surgical Science, Gothenburg, Sweden). Con-struct validity, learning curves, and transfer of skillslearned on the LapSim have been demon-strated.5,12-17 The system consists of 11 basic tasks,10 advanced tasks, and 6 procedural tasks specificto general surgery. Not all tasks on the LapSim,however, are relevant to laparoscopic colorectalsurgery. The role of the Delphi panel was to deter-mine through expert consensus those tasks thatare relevant to teaching the technical skills re-quired to perform laparoscopic colorectal surgery.These tasks were compiled into an on-line surveyusing via Survey Monkey (Palo Alto, CA). The par-ticipants in the expert panel were required to rateeach identified task on a Likert scale from 1 to 5detailing the degree to which they agreed ordisagreed that a particular component should beincluded in a final technical skills curriculum. Re-sults of the survey were sent back to participantswith group averages and standard deviations untilexpert consensus was reached. Expert consensuswas pre-defined as Cronbachs a >0.8, which hasbeen shown to be an acceptable method of consen-sus determination.18
Final structure of the technical skills curricu-lum. After Cronbachs a >0.8 was achieved for theon-line survey, an outline of the final technicalskills portion of the curriculum was created. Spe-cific curricular tasks that over 80% of the expertsrated as either 4 (agree) or 5 (strongly agree) onthe final scale were included in the final technicalskills curriculum. The technical portion of the fi-nal curriculum will require execution of the iden-tified tasks on the easy, medium, and hard
March 2012levels of the simulator. The settings for each levelwere taken from a recent European study in which
Surgery Palter et al 393consensus was reached on defining levels for theLapSim Basic Skills 3.0 package.19 Levels for thetasks that were not discussed in the European con-sensus document (handling intestines, and stitchand square knot) were defined using an identicalconcept with local expertise.
Participants for expert benchmark levels. Nineexperts in minimally invasive surgery were identi-fied. An expert was defined as an individual whohas completed >100 advanced minimally invasiveprocedures.
Tasks. Each expert was familiarized to the sim-ulator by a member of the study team (VP or MG).Experts watched the instructional video for eachtask but did not warm up or practice on thesimulator. During the familiarization period, theexpert had opportunities to ask questions. Eachexpert completed each component of the curric-ulum on the easy, medium, and hard levelsin a predefined sequence. No assistance was pro-vided during completion of the curriculum.
Generation of expert benchmark scores. Ex-perts were scored based on the automatic assess-ment parameters generated by the simulator. Theparameters of interest were time, as well as thoseparameters related to economy of motion, specif-ically instrument angular path and path length.Expert benchmark scores for time, pathlength, and angular path were determined foreach curricular component on the 3 levels (easy,medium, and hard) by calculating the medianscore of the 10 experts.
Statistical analysis. For the second round of theconsensus survey, mean values and standard devi-ations were calculated for all LapSim tasks andwere reported back to the expert panelists. Cron-bachs a was used to determine consensus on thefinal curriculum tasks among the expert panelists.Median expert scores were calculated in order todetermine expert levels of proficiency for the iden-tified curricular tasks. All statistical analysis wasperformed on SPSS (Statistical Package for SocialSciences version 18.0, Chicago, IL).
Of the 20 experts contacted to participate in theDelphi panel, 19 responded to the first round ofthe consensus survey. The responses of 2 respon-dents were excluded from analysis, the first be-cause the respondent completed less than onethird of the survey, and the second because therespondent gave each curricular task the samescore, indicating an apparent lack of attention to
Volume 151, Number 3the survey process. After the first round of thesurvey, Cronbachs a was 0.715, indicating a lack ofconsensus. Twelve experts completed the secondround of the survey. After the second round, con-sensus was achieved with a Cronbachs a of 0.865.After consensus was reached, the LapSim tasksthat 80% of the panel rated as a 4 or 5 on the Lik-ert scale were included in the final curriculum(Table I). Easy, medium, and hard levels were de-fined for each curricular task (levels available onrequest). Nine experts completed the VR curricu-lum in its entirety. Median expert scores were setas benchmarks for each task on each of the 3 levelsof difficulty (Table II).
This study used the concept of a Delphi con-sensus methodology to develop a curriculum foran advanced laparoscopic procedure that is reflec-tive of current clinical practice on an internationallevel and conforms to current educational stan-dards of proficiency-based training. The mainpurpose of our study was to design a proceduralskills curriculum that results in improved technicalproficiency when performing laparoscopic colo-rectal surgery in the operating room. Very few VRcurricula have been developed for minimally inva-sive procedures.7-9 Moreover, these curricula havebeen developed based on local expertise. The de-sign of a VR curriculum for laparoscopic colorectalsurgery in this study represents a departure fromthese traditional methods of curricula develop-ment. The Delphi methodology was used suchthat the final curriculum represents consensusamong international experts regarding which com-ponent psychomotor tasks are essential to decreasethe learning curve associated with this procedure.This approach is particularly important becausemany of the LapSim tasks show construct valid-ity,12,19-22 but are not necessarily relevant for learn-ing laparoscopic colorectal surgery.
The advantages of the Delphi method havebeen well described in the literature.18 Althoughthe Delphi methodology has been successfully uti-lized in the development of diagnostic criteria,clinical scales, research questions, and evaluationtools for technical skills,18,23-25 to our knowledgethis is the first time that it has been used in the de-velopment of a technical skills curriculum. Theidentified tasks represent the consensus of expertsin North America and Europe. Currently, there isno consensus in the literature regarding the num-ber of experts required for a robust expert panel inthe Delphi consensus process.26,27 We elected tocontact 20 experts with an expected response of
15 because we were hoping to balance a wide vari-ety of expert opinion with selecting individuals
e of ethe
Surgery394 Palter et alTable I. LapSim tasks included in the final virtual
Camera navigationInstrument navigationCoordinationGraspingCuttingClip applyingLifting and graspingSuturingPrecision and speedHandling intestinesFine dissectionCholecystectomy part 1Cholecystectomy part 2Needle passingInterrupted stitchingRunning stitchingSquare knotSurgeons knotwho were thought to be invested in the process.During the survey process, the number of expertrespondents decreased by 29%, from 17 in the firstround, to 12 in the second round. This decrease inrespondents is relatively consistent with what hasbeen described in the literature.18,28,29 Becausethe expert panelists were chosen among a fairly ho-mogeneous group with respect to qualifications, itwas thought that this dropout rate would not alterthe final result of the panel. Although the dropoutof respondents in the panel of experts of often ex-pected with the Delphi process, using Cronbachsa to determine consensus helps to mitigate thisphenomenon, because analysis using Cronbachsa is sensitive to the number of panelists, and a de-crease in panelists is reflected by a correspondingdecrease in a. The increase in Cronbachs a onthe second round of the survey suggests that theincrease in consensus more than offset the loss ofpanel members.
Based on the results from the Delphi process, 7basic tasks and 5 suturing tasks were rated by >80%
Stitch and square knotStitch and surgeons knotInterrupted suturingRunning suturingSide-to-side anastomosisAppendectomy loop techniqueAppendectomy single staple taskAppendectomy dual staple taskAppendectomy optional staple taskPeg transferPattern cuttingEndoloopty (VR) curriculum
xperts endorsing the task infinal curriculum
Task included in thefinal VR curriculum
66 No58 No83 Yes83 Yes100 Yes92 Yes92 Yes58 No25 No100 Yes92 Yes17 No8 No50 No76 No83 No76 No67 No
March 2012of the experts as either a 4 or 5 on the Likert scale.After review of the tasks selected, we elected tochoose the stitch and square knot task as a repre-sentative suturing task, because it was felt thatincluding 5 suturing tasks was redundant andwould potentially be a source of frustration forthe trainees. Although suturing in a VR environ-ment has transfer validity, trainees at a more juniorlevel can have difficulty learning suturing on a VRsystem and report low levels of face validity for thisparticular task.30-32
Creating expert benchmark levels of proficiencyis essential in the development of a proficiency-based curriculum.33 Several studies have demon-strated that learning curves on VR simulators varybetween individuals at the same level of train-ing.34,35 All experts completed this phase of thestudy. The variability of the experts scores is re-flected in the relatively large interquartile rangesfor each benchmark level of proficiency (TableII). This variability can be explained by the factthat although the experts had all completed >100
83 Yes83 No92 No92 No76 No33 No67 No33 No50 No33 No42 No58 No
Surgery Palter et al 395Table II. Expert levels of proficiency for the curric
Level Time (s)Right instrumen
1 59 (4776) 1.79 (1.492.42 55 (4166) 1.52 (1.441.73 67 (5686) 1.89 (1.612.1
1 Right instrument time: 40 (3549) 1.654 (1.452.1Left instrument time: 41 (3451)
2 Right instrument time: 51 (4186) 1.98 (1.822.6Left instrument time: 67 (5582)
3 Right instrument time: 61 (5486) 2.46 (2.202.7Left instrument time: 90 (5596)
1 92 (75108) 1.17 (0.961.42 70 (5184) 0.87 (0.701.03 88 (67154) 1.03 (0.751.4
Volume 151, Number 3minimally invasive procedures independently, theyhad variable levels of experience on the VR simula-tor, and this variability likely resulted in several out-lying scores. Median scores, rather than meanscores, were used to minimize this outlier effecton the various benchmarks. In addition, it wassomewhat surprising that the experts medianscores did not consistently decrease as they pro-gressed through the 3 levels of difficulty for eachtask (Table II). In fact, a common pattern was forthe expert score to increase from level 1 to 2 andthen to decrease at level 3. This observation maybe related to the fact that the change in difficultyfrom level 1 to 2 might not have been great enoughto completely mitigate any familiarization andlearning effects on the simulator, whereas the in-crease in difficulty from level 1 to 3 was substantialenough to produce a decrease in performanceacross most tasks and performance measures.
1 104 (80134) 1.27 (1.131.70)2 111 (60165) 1.46 (0.843.25)3 109 (84143) 1.27 (1.162.30)
Lifting and grasping
1 103 (90115) 1.98 (1.602.28)2 98 (83112) 1.95 (1.692.28)3 122 (99139) 1.91 (1.732.43)
1 88 (66101) 2.01 (1.642.24)2 101 (74113) 2.81 (2.054.65)3 137 (111179) 5.32 (3.605.58)
1 72 (6779) 0.56 (0.530.64)2 91 (73125) 0.69 (0.500.93)3 85 (67117) 0.63 (0.520.77)
Stitch and square knot
N/A 329 (287371) 5.41 (3.546.73)
Data are presented as median values (interquartile range).tasks
Right instrumentangular path (8)
Left instrumentpath length (m)
Left instrumentangular path (8)
492 (392729) 0.66 (0.180.87) 178 (73283)
419 (368455) 0.51 (0.360.79) 134 (109225)
478 (418540) 0.73 (0.641.46) 274 (219478)
1.55 (1.352.15) 380 (297395)
2.25 (1.642.96) 408 (348583)
2.45 (2.023.11) 443 (361556)
239 (203352) 0.93 (0.861.35) 234 (195313)
180 (141226) 0.57 (0.500.71) 157 (122175)
190 (143305) 0.85 (0.621.30) 212 (127285)The LapSim VR simulator automatically com-putes performance metrics, such as time, economyof motion, and error parameters. Interestingly,construct validity seems to be limited to time andeconomy of motion scores rather than errorscores.12,19-22 Aggarwal et al36 attribute this to theinherent difficulties in defining a surgical error.As such, in this study, the parameters of time andeconomy of motion rather than error scores werechosen to represent the expert benchmark levelsof proficiency. In addition, because construct valid-ity has been determined for the majority of theLapSim tasks, repeating a construct validity assess-ment was deemed unlikely to add additional valueto this described study.
Although the use of the LapSim as the VRsimulator for this curriculum can be criticizedbecause unlike some other VR simulators, theLapSim does not contain procedural-based tasks
174 (164242) 1.41 (1.111.70) 223 (177286)
268 (106470) 1.41 (0.692.25) 217 (94345)
196 (125461) 1.44 (0.831.65) 249 (116282)
420 (346490) 2.12 (1.652.23) 445 (353498)
402 (343480) 1.93 (1.802.39) 431 (377461)
429 (382527) 2.06 (1.802.41) 428 (404505)
527 (440601) 2.64 (1.783.89) 712 (482989)
741 (5521,161) 3.34 (2.585.00) 921 (6421,223)
1,361 (9271,548) 6.19 (4.387.42) 1,549 (1,1991,702)
117 (111124) 0.37 (0.310.46) 82 (70104)
116 (99176) 0.33 (0.260.41) 74 (5196)
112 (104115) 0.28 (0.270.41) 63 (5788)
1,323 (9101,650) 5.79 (4.057.20) 1,277 (9591,723)
the acquisition of laparoscopic psychomotor skill. Am J
Surgery396 Palter et alspecifically related to laparoscopic colorectal sur-gery, it should be emphasized that the role of thistechnical skills training curriculum is to teach thepsychomotor component skills to perform an ad-vanced laparoscopic procedure. Currently, thereare no studies comparing the efficacy of VR task-based training with procedural based-training ontechnical proficiency in the operating room.Rather, the bulk of evidence seems to suggestthat basic task training on VR simulators translatesto improved performance on both the simulator aswell as to nonanalogous tasks in the operatingroom. Moreover, it is important to emphasize thatthis technical skills curriculum, based on VRtraining, is designed to teach the psychomotortasks necessary to perform laparoscopic colorectalsurgery, not the cognitive elements related toperforming the procedure, such as understandingthe flow of the operation or troubleshooting. It isalso important to note that the strength of thecurriculum is contingent on the rigor of theDelphi process and the experts who contributedtheir opinion. Unfamiliarity regarding the subtle-ties of the specific VR tasks could lead the expertsto potentially underrate or overrate exercises, thusintroducing a level of bias into the VR curriculum.We attempted to minimize this potential source ofbias by first being judicious in the expert panelselection and ensuring that panelists were familiarwith the LapSim system, not simply VR simulationin a general sense. In addition, we chose the cutofffor task inclusion as >80% of the participantsrating the task as either a 4 (agree) or a 5(strongly agree) on a Likert scale to ensure thatpotential outlying opinions were not factored intothe final consensus of the group.
This technical skills curriculum based on VRsimulation, using the LapSim, is to our knowledgethe first curriculum designed for an advancedminimally invasive procedure such as colorectalsurgery. Moreover, although the design of thecurriculum conforms to current educational theo-ries regarding proficiency-based training, our ap-proach represents a departure from the traditionalcurricula designed locally in the literature. Thetasks contained within this curriculum were de-cided on based on international expert consensusdetermined using Delphi consensus methodology.This approach represents 1 means of developing acomprehensive technical skills curriculum for lap-aroscopic colorectal surgery. After acquiring thenecessary psychomotor tasks on the VR compo-nent of the curriculum, we believe that trainees
should also participate in cognitive training, as wellas a training session in the cadaver laboratorySurg 2006;191:128-33.9. Panait L, Bell RL, Roberts KE, Duffy AJ. Designing and val-
idating a customized virtual reality-based laparoscopic skillscurriculum. J Surg Educ 2008;65:413-7.
10. Kim J, Edwards E, Bowne W, Castro A, Moon V, Gadangi P,et al. Medial-to-lateral laparoscopic colon resection: a viewbeyond the learning curve. Surg Endosc 2007;21:1503-7.
11. Schlachta CM, Mamazza J, Seshadri PA, Cadeddu M, Gre-goire R, Poulin EC. Defining a learning curve for laparo-scopic colorectal resections. Dis Colon Rectum 2001;44:217-22.
12. Eriksen JR, Grantcharov T. Objective assessment of laparo-scopic skills using a virtual reality stimulator. Surg Endosc2005;19:1216-9.
13. Woodrum DT, Andreatta PB, Yellamanchilli RK, Feryus L,Gauger PG, Minter RM. Construct validity of the LapSimlaparoscopic surgical simulator. Am J Surg 2006;191:28-32.designed to integrate their acquired motor andcognitive skills. We expect that residents trainedusing this method of systematic technical skillstraining will exhibit superior technical ability inthe operating room compared with residentstrained using conventional methods. This ap-proach has the potential to affect not only techni-cal skills acquisition, but also to improve patientcare in the operating room. The use of thiscombined approach of technical skills trainingusing VR simulation, cognitive training, and train-ing in a cadaver laboratory requires further inves-tigation and validation.
1. Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. Sys-tematic review of randomized controlled trials on the effec-tiveness of virtual reality training for laparoscopic surgery.Br J Surg 2008;95:1088-97.
2. Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR. Vir-tual reality training for surgical trainees in laparoscopic sur-gery. Cochrane Database Syst Rev 2009;1:CD006575.
3. Carter FJ, Schijven MP, Aggarwal R, Grantcharov T, FrancisNK, Hanna GB, et al. Consensus guidelines for validation ofvirtual reality surgical simulators. Simulation in Healthcare:Journal of the Society for Simulation in Healthcare 2006;1:171-9.
4. Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cre-gan P, Scott D, et al. Surgical simulation: a systematic review.Ann Surg 2006;243:291-300.
5. Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hog-man C, McClusky DA III, et al. Proficiency-based virtual re-ality training significantly reduces the error rate forresidents during their first 10 laparoscopic cholecystecto-mies. Am J Surg 2007;193:797-804.
6. Palter VN, Grantcharov TP. Simulation in surgical educa-tion. CMAJ;182:11916.
7. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A.Development of a virtual reality training curriculum for lap-aroscopic cholecystectomy. Br J Surg 2009;96:1086-93.
8. Aggarwal R, Grantcharov T, Moorthy K, Hance J, Darzi A.A competency-based virtual reality training curriculum for
March 201214. Hart R, Doherty DA, Karthigasu K, Garry R. The value of vir-tual reality-simulator training in the development of
laparoscopic surgical skills. Journal of Minimally InvasiveGynecology 2006;13:126-33.
15. Larsen CR, Grantcharov T, Aggarwal R, Tully A, SorensenJL, Dalsgaard T, et al. Objective assessment of gynecologiclaparoscopic skills using the LapSimGyn virtual reality sim-ulator. Surg Endosc 2006;20:1460-6.
16. Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T,Schouenborg L, Ottosen C, et al. Effect of virtual realitytraining on laparoscopic surgery: randomised controlledtrial. BMJ (Clin Res Ed) 2009;338:b1802.
17. Palter VN, Grantcharov TP. Simulation in surgical educa-tion. CMAJ 2010;182:1191-6.
18. Graham B, Regehr G, Wright JG. Delphi as a method to es-tablish consensus for diagnostic criteria. J Clin Epidemiol2003;56:1150-6.
19. van Dongen KW, Ahlberg G, Bonavina L, Carter FJ,Grantcharov TP, Hyltander A, et al. European consensuson a competency-based virtual reality training program forbasic endoscopic surgical psychomotor skills. Surg Endosc2011;25:166-71.
20. van Dongen KW, Tournoij E, van der Zee DC, Schijven MP,Broeders IA. Construct validity of the LapSim: can theLapSim virtual reality simulator distinguish between novicesand experts? Surg Endosc 2007;21:1413-7.
25. Palter VN, MacRae HM, Grantcharov TP. Development ofan objective evaluation tool to assess technical skill in lapa-roscopic colorectal surgery: a Delphi methodology. Am JSurg 2011;201:251-9.
26. Fink A, Kosecoff J, Chassin M, Brook RH. Consensusmethods: characteristics and guidelines for use. Am J PublicHealth 1984;74:979-83.
27. Williams PL, Webb C. The Delphi technique: a methodolog-ical discussion. J Adv Nurs 1994;19:180-6.
28. Rosengart MR, Nathens AB, Schiff MA. The identificationof criteria to evaluate prehospital trauma care using the Del-phi technique. J Trauma 2007;62:708-13.
29. Nathens AB, Cook CH, Machiedo G, Moore EE, Namias N,Nwariaku F. Defining the research agenda for surgical infec-tion: a consensus of experts using the Delphi approach.Surg Infect (Larchmt) 2006;7:101-10.
30. Palter VN, Orzech N, Aggarwal R, Okrainec A, GrantcharovTP. Resident perceptions of advanced laparoscopic skillstraining. Surg Endosc 2010;24:2830-4.
31. Muresan C 3rd, Lee TH, Seagull J, Park AE. Transfer oftraining in the development of intracorporeal suturing skillin medical student novices: a prospective randomized trial.Am J Surg 2010;200:537-41.
32. Verdaasdonk EG, Dankelman J, Lange JF, Stassen LP. Trans-
SurgeryVolume 151, Number 3
Palter et al 39721. Duffy AJ, Hogle NJ, McCarthy H, Lew JI, Egan A, Christos P,et al. Construct validity for the LAPSIM laparoscopic surgi-cal simulator. Surg Endosc 2005;19:401-5.
22. Sherman V, Feldman LS, Stanbridge D, Kazmi R, Fried GM.Assessing the learning curve for the acquisition of laparo-scopic skills on a virtual reality simulator. Surg Endosc2005;19:678-82.
23. McKinley RK, Strand J, Gray T, Schuwirth L, Alun-Jones T,Miller H. Development of a tool to support holistic genericassessment of clinical procedure skills. Med Educ 2008;42:619-27.
24. Nathens AB, Rivara FP, Jurkovich GJ, Maier RV, JohansenJM, Thompson DC. Management of the injured patient:identification of research topics for systematic review usingthe Delphi technique. J Trauma 2003;54:595-601.fer validity of laparoscopic knot-tying training on a VR sim-ulator to a realistic environment: a randomized controlledtrial. Surg Endosc 2008;22:1636-42.
33. Stefanidis D, Heniford BT. The formula for a successful lap-aroscopic skills curriculum. Arch Surg 2009;144:77-82.
34. Grantcharov TP, Funch-Jensen P. Can everyone achieve profi-ciency with the laparoscopic technique? Learning curve pat-terns in technical skills acquisition. Am J Surg 2009;197:447-9.
35. Schijven MP, Jakimowicz J. The learning curve on the XitactLS 500 laparoscopy simulator: profiles of performance.Surg Endosc 2004;18:121-7.
36. Aggarwal R, Grantcharov TP, Eriksen JR, Blirup D, Kristian-sen VB, Funch-Jensen P, et al. An evidence-based virtual re-ality training program for novice laparoscopic surgeons.Ann Surg 2006;244:310-4.
Designing a proficiency-based, content validated virtual reality curriculum for laparoscopic colorectal surgery: A Delphi a ...MethodsStudy designParticipants for Delphi consensusOn-line surveyFinal structure of the technical skills curriculumParticipants for expert benchmark levelsTasksGeneration of expert benchmark scoresStatistical analysis