The Impact of External Feedback onComputer-Assisted Learning for Surgical

Technical Skill TrainingDavid A. Rogers, MD, Augusta, Georgia, Glenn Regehr, PhD, Toronto, Ontario, Canada,Thomas R. Howdieshell, MD, Karen A. Yeh, MD, Ellen Palm, MSMI, Augusta, Georgia

BACKGROUND: Computer-assisted learning (CAL)offers a number of potential advantages for sur-gical technical skills teaching. The purpose ofthis study was to evaluate the impact of individu-alized external feedback on surgical skill acquisi-tion when a CAL package is used for instruction.

METHODS: Freshman and sophomore studentsparticipated in a 1-hour CAL session designed toteach them how to tie a two-handed squareknot. One group received individualized externalfeedback during the session and the other groupdid not. Subjects were videotaped performingthe skill before and after the session. The tapeswere independently analyzed, in blinded fashion,by three surgeons. Three measures were ob-tained: the total time for the task, whether or notthe knot was square, and the general quality ofthe performance using a rating scale.

RESULTS: Data from 105 subjects were availablefor final analysis. For both groups there weresignificant increases in the proportion of knotsthat were square when the posttest performancewas compared with the pretest performance butthere was no difference between groups on thismeasure. Comparison of the performance scoresdemonstrated that both groups had a significantimprovement after the session but the perfor-mance scores were significantly better in thegroup that had received feedback.

CONCLUSIONS: Novices in both groups using CALshowed improvement in two of the outcomesmeasured, suggesting that subjects in bothgroups attained some degree of competencewith this skill. The higher posttest performancescore for the group receiving feedback demon-strates that external feedback results in a higherlevel of mastery when CAL is used to teach sur-

gical technical skills. Am J Surg. 2000;179:341–343. © 2000 by Excerpta Medica, Inc.

Our initial interest in CAL was that it might be amore efficient method to teach basic surgicalskills.1 In that study, the students generally indi-

cated that they preferred the more traditional lecture andfeedback seminar because of the availability of the contentexpert to provide feedback as they learned the task. Weconcluded that this lack of expert feedback in the CALgroup was the primary cause for the inferior level of masteryattained by this group. However, there were other differ-ences between the groups and so we could not completelydiscount the possibility that confounding variables mayhave been, at least in part, responsible for our findings.Further, we had observed that many of the students whohad used the CAL program alone had obtained a level ofcompetence with this task and were interested in quanti-fying the degree to which this occurred.

Our goals in this study were to rigorously evaluate theimpact of external feedback on the acquisition of a basicsurgical skill and to measure the degree to which learner’susing CAL alone had acquired a basic surgical skill.

METHODSApproval was obtained from the Institutional Review

Board at the Medical College of Georgia. Freshman andsophomore medical students were enrolled in the study,and each was randomly assigned to either the CAL alonegroup or the CAL and feedback group. The subjects werenot paid to participate in the study but were allowed tokeep the knot-tying board, manual, and rope. Both groupsused the CAL program that has been described previously.1

The subjects were placed in groups of seven or eight andeach interacted with a single computer in a speciallyequipped classroom.

At the beginning of the session, the subject was in-structed to tie his or her “best and fastest pair of two-handed square knots,” and the performance was video-taped. The subject was then supplied with a knot-tyingboard and tying rope and participated in a 1-hour educa-tional session. In the CAL with feedback group, a contentexpert was present to evaluate the subject’s performanceand provide instruction. The feedback was provided inter-mittently throughout the session. In the CAL alone group,no feedback was given during the session. At the end of thesession, the subject was given identical instructions and theposttest performance was videotaped. Finally, the subjectwas surveyed regarding his opinions about the session.

Each videotaped performance was reviewed indepen-

From the Department of Surgery (DAR, TRH, KAY, EP), MedicalCollege of Georgia, Augusta, Georgia, and the Department ofSurgery (GR), University of Toronto, Toronto, Ontario, Canada.

Requests for reprints should be addressed to David A. Rogers,MD, Section of Pediatric Surgery, Medical College of Georgia, BT5729, Augusta, Georgia 30912-4070.

Manuscript submitted September 23, 1999, and accepted inrevised form March 6, 2000.

Presented at the 19th Annual Meeting of the Association forSurgical Education, Boston, Massachusetts, April 7–10, 1999.

© 2000 by Excerpta Medica, Inc. 0002-9610/00/$–see front matter 341All rights reserved. PII S0002-9610(00)00341-X

dently by three surgical faculty members. They recordedwhether or not the knot was square, and in cases where allthree did not agree, the majority opinion was recorded.Further, the quality of the knot tying was evaluated usinga rating scale that explicitly identified all of the actionsnecessary for an optimal performance. The number gener-ated using the rating scale was termed a performance score,with a maximum value of 24. Finally, the time for the taskwas recorded for each performance when a performancewas attempted. Data were entered into the Paradox 8.0database (Corel Corp., Ontario, Canada) and analyzedusing Quattro Pro 8.0 (Corel Corp., Ontario, Canada) andSPSS 8.0 for Windows (SPSS, Inc., Chicago, IL).

RESULTSOne hundred and eight subjects were enrolled in the

study. Data from 3 of the subjects were excluded because itwas not possible to evaluate one of the performances, andso data from 105 subjects were available for final analysis.Fifty-four subjects were in the CAL with feedback groupand 51 subjects were in the CAL alone group. Average age,gender distribution, and handedness were similar for bothgroups.

A two-way analysis of variance (ANOVA) evaluating theeffect of time (pretest versus posttest) and group (CALalone versus CAL with feedback) revealed a powerful maineffect of time (F 5 532.3, P ,0.001) and group (F 5 19.7,P ,0.001) with a significant interaction between time andgroup (F 5 12.12, P ,0.001). A subsequent comparison ofthe average performance scores for each of the two groups(Table I) using the paired Student’s t test showed that thescores increased significantly within both the CAL alonegroup (t 5 13.4, P ,0.001) and the CAL with feedbackgroup (t 5 19.4, P ,0.001). The between-group averageperformance scores were compared using an independentsamples Student’s t test showing that the posttest perfor-mance score was significantly higher in the CAL withfeedback group when compared with the CAL alone group(t 5 4.38, P ,0.001).

The proportion of knots squared (Table II) increasedsignificantly within both groups (McNemar P ,0.001) butthere was no difference between groups in either the pre-test (chi square 5 1.36, P 5 0.24) or posttest performance(chi square 5 0.733, P 5 0.39). No comparison of time wasmade for the pretest group because so few of the subjects

could successfully complete the task, and comparison of theposttest times for the two groups demonstrated no signifi-cant difference between the CAL alone and the CAL withfeedback groups (14.5 versus 16.1 seconds, t 5 1.3, P 50.19). Fifty-five percent of the subjects in the CAL withfeedback group reported that having expert feedback avail-able was a major positive factor in the acquisition of theskill.

COMMENTSThe results of this study not only provide additional

evidence for the importance of external feedback from acontent expert in the instruction of basic surgical skills butthey also provide some evidence for the effectiveness ofCAL alone for this purpose.

The acquisition of motor skills is subject to continuingresearch and several theories have evolved. These theorieshave become divergent enough to be considered a different“schools.”2,3 One unifying characteristic of these theories isthe importance of feedback in skill mastery.2 The interac-tion of feedback and skill acquisition can be illustrated byexamining one of the theories of skill acquisition that hasreceived attention in the surgical literature.4,5 In this the-ory, psychomotor tasks are said to be learned in threephases: cognitive, associative, and autonomous.2 In thecognitive phase, the learner attains a degree of understand-ing of the task. During the associative phase, the learnerpractices the task and the performance is compared withthat of the expert. The difference between the expert andlearner performance is considered the error, and the learnerattains mastery by minimizing this error. In the autono-mous phase, the skill is performed without distinct cogni-tive awareness.

It is during the associative phase that feedback is felt toplay a major role. Feedback may be either internal orexternal. Internal feedback is that generated by the learnerwhen the learner’s performance is compared with that ofthe experts. This comparison occurs through the sensorysystem, and in our study would occur when subjects com-pared their visual assessment of their own performancewith that contrasted by the CAL representation of theexpert’s performance. The fact that both the proportion ofknots squared and the performance scores improved in theCAL alone group suggests that the CAL depiction wassuccessful in providing learners with sufficient informationabout the task so that they were able to develop a cognitivepicture of the task and use their internal feedback tocorrect some of the errors. Prior research in medical skills

TABLE IAverage Performance Scores

Pretest PosttestWithin-Group

Difference

CAL alone (n 5 51) 2.6 (1.94) 12.0 (4.87) 9.4*CAL plus feedback

(n 5 54) 3.1 (2.22) 15.8 (4.15) 12.7*Between-group

difference 0.5 3.8†

Values shown are the average scores with the standard deviation shown inparentheses.* Significantly different by Student’s paired t test.† Significantly different by the independent sample Student’s t test.CAL 5 computer-assisted learning.

TABLE IIProportion of Knots Squared

Pretest PosttestWithin-Group

Difference

CAL alone (n 5 51) 33 87 54*CAL plus feedback (n 5 54) 44 92 48*Between-group difference 11† 5†

Values shown are the percentage of subjects who tied a square knot.* Significant difference by the McNemar test.† No significant difference by chi square of proportions.CAL 5 computer-assisted learning.

IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL

342 THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000

instruction has also demonstrated the effectiveness of in-ternal feedback for skill mastery.5,6

The other major type of feedback is external, as thesource of the feedback is external to the learner. Thissource is typically a content expert and this type of feed-back is generally divided into knowledge of results (KR)and knowledge of performance (KP). KR is defined as“information about the success of an action with respect tothe environmental goal.”7 In contrast, KP is defined as“information about the pattern of the movement thelearner has made.”7 In our study the desired result is asquared knot, and our results suggest that the contentexpert appeared to offer no advantage over the learner’sown internal feedback process when CAL was used toprovide KR. In contrast, the fact that the CAL group thatreceived feedback had a significantly higher performancescore suggests that the external feedback provided a sub-stantial advantage in this group of learners understandingof KP. We believe that this was due, in part, to the fact thatthe skill is complex and the learners were novices. Thelearner could recognize that the performance did not

match that of the expert but it required the expert toidentify the error and outline the steps necessary to correctthe learner’s performance.

REFERENCES1. Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computer-assisted learning versus a lecture and feedback seminar for teachinga basic surgical technical skill. Am J Surg. 1998;175:508–510.2. Kaufman HH, Wiegand RL, Tunick RH. Teaching surgeons tooperate: principles of psychomotor skills training. Acta Neurochir.1987;87:1–7.3. DesCoteaux JG, Leclere H. Learning surgical technical skills.Can J Surg. 1995;38:33–38.4. Kopta JA. An approach to the evaluation of operative skills.Surgery. 1971;70:297–303.5. Nicks CM, Nelson DL, Lang NP. Use of the surgical skillslaboratory for teaching medical students. Focus Surg Educ. 1986;3:13–14.6. Kardash K, Tessler MJ. Videotape feedback in teaching laryn-goscopy. Can J Anaesth. 1997;44:54–58.7. Schmidt RA. Motor Learning and Performance. From Principles toPractice. Champaign, Ill: Human Kinetics Books; 1991.

IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL

THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000 343