Surgical Teaching Quality Makes a DifferenceAmy V. Blue, PhD, Charleston, South Carolina, Charles H. Griffith III, MD, MSPH, John Wilson, PhD,

David A. Sloan, MD, Richard W. Schwartz, MD, Lexington, Kentucky

BACKGROUND: This study examined the influenceof the quality of faculty members’ teaching onstudent performance in a third-year surgeryclerkship.

METHODS: Eighty-nine third-year students on asurgery clerkship completed preceptor evalua-tion forms. The faculty member’s overall scorewas the mean of ratings from all the third-yearstudents for whom that faculty member servedas preceptor during the year. We examined as-sociations between these ratings and studentperformance on the National Board of MedicalExaminers (NBME) surgery subject examinationand clerkship Objective Structured ClinicalExamination (OSCE) by using an analysis of co-variance that controlled for prior academicachievement [United States Medical LicensureExamination (USMLE) Part I].

RESULTS: The average mean teaching evaluationscore was associated with the scores on theNBME surgery subject examination (P 5 0.0005).Students with attendings who received poorteaching evaluations performed more poorly onOSCE data-gathering stations than did studentswith attendings rated as average or good.

CONCLUSIONS: The study results indicate that theteaching quality of surgery faculty appears tohave an impact on student performance. Am JSurg. 1999;177:86–89. © 1999 by Excerpta Med-ica, Inc.

The characteristics of highly rated teachers have beenextensively reported in the medical education liter-ature. These characteristics include enthusiasm,

clear, well-organized presentation of instructional material,skill in interaction with students/residents and group set-tings, involvement of the learner in the teaching process, ahumanistic orientation, content knowledge of the subject,and use of case-based teaching scripts.1–3

The examination of highly rated teaching characteristicsis valuable for faculty development efforts aimed at improv-ing faculty teaching skills. However, whether the studentsof these highly rated teachers actually learn more is largely

unknown. College students have performed better on ex-aminations in classes in which they give instructors higherratings.4–6 However, these findings are from electiveclasses, and students’ higher scores and the higher ratingsgiven to instructors may reflect the students’ enthusiasm forthe subject.

Griffith et al7 recently examined the relationship of in-ternal medicine attending physician teaching effectivenesswith objective measures of student performance in a2-month internal medicine third-year clerkship. The studyresults indicated that teaching quality has an impact onstudent performance, as measured by increases in pre- andpost-clerkship NBME medicine subject examination scoresand clerkship CPE scores. Griffith et al7 write that futurestudies should evaluate effects of teaching quality in avariety of disciplines. The purpose of the present study is toexamine the influence of the quality of faculty members’teaching on student performance in a third-year surgeryclerkship.

METHODSThe subjects were the 89 students who rotated on the

third-year surgery clerkship during the 1995-96 academicyear. The surgery clerkship consists of one 4-week rotationon a general surgery service (gastrointestinal, vascular,oncology, and trauma services) and one 4-week rotation ona specialty surgery service (urology, otolaryngology, ortho-pedics, plastics, cardiothoracic surgery, and neurosurgery).During each 4-week rotation, 2 to 3 students were ran-domly assigned to a specific faculty member preceptor. Thepreceptor was responsible for the students’ educationalexperience during the assigned month. This responsibilityincluded teaching surgical concepts to the students duringformal preceptor-teaching rounds, organizing the students’clinical activities (operating room, clinic, and call experi-ences), having the student participate in preceptor surger-ies and clinics, and evaluating the students’ clinical per-formance. During surgeries, the preceptor was expected todiscuss with the student the patient’s pathophysiology.Preceptors were to assign at least 2 patients each week tothe student. Students were expected to take a history andperform a physical examination on each assigned patient.The preceptor and student subsequently discussed perti-nent findings and treatment recommendations. Thus, stu-dents had approximately 15 to 20 hours of contact with thepreceptor each week, primarily in the patient care context,and also 3 to 4 hours of formal teaching rounds.

In addition to, and separate from, the students’ clinicalactivities supervised by the preceptor, students’ time duringthe clerkship is spent every week in two problem-basedlearning (PBL) sessions with a PBL tutor, a 2-hour clinicalskills workshop organized by clerkship faculty, a 1-hourteaching session with the clerkship director, and a 1-hourexpert resource session (ERS) in which a faculty member

From the Department of Family Medicine (AVB), Medical Uni-versity of South Carolina College of Medicine, Charleston, SouthCarolina; and the Departments of Internal Medicine (CHG, JW),Behavioral Science (JW), and Surgery (DAS, RWS), University ofKentucky College of Medicine, Lexington, Kentucky.

Requests for reprints should be addressed to Amy V. Blue, PhD,Department of Family Medicine, Medical University of SouthCarolina, 171 Ashley Avenue, Charleston, South Carolina 29425-2201.

Manuscript submitted August 17, 1998 and accepted in revisedform September 30, 1998.

86 © 1999 by Excerpta Medica, Inc. 0002-9610/99/$–see front matterAll rights reserved. PII S0002-9610(98)00304-3

expert discusses unresolved learning issues and objectivesof the week’s PBL case in an interactive fashion with thestudents.

Thirty-six faculty members served as preceptors duringthe study period (the 1995-96 academic year). The teach-ing quality of the preceptors was measured using a 28-item,departmentally developed evaluation form that each stu-dent completed at the end of each 4-week rotation. Stu-dents were required to complete the form at the end ofeach month before they were able to view their evaluationsby faculty and before they received their final grade in theclerkship. Students were asked to evaluate the preceptor’sbehavior in the areas of teaching activity (ie, “your pre-ceptor explained the rationale for his diagnostic thinkingwhen you shadowed him in the clinic and on attendingrounds”), providing feedback (ie, “your preceptor gave youconstructive feedback when he discussed with you thepathophysiology of the two patients that you were assignedeach week”), and role modeling (ie, “your preceptor dem-onstrated empathetic responses to patient questions in theclinic and on attending rounds”). Each item was rated ona 3-point scale (1, preceptor did not do; 2, preceptor didpartially; 3, preceptor did consistently).

The inter-item correlation across all evaluation forms washigh (a 5 .93), indicating that items on our evaluationform had a high degree of internal consistency. There waslittle discrimination for the individual items on the form,and therefore the mean rating across the 28 items of theform was used as one overall rating of teaching quality. Thefaculty member’s overall score was the mean of ratings fromall the third-year students for whom that faculty memberserved as preceptor during the year.

In addition to analyzing the total teaching evaluationscore as a continuous variable, we also constructed catego-ries of teaching quality (after Griffith et al7). We defined“best” teaching preceptors as those receiving the top 20%of overall student evaluations in the study period, and“worst” teaching attendings as those receiving the lowest20% of overall student evaluations. The upper and lowercutoffs of 20% were chosen because we believe that mostfaculty (.50%) are neither the best nor the worst teachers.

The measures of student performance included (1) thestudent’s score on the NBME surgery subject examinationadministered at the end of the clerkship, and (2) thestudent’s score on an end-of-clerkship Objective Struc-tured Clinical Examination (OSCE). The clerkship OSCEhas been described elsewhere.8

Analysis was conducted in two phases. In the first phase,multiple regression approaches using the General LinearModel9 were used to examine the effect of mean teachingquality score on NBME surgery subject examination score,and data-gathering and data-interpretation scores from theOSCE stations. In this analysis, using all continuous vari-ables, the dependent variable was the student’s NBMEsubject examination score, and the independent variablewas the mean of each student’s two attending faculty’s totalscore on the 28-item teaching evaluations. In addition, inorder to control for preclerkship academic performance,the student’s score on the (USMLE) Part I examinationscore was entered first in the model.

In a second phase of the analysis, which considered

teaching quality defined categorically rather than contin-uously, we categorized students as having either had a“best” category attending, a “worst” category attending, orhaving had neither a “best” nor “worst” (ie, two middleranking attendings). Six students experienced both a“worst” and a “best” attending. These students were ex-cluded from this phase of the analysis because of the smallsample size. General linear model approaches, which func-tion effectively with both continuous and categorical vari-ables, were once again used in the analysis, with the three-category variable described above as an independentvariable, the score on the NBME post-clerkship surgeryexamination and the data-gathering and data-interpreta-tion scores from the OSCE examination as dependentvariables, and the score on the USMLE Part I examinationentered first to control for pre-clerkship academic perfor-mance.

RESULTSEighty-nine students completed 178 evaluations on 36

faculty (91% response rate). The mean number of evalua-tions per faculty member was 4.86. Importantly, the meanfaculty rating did not correlate significantly with the num-ber of evaluations per faculty member ( r5 .02, P 5 0.90).Therefore, we did not need to exclude a faculty memberbased on the number of evaluations he or she had received.Also students’ ratings of their first attending were uncor-related (r 5 .10, P 5 0.38) with ratings of their secondattending. A substantial correlation would have suggestedthe possibility of a rater bias, which therefore is unlikely tohave affected our findings in the study. Seven of the 36faculty members scored in the “best” category and 6 of the36 faculty members scored in the “worst” category. Previ-ous academic achievement, operationalized as USMLE partI score and used as a control variable in our analyses, wasassociated with better scores on the NBME surgery subjectexamination (F[1,85] 5 107.1, P ,0.0001), OSCE datagathering station (F[1,85] 5 8.75, P ,0.005), and datainterpretation station (F[1,85] 5 6.34, P 5 0.013) scores.

After controlling for prior academic achievement, theaverage mean teaching evaluation score was also associatedwith scores on the NBME surgery subject examination(F[1,85] 5 13.03, P 5 0005). Students with attendingfaculty with higher teaching quality scores scored higher onthe examination than did students whose attendings wererated lower. The size of this relationship was substantial,explaining 12.8% of the variance in examination scoresafter controlling for pre-clerkship academic achievement.Results for the effect of mean teaching quality score on thedata gathering (F[1,85] 5 2.23, P 5 0.14) and data inter-pretation (F[1,85] 5 1.49, P 5 0.22) were in the samedirection as that of the NBME surgery subject examina-tion, but were not statistically significant. The main andinteraction effects of teaching quality for data gatheringstation scores (F[1,85] 5 3.35, P 5 070) and data inter-pretation station scores (F[1,85] 5 1.82, P 5 0.18) weresimilar in direction, but did not reach conventional levelsof statistical significance.

In order to examine more specifically the potential effectsof “best” or “worst” teaching quality attendings, the secondwave of analyses compared performance scores of 28 stu-dents who had one or more “best” attendings and no

SURGICAL TEACHING QUALITY/BLUE ET AL

THE AMERICAN JOURNAL OF SURGERY® VOLUME 177 JANUARY 1999 87

“worst” attendings, 30 students who had neither a “best” or“worst” attending, and 25 students who had “worst” and no“best” attendings. This analysis allows more discrete exam-ination of the differential effects of a “best” or “worst”attending. Scores on the performance variables are de-picted in the Table. Students who experienced at least oneof the “best” attendings performed better on the NBMEsurgery subject examination than did those who had atleast one of the “worst” attendings (P 5 0.013). Studentswho experienced one of the “worst” attendings performedmore poorly on the data gathering stations of the OSCEthan did students who experienced neither the “best” northe “worst” (P 5 0.039) and marginally worse than stu-dents who had a good attending (P 5 0.075). There wereno differences on student performance on the data inter-pretation OSCE stations.

COMMENTSOur study results indicate that the teaching quality of

surgery faculty appears to have a significant impact onstudent performance. Students who worked with the betterquality teaching surgery faculty performed better on mea-sures of knowledge acquired during the clerkship as dem-onstrated through students’ higher scores on the NBMEsurgery subject examination. As indicated by the perfor-mances of students who worked with at least one “best”attending, outstanding teaching by a faculty member ap-pears to positively influence student performance. Further-more, student exposure to “best” and “worst” surgery fac-ulty influenced different areas of student performance.

Students who worked with the “worst” surgery faculty didnot perform as well as other students on the data-gatheringstations of the OSCE. Aspects of the clerkship experiencethat are not attributable to preceptor teaching quality, suchas the volume of OR scrubs, procedures and patients ex-amined, may contribute to student learning, but one reportconcluded that such aspects do not have an effect onstudent examination performance.10 Students value a sup-portive learning environment when evaluating a surgicalteacher, as well as the teacher’s mentoring, communica-tion, and organizational abilities.11,12 Better teachers maybe able to establish a positive learning climate for studentsthat stimulates their interest in learning. This interest inturn provides students a greater base of surgical care knowl-edge that they demonstrate on written examinations. Stu-dents who worked with teachers who had poor role-mod-

eling skills (developing patient rapport and empathiccommunication) may have been exposed to less effectiveexamples of patient interaction, in turn resulting in stu-dents’ poor performance on data-gathering, or patient en-counter, OSCE stations. Sloan et al12 report that studentsregard role-modeling as central to the surgical teacher’seffectiveness.

Our study findings are similar to those reported by Griffithet al7 for internal medicine teaching faculty. Both sets offindings validate the need for faculty development pro-grams that focus on teaching skills and justify targetingfaculty members who need assistance with those skills.These conclusions correlate with the observations by Co-hen et al13 that good and average teachers maintain stableteaching effectiveness scores over time, but that poorteachers can improve scores if incentives are present. Pro-viding evidence that quality teaching has a relationship tostudent performance in a clerkship can serve as such anincentive.

There are several limitations to our study. First, this studyis from one institution only. While the results confirmthose of Griffith et al,7 who examined internal medicinefaculty at our institution, they may reflect particular char-acteristics of the institution’s teaching faculty or studentpopulation during the study periods. Future research shouldevaluate the effects of teaching quality at other institutionsas well. Secondly, our measure of teaching quality is basedonly on student evaluations. Studies in the literature havefound the reliability of student evaluations of lecturingskills in medicine to be good; with respect to clinicalteaching, reliability coefficients of 0.80 have been demon-strated on evaluations of a clinician’s teaching whenenough ratings were available for comparison.14,15 None-theless, additional sources of evaluation, such as peers andresidents, could provide further measures of teaching qual-ity. However, faculty rarely observe one another interact-ing and teaching students. While residents and students areoften together with the faculty, the resident experience insurgery is typically distinct from the student experience.Residents may evaluate the “best” and “worst” faculty dif-ferently than students because their residency training cre-ates particular demands. Future research could explore thispossibility.

The present study reinforces the value of quality interac-tion between clinical faculty and students, and the impor-

TABLEScores on Performance Variables

Preceptor* N

NBME Surgery†Data

Gathering‡Data

Interpretation

M§ SD M SD M SD

Experienced ‘‘best’’ 28 488.1 83.0 70.3 4.1 59.6 6.9Experienced ‘‘middle’’ 30 472.1 64.9 70.8 7.2 60.9 6.7Experienced ‘‘worst’’ 25 448.6 104.2 67.1 7.4 59.9 8.9

* Six students experienced both best and worst teaching quality attendings and were excluded from covari-ance analysis due to small n.† ‘‘Best’’ . ‘‘worst’’ P 5 0.013.‡ ‘‘Best’’ . ‘‘worst’’ P 5 0.075; ‘‘middle’’ . ‘‘worst’’ P 5 0.039.§ Mean adjust for prior academic achievement (USMLE Part I).

SURGICAL TEACHING QUALITY/BLUE ET AL

88 THE AMERICAN JOURNAL OF SURGERY® VOLUME 177 JANUARY 1999

tant teaching function of faculty as mentors. While in-structional technology expands its role in the education ofmedical students, the impact of one-on-one faculty teach-ing contact with students cannot be neglected. In an era inwhich outcomes are increasingly used as the basis for re-wards such as promotion and tenure, the study findingsprovide evidence that the efforts of excellent teachers canbe measured in ways other than student satisfaction withteaching efforts. Quality faculty teaching efforts should bean integral criterion in the promotion and tenure process,and outstanding teaching efforts should be recognized andrewarded.

REFERENCES1. Irby DM. Clinical teacher effectiveness in medicine. J MedEduc. 1978;53:808–815.2. McLeod PJ, Harden RM. Clinical teaching strategies for physi-cians. Med Teach. 1985;7:173–189.3. Irby DM. What clinical teachers in medicine need to know.Acad Med. 1994;69:333–342.4. Cohen PA. Student ratings of instruction and studentachievement: a meta-analysis of multisection validity studies. RevEduc Res. 1981;51:281–309.5. Feldman KA. Grade and college students’ evaluation of theircourses and teachers. Res High Educ. 1976;4:60–111.6. Cashin WE. IDEA Technical Report no. 32: Student Ratings of

Teaching: The Research Revisited. Manhattan: Kansas State Univer-sity, Center for Faculty Evaluation and Development; 1995.7. Griffith CH III, Wilson JF, Haist SA, Ramsbottom-Lucier M.Relationships of how well attending physicians teach to theirstudents’ performances and residency choices. Acad Med. 1997;72(suppl 1):S118–S120.8. Blue A, Stratton T, Plymale M, et al. The effectiveness of theStructured Clinical Instruction Module. Am J Surg. 1998;176:67–70.9. Cohen J, Cohen P. Applied Multiple Regression/Correlation Anal-ysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: LawrenceErlbaum Associates; 1983.10. Chatenay M, Maguire T, Skakun E, et al. Does volume ofclinical experience affect performance of clerks on surgery exitexaminations? Am J Surg. 1996;172:366–372.11. Dunnington G, DaRosa D, Kolm, P. Development of a modelfor evaluating teaching in the operating room. Curr Surg. 1993;50:523–527.12. Sloan DA, Donnelly MB, Schwartz RW. The surgicalclerkship: characteristics of the effective teacher. Med Educ. 1996;30:18–23.13. Cohen R, MacRae H, Jamieson C. Teaching effectiveness ofsurgeons. Am J Surg. 1996;171:612–614.14. Stillman PL, Gillers MA, Heins M, et al. Effect of immediatestudent evaluations on a multi-instructor course. J Med Educ. 1983;58:172–178.15. Irby D, Rakestraw P. Evaluating clinical teaching in medicine.J Med Educ. 1981;58:181–186.

SURGICAL TEACHING QUALITY/BLUE ET AL

THE AMERICAN JOURNAL OF SURGERY® VOLUME 177 JANUARY 1999 89