Association for surgical education

Trainees’ opinions of the skills required of basic surgical trainees

Peter J. Driscoll*, Anna M. Paisley, Simon Paterson-BrownaUniversity Department of Clinical and Surgical Sciences (Surgery), Royal Infirmary, Edinburgh, United Kingdom

Manuscript received September 3, 2002; revised manuscript December 24, 2002

Abstract

Background: The Edinburgh Basic Surgical Trainee Assessment Form (EBSTAF) is a feasible, reliable and construct valid tool forassessment of surgical trainees. Our aim was to determine its acceptability as a formative training tool.Methods: Thirty-three trainees on the South-East Scotland Basic Surgical Training Program ranked the 70 skills examined by the form asessential, important, useful, or irrelevant. Responses were compared with those of consultant surgeons obtained during development of theform.Results: There was total agreement in 44 skills (63%,� � 0.34). Trainees assigned greater importance to 24 (34%). For individual skillsdomains, trainees assigned significantly greater value: communication, 86% versus 78%; application of knowledge, 75% versus 67%;team-working, 84% versus 77%; clinical skills, 86% versus 83%; and technical skills, 84% versus 79%. Responses were internally consistent(� � 0.74 to 0.93).Conclusions: Trainees attach greater value than consultants to the qualities assessed by EBSTAF. Trainees therefore agree with consultantopinion on what is important in a surgical trainee, supporting use of this form as a formative training tool. © 2003 Excerpta Medica, Inc.All rights reserved.

Keywords: Surgery; Training; Feedback; Formative assessment; Acceptability

Assessment is the key to adult learning. The method andcontent of assessment is a strong driving force for studybehavior [1–7] but has been viewed by many with consid-erable skepticism owing to the dominance of external sum-mative (pass or fail) assessments [8], the psychometricproperties of which have often not been determined. Beforeany assessment tool is introduced it must be shown to fulfillthe criteria of good assessment: feasible, reliable, valid, andacceptable to those being evaluated.

The adult learning model of Kolb and Fry [9] is analo-gous to the processes of surgical audit (the two are com-bined in Fig. 1). For trainees to optimize their training theymust have a concept of their final goal, the ability to com-pare actual and desired performance and the means to lessenthe gap between the two [10,11]. First, the attributes orqualities that make a good surgeon must be identified; thisshould be an expert consensus view in order to maximizeobjectivity in an area so difficult to quantify. Second, as-sessment must be able to measure actual performance anddemonstrate an improvement as a result of training (con-

struct validity), allowing comparison with the agreed stan-dard. Finally, the means to address the trainee’s strengthsand weaknesses can only come from directed training on thepart of trainers, itself benefiting from accurate assessment asa guide. The repeated use of assessment in this context istermed “formative” assessment and has been shown to re-sult in substantial learning gains in the field of educationalresearch [12]. Thus, a truly robust assessment will (1) doc-ument the acquisition of skills, thereby maintaining stan-dards; (2) facilitate the formative process of feedback; (3)be utilized in the selection process; and (4) motivate bothtrainee and trainer to take an active role in the trainee’sprofessional development [13].

There remains considerable concern regarding the train-ing of basic surgical trainees (BSTs) in the United Kingdom(BSTs are roughly equivalent to postgraduate years 1 and 2)[14–18] and their assessment has historically been highlysubjective [19], flawed by preexisting prejudice or person-ality clashes. More recently, the shorter surgical workingweek [20] and overall period of surgical training [21] havedirectly affected surgical team working patterns [22] andmade objective assessment even more difficult. With this inmind, the Edinburgh Basic Surgical Trainee Assessment

* Corresponding author.E-mail address: pjdriscoll@rcsed.ac.uk

The American Journal of Surgery 186 (2003) 77–80

0002-9610/03/$ – see front matter © 2003 Excerpta Medica, Inc. All rights reserved.doi:10.1016/S0002-9610(03)00110-7

Form (EBSTAF) was developed using a modified Delphitechnique with consultation of a total of 111 consultant(attending) surgeons in the South-East of Scotland withrepresentation from each main surgical subspecialty. StageI requested them to anonymously identify skills and quali-ties required for a successful training in surgery and thetechnical skills of a BST after 6 months in their own unit.Sixty-eight consultant surgeons (61%) responded, and theidentified qualities were subsequently separated into fivedomains: communication with patients and relatives; appli-cation of knowledge; team-working skills; clinical skills;and technical skills. Stage II used a second anonymousquestionnaire, returned to all the original 111 consultants,listing the identified attributes and asked them to rank eachusing tick-boxes as essential (4), important (3), useful (2), orirrelevant (1). Responses were received from 78 consultants(70%) across all specialties, and the resultant form ad-dressed a total of 70 skills (details on request) [23]. Assess-ment of trainee ability by multidisciplinary assessors (in-cluding medical, nursing, and secretarial staff) was thenevaluated among all BSTs in the South-East of Scotlandbasic surgical training scheme in the subsequent 18-monthperiod. The EBSTAF was shown to be feasible, reliable,internally consistent, and construct valid after detailed anal-ysis [24].

The training of a surgical trainee may be thought of as a“ training contract,” the trainer agreeing to teach the traineewhat is thought to be important for his/her progressiontoward a consultant post. Like any contract, however, train-ees must also agree to the terms therein and value thetraining they are given; if not, it is likely to be ignored andcome to nothing, wasting the efforts of both parties. Thusthe acceptability to the trainee of the criteria examined by anassessment tool in formative assessment is vital to its ap-plication. This is also true of a truly transparent summativeor selective processes, if the trainee is to accept the finaloutcome, particularly should it prove to be unfavorable. Theaim of this study therefore was to determine the opinions ofBSTs of the fields examined by the EBSTAF and to com-pare these with those of the consultants from the initialdevelopment of the form, thus determining the form’s ac-ceptability as a formative assessment tool.

Methods

Stage II of the previously employed Delphi techniquewas repeated with ranking questionnaires sent to all 33BSTs currently in post on the South-East of Scotland BasicSurgical Training scheme. This was carried out betweenJuly and August of 2001 so as to include trainees just aboutto leave the program and those who had just joined. Expe-rience thus ranged from naive surgical trainees to those after2.5 years training who were about to take up research orspecialist registrar posts (roughly equivalent to postgraduateyears 1 and 2).

Questionnaires listed the fields from the EBSTAF bydomain, but in different order so as to remove any inferredfield ranking from the form itself. In common with theinitial consultant questionnaire, trainees were asked to in-dicate for each skill whether they considered it to be essen-tial (4), important (3), useful (2), or irrelevant (1). They werealso requested to list any attributes that they felt had beenomitted from the EBSTAF, allowing them to express their ownopinion, independent of consultants, on what skills favor asuccessful surgical career. Return of completed questionnaireswas requested within 2 weeks of receipt and trainees werereminded by mail, then by telephone call, and finally per-sonal visit at 4, 6, and 8 weeks, respectively, as required.

Statistics

Weightings assigned to each field by both trainees andconsultants were analyzed using the Statistical Package forthe Social Sciences software for Windows (SPSS, Chicago,Illinois). Nonparametric statistical methods were usedthroughout analysis.

Comparison of rankings

Responses for each task were summed to create a singlescore for each domain, regarded as an indication of anindividual’s overall assessment of the importance of thatdomain. Although these have no arithmetic meaning (ie, ascore of 36 is not twice as good as a score of 18), it doesallow a summary score to be developed for each domain.This was then expressed as a percentage of the maximumpossible score for that domain (ie, number of fields thereinmultiplied by four (essential weighting)) and the medianscores from the two groups (designated median %) com-pared by Mann-Whitney U statistics.

Internal consistency

Internal consistency within each domain and overall fortrainee and consultant groups was determined by the appli-cation of Cronbach’s alpha (�). Values above 0.7 are gen-erally agreed to be acceptable while those above 0.8 and 0.9are classed as good and excellent, respectively.

Fig. 1. Surgical audit and adult learning.

78 P. Driscoll et al. / The American Journal of Surgery 186 (2003) 77–80

Agreement

For each field, a median ranking for BST and Consultantgroups (the latter taken from the previous study [23]) wasdetermined, and these were compared for each field in turn.Firstly, the number of fields where median consultant andtrainee rankings agreed precisely was determined and ex-pressed as a percentage (designated as exact % agreement).Similarly fields where trainees’ opinion differed from thatof consultants were identified and the number of fieldsassigned greater or lesser value determined and expressed asa percentage (designated exact % greater and exact % lesserrespectively). However, this method takes no account of thepotential effects of chance and so a second analysis wascompleted by the application of weighted kappa (�) whosevalues are classified as shown in Table 1 [25].

Results

All 33 trainees returned their questionnaires within thestudy period giving a 100% response rate. Forms werereturned promptly by 23, after mail reminder by 7 andtelephone call by 3. No additional attributes were suggestedby any trainee. Results are summarized in Table 2.

Internal consistency of responses in both groups, as de-termined by �, was determined as “good” to “excellent” inall but consultant opinion of team-working skills where itwas “acceptable.” Comparison of median percentage scores

demonstrated that trainees consistently assigned signifi-cantly greater importance than consultants to the fields ex-amined by the EBSTAF.

Of the 70 qualities examined by median response (Fig.2), there total agreement between BST and consultantgroups in 44 (exact % agreement � 63%). Of the remaining26 fields, 24 were valued more highly by the BSTs (exact %greater � 34%) while only two, namely “maintains accuratenotes” and “obtains additional information from relatives,”were considered less important by the trainee group (exact% lesser � 3%).

Determination of � was not valid for the domains of “com-munication with patients and relatives” and “application ofknowledge” ; this test requires variation within both compari-son groups and the median scores for all fields here wereassigned by consultants as “ important (3).” Overall agreementbetween consultant and trainee groups was determined as be-ing “fair” (� � 0.34) while agreement within individual do-mains could be classed as “moderate,” “ fair-to-moderate,” and“fair” for team-working (� � 0.46), clinical skills (� � 0.39),and technical skills (� � 0.25), respectively.

Comments

This study has demonstrated that the fields examined bythe EBSTAF are acceptable to trainees; they value the same

Table 1Degrees of agreement as determined by weighted Kappa [25]

Value of kappa Degree of agreement

�0.20 Poor0.21–0.4 Fair0.41–0.6 Moderate0.61–0.8 Good0.81–1.0 Very good

Table 2Summary of results

Domain Total domain ratings* [median % (�)] Agreement† Disagreement†

Consultant Trainee Mann-Whitney � Degree ofagreement

Exact %agreement

Exact %greater

Exact %lesser

Communication 78 (0.91) 86 (0.93) 0.006 — — 56 44 —Knowledge 67 (0.81) 75 (0.89) 0.001 — — 56 44 —Teamwork 77 (0.74) 84 (0.87) 0.025 0.46 Moderate 73 27 —Clinical skills 83 (0.90) 86 (0.93) 0.05 0.39 Fair–moderate 64 27 9Technical skills 79 (0.89) 84 (0.90) 0.004 0.25 Fair 63 37 —All 78 (0.85) 83 (0.89) �0.001 0.34 Fair 63 34 3

* Median % is median percentage value score as determined by [(sum of responses/maximum possible score) � 100]; � is Cronbach’s alpha determinationof internal consistency where �0.7 � acceptable, �0.8 � good and �0.9 � excellent; Mann-Whitney is comparison of median % for consultant and traineegroups (P � 0.05 � significant).

† � is the weighted kappa determination of agreement as defined by Altman (see Table 1); exact % agreement is percentage where median trainee responseis equal to median consultant response; exact % greater is percentage where median trainee response greater than median consultant response; exact % lesseris percentage where median trainee response less than median consultant response.

Fig. 2. Diagrammatic representation of consultant-trainee agreement.

79P. Driscoll et al. / The American Journal of Surgery 186 (2003) 77–80

qualities previously determined to be important by consult-ants, attaching greater or equal value to all but two of its 70fields. The 100% response rate (aided considerably by theinvolvement of SPB, also chairman of the training program)would support the validity of our results.

The determination of agreement of opinion in this in-stance is not as straightforward as it might at first appear.Clearly, when both groups assign the same importance to aparticular quality it may be regarded as agreement. How-ever, the � statistic will regard a trainee ranking of “essen-tial” compared with a consultant one of “ important” as adisagreement, despite the fact that both groups examinedconsider value in the quality concerned. Thus, examinationof simple agreement percentages may be more valid if oneconsiders a trainee ranking of at least that of consultants tobe an agreement. If this is carried out, as shown in Fig. 2,overall agreement as to the qualities assessed in EBSTAFwould be 97%, with only 2 (3%) thought to be of less valueby trainees. However, even this may suffer from bias sincetrainees may be expressing what they perceive their con-sultants want them to think, rather than what they them-selves value and employ in their everyday clinical practice.The fact that no additional skills were suggested by thetrainees would support the validity of the conclusions of thisstudy but that can not be relied upon as abstention is not initself an agreement.

Bearing the above reservations in mind, the finding thatthe EBSTAF is so acceptable to trainees is encouraging;detailed assessment in areas felt to be important by bothtrainee and trainer should improve surgical training. It alsosuggests that trainees will value feedback of assessmentsobtained using such a highly acceptable assessment tool.This lends further support to the application of the EBSTAFin the assessment of trainees.

The EBSTAF is currently completed by multiple asses-sors for each BST in the South-East Scotland Basic SurgicalTraining Scheme and an anonymous summary given to eachtrainee every 6 months as part of the appraisal process. Thiswill allow us to study the effect of formative assessmentwith regular feedback to surgical trainees.

In conclusion, trainees attach equal or greater value thantheir consultants to the qualities assessed in the EdinburghBasic Surgical Trainee Assessment Form, supporting the ac-ceptability of the form to basic surgical trainees. The EBSTAFhas been previously shown to be feasible, reliable, andvalid, and this study provides further evidence to support itsuse as a means of assessment for basic surgical trainees.

Acknowledgments

The authors would like to thank The Scottish Council forPostgraduate Medical and Dental Education and The RoyalCollege of Surgeons of Edinburgh who funded this work.

We would also like to thank Dr. Jeremy Walker in theUniversity Department of Clinical and Surgical Sciences(General Practice) for his expert advice on the statisticalmethods used in this study.

References

[1] Newble DI, Jaeger K. The effect of assessments and examinations onthe learning of medical students. Med Educ 1983;17:165–71.

[2] Newble DI, Entwistle NJ. Learning styles and approaches: implica-tions for medical education. Med Educ 1986;20:162–75.

[3] Stillman PL, Haley HL, Regan MB, Philbin MM. Positive effects ofa clinical performance assessment program. Acad Med 1991;66:481–3.

[4] Fowell SL, Southgate LJ, Bligh JG. Evaluating assessment: the miss-ing link? Med Educ 1999;33:276–81.

[5] Stillman PL, Regan MB, Swanson DB. A diagnostic fourth-yearperformance assessment. Arch Intern Med 1987;147:1981–5.

[6] Bouhuijs PA, van der Vleuten CP, van Luyk SJ. The OSCE as a partof a systematic skills training approach. Med Teach 1987;9:183–91.

[7] Malik SL, Manchanda SK, Deepak KK, Sunderam KR. The attitudesof medical students to the objective structured practical examination.Med Educ 1988;22:40–6.

[8] Black P. Research and development in educational research. OxfordRev Educ 2000;26:407–19.

[9] Kolb D, Fry R. Towards an applied theory of experiential learning. In:Cooper C, editor. Theories of group processes. London: Wiley, 1975.

[10] Gipps C. Beyond testing: towards a theory of educational assessment.London: Falmer Press, 1994.

[11] Sadler DR. Formative assessment and the design of instructionalsystems. Instruct Sci 1989;18:119–44.

[12] Black P, William D. Assessment and classroom learning. AssessEduc 1998;5:7–71.

[13] Jolly B, Grant J. The good assessment guide: a practical guide toassessment and appraisal for higher specialist training. London: JointCentre for Education in Medicine, 1997.

[14] Collins RE. Surgeons and the new deal—good deal or raw deal? AnnR Coll Surg Engl 1995;77:297–8.

[15] Bulstrode C, Bell Y, Gray M. Senior house officers: the lost tribes.Br J Hosp Med 1993;50:572–3.

[16] Grant J, Marsden P. The plight of senior house officers—some facts.Postgrad Med J 1989;65:869–71.

[17] Livingstone JI, Thomas JM. Surgery. Training or derailment? Lancet1996;348(suppl 2):25.

[18] Reed MWR. Evaluation of surgical training—urgent improvementneeded. Ann R Coll Surg Engl 1993;75:198–9.

[19] Maguire P. Assessing clinical competence. BMJ 1989;298:4–5.[20] NHS Management Executive. Junior doctors, the ‘New Deal’ . Work-

ing arrangements for hospital doctors and dentists in training. Lon-don: NHS Management Executive, 1991.

[21] Working Group on Specialist Medical Training. Hospital doctors.Training for the future: the report of the Working Group on SpecialistMedical Training. Department of Health, London, 1993.

[22] Hilton JR, Shiralka SP, Samsudin A, et al. Disruption of the on-callsurgical team. Ann R Coll Surg Engl 2002;84(suppl):50–3.

[23] Baldwin PJ, Paisley AM, Paterson-Brown S. Consultant surgeons’opinion of the skills required of basic surgical trainees. Br J Surg1999;86:1078–82.

[24] Paisley AM, Baldwin PJ, Paterson-Brown S. Feasibility, reliabilityand validity of a new assessment form for use with basic surgicaltrainees. Am J Surg 2001;182:24–9.

[25] Altman DG. Practical statistics for medical research. London: Chap-man & Hall, 1991.

80 P. Driscoll et al. / The American Journal of Surgery 186 (2003) 77–80