Running an Objective Structured Clinical Examination on a Shoestring Budget

Dan Poenaru, MD, Diane Morales, BA, RN, Ann Richards, BA, RN, H. Michael O’Connor, MD, MSc, Kingston, Ontario

BACKGROUND: A major obstacle in the wide im- assessing the interpersonal skills of medical students,4 as plementation of objective clinical structured ex- well as their clinical competence in problem solving.5 Both aminations (OSCEs) is their high cost, averaging Cohen et al’ and Sloan et aP7 showed the method to be $200 to $300 (Canadian dollars, CDN) per candi- highly reliable in assessing the clinical skills of surgical res- date and estimated at up to CDN$SOO per candi- idents, calling the OSCE a “new gold standard for evalu- date if all “hidden” administrative costs are in- ating postgraduate clinical performance.“6 The specific do- eluded. mains sampled by the OSCE have led to their successful

METHODS: A detailed cost analysis of preparing incorporation in integrated postgraduate evaluation pro- and administering OSCEs at 1 institution was un- grams in surgery.s dettaken over 2 years. An 18-station, 6-minute- Most of the criticism against OSCEs has centered around per-station OSCE was given to a 72-student their high cost. Initial reports on OSCE costs have provided third-year medical class. very disparate data, ranging from a low of $11 per student,’

RESULTS: The total cost of the OSCE was CDN&OlO, or $70 per student. The key factors in

to a high of $1,200 p er student in Medical Council of Can- ada’s pilot licensure report.‘” Several factors contribute to

reaching this significantly lower per-student OSCE cost included: judicious use of standard-

this huge variability. First, the format of individual OSCEs

ized patients, use of academic faculty for (number of stations, number of standardized patients, num-

preparing and marking the stations, and de- ber of raters) is quite variable. This would have a direct

creased secretarial and other administrative impact on the direct examination costs. Second, there are

costs. institutional differences in administrative and faculty costs,

CONCLUSIONS: Data suggest that OSCEs can be and available educational resources. Third, purpose of the

set up with reasonable cost and limited re- examination (“high stakes” versus “low stakes”) will influ-

sources even in smaller institutions. Cost should ence cost through differences in examination security and

not be considered a major obstacle in implement- amount of donated versus paid faculty time. Finally, OSCEs

ing this excellent examination type in undergrad- are notorious in their “hidden” administrative costs, and

uate medical education. Am J Surg. low-cost reports are often limited in their expense reporting

1997;173:538-541. 0 1997 by Excerpta Medica, process.‘@ These “hidden” administrative costs include sal-

Inc. aried time of faculty, secretaries and other staff, as well as potential indirect costs of facilities. A thorough recent anal- ysis aimed at providing guidelines for calculating true OSCE

T he objective structured clinical examination costs estimated between $496 and $870 per student,‘” al- (OSCE) has emerged during the past 2 decades as a though this was a hypothetical calculation rather than an choice assessment method in undergraduate medi- actual examination cost.

tine. This examination type involves direct observation of Convinced of the usefulness of the OSCE, yet unable to clinical performance using standardized patients and struc- cover such a high cost in our institution, we undertook a tured checklists for marking. A variety of areas of medical detailed analysis of actual, as opposed to hypothetical, costs performance can thus be evaluated, including interviewing skills, physical examination skills, interpersonal skills, and

of running an OSCE in a small medical school. Our purpose was to derive detailed actual costs for OSCEs in our insti-

SO on. The OSCEs’ reliability and validity have been suc- cessfully demonstrated,’ and its use has been more recently

tution, and then compare these with previously released

expanded to postgraduate medicine as well as medical li- data in order to identify useful cost-cutting methods for small institutions.

censure.‘.’ OSCEs have found a significant role in undergraduate and

postgraduate surgical training. They have been successful in SUBJECTS AND METHODS The undergraduate medical class at our institution has 72

From the Departments of Surgery and Emergency Medicine, students. OSCEs form part of the evaluation in the Clinical Faculty of Medicine, and School of Nursing, Queen’s University, Skills course spanning all 3 pre-clerkship years. This course Kingston, Ontario. aims at refining the students’ clinical assessment skills and

Requests for reprints should be addressed to Dan Poenaru, MD, Hotel Dieu Hospital, 166 Brock Street, Kingston, Ontario K7L 5G2.

their integration into a comprehensive therapeutic ap-

Manuscript submitted August 13, 1996 and accepted in revised preach towards clinical problems. OSCEs are used in this form October 16, 1996. course for the purpose of summative evaluation, contrib-

uting towards a pass/fail/honors course grade.

538 0 1997 by Excerpta Medica, Inc. 0002-961 O/97/$1 7.00 All rights reserved. PII SOOO2-9610(97)00007-X

1 OBJECTIVE STRUCTURED CLINICAL EXAMINATION ON A TIGHT BUDGET/POENARU ET AL 1

TABLE I Comparison of Costs of Two Model OSCEs and Current Study

Budget Item High-end Cost** ($)+ Low-end Cost*” ($)+

Principal implementation staff Project leader 10,000 0 Standardized-patient trainer 20,000 20,000 Secretary 15,000 15,000

Total 45,000 35,000 Examination development phase

Review of course objectives; scenarios creation 600 0 Faculty honoraria for workshops 12,000 0 Catering for case development workshops 360 360

Total 14,200 360

Examination production phase Patient training 2,000 2,000 Printing of examination materials 3,000 500 Artifacts, props, and supplies 1,000 800

Total 6,000 3,300 Examination administration phase

Standardized patients 13,500 13,500 Physician examiners 13,500 0 Physician markers i ,800 0 Support staff 2,700 2,700

Catering 3,000 3,000 Tofal 34,500 19,200

Post-examination analysis and reporting phase Data entry and review 1,600 0 Statistical analysis 1,600 1,600 Report generation 1,500 0

Total 4,700 1,600 Total for entire implementation 104,400” 59,460” Total per-student cost 870 496

* Table format and first two columns information from Reznick'" and others. +A// prices in Canadian dollars. i Other pard hours inciuded in in&id& phase cost breakdowns. 5 Based on 720 students.

Based on 72 students.

Current” ($)+

0 aoo*

1,040*

i ,840

0 0

N/A 0

510 60 70

640

1,600 0 0

500

150 2,250

0 280

0 280

5,010” 70

The third year examination is given during one evening once a year. It consists of 18 stations of 6 minutes each administered simultaneously on 2 floors. This allows half the class (36 students) to be examined at the same time, and the entire class thus requires 2 consecutive runs of the exam, for a total examination time of 3 l/2 to 4 hours. Twelve of the stations were “observed,” using professional standardized patients (SPs), one for each station. The other six stations were “unobserved”, involving questions based on photographs, radiographs, and laboratory tests. Twenty- four voluntary faculty members act as evaluators, one in each observed station. The students’ responses are scored by the evaluators on pre-printed checklists, which include both specific desirable responses as well as two global rat- ings.

The preparation and administration of OSCEs at our in- stitution falls within the responsibility of one full-time Clin- ical Skills coordinator (AR) and one full-time Standardized Patient coordinator (DM) whose duties include various ac- tivities of the Clinical Leaning Center in addition to several OSCEs in undergraduate medicine. The overall implemen- tation of the OSCE rests with a faculty undergraduate Clin- ical Skills director (HMO). Individual stations are designed by clinicians in each area of specialty, then pass several

revisions through a multidisciplinary faculty OSCE com- mittee.

Because the Clinical Learning Center is an individual costing center, accurate records of both materials and man- power were kept. The 1994/5 academic year data were used for calculations.

Costing data were initially divided into direct costs (ma- terials, honoraria) and indirect costs (hours of salaried work). To compare our data with the standards set by Rez- nick et al,‘” we then grouped expenses in accordance with their guidelines into four areas: examination development, production, administration, and analysis/reporting.

RESULTS The total cost incurred in running 1 OSCE exam to our

72 students was CDN$5,010. The key expense components were the standardized patients ($2,1 lo), the salaried hourly wages of the OSCE coordinators in examination prepara- tion and administration ($2,120), and the employee hon- oraria for the examination evening ($500). Our SPs, having each participated in several previous OSCEs, were able to perform adequately after an average of only 2 hours per station. As the examination was held outside regular work hours, all supervisory staff had to be paid a direct exami-

THE AMERICAN JOURNAL OF SURGERY@ VOLUME 173 JUNE 1997 539

TABLE II Suggestions for Potential OSCE Cost-saving Measures

Use volunteer faculty for examination preparation and supervision Share coordinator/secretary jobs with other tasks/departments/faculties Condense total examination time Share skilled standardized patients with other faculties Use one standardized patient per station Use students for examination setup and cleanup Do not cater food-use supermarket refreshments Perform own statistical analysis

nation honorarium separate from their salary pay. The em- ployees’ regular salaried work was comprised mostly of standardized patient training (40 hours), examination prep- aration (29 hours), and post-exam marking and administra- tion (18 hours).

Table I compares Reznick’s’” hypothetical “high-end cost” and “low-end cost” with our actual costs. While all four project phases see significantly lower costs in our actual data, the “principal implementation staff’ makes the largest impact. This occurs through the absence of full-time or part-time project staff exclusively dedicated to one OSCE-rather, salaried time spent by two Clinical Skills coordinators is accounted for. All faculty costs are based on voluntary work, a situation both in keeping with current institutional practices and consistent with commonly im- plicit academic faculty obligations.

COMMENTS Multiple efforts are needed in a production of an OSCE,

and costing estimates must take all expenses, obvious and hidden, into account. A common error in this respect is accounting only for “out-of-pocket” expenses such as ma- terials and Sl’s, yet ignoring salaried time usage. We have attempted to uncover all associated costs, using the excel- lent model provided by Reznick and associates.‘@

Another confounding variable is the remuneration of faculty work. While professional bodies and licensing or- ganizations routinely pay faculty honoraria, the majority of medical schools need to rely on faculty time paid only indirectly or simply volunteered. This situation is in no way different from any other academic effort of medical faculty, and this allows medical schools to maintain ed- ucation without bankruptcy. We, like others” have used this system in our own institution, and believe that a similar model can be adopted by most North American schools. Volunteered faculty time in the initial exami- nation development phase also reduces the cost differ- ence between a first-time and subsequent examinations, although the time investment will necessarily be largest the first year of the OSCE.

Table I reveals that savings at our institution were re- alized in most areas, yet are most marked in staff costs. Rather than having specific secretarial and standardized- patient trainer positions, we relied on the Standardized Patient and Clinical Skills coordinators who devoted to the OSCE only the fraction of their time necessary for the task. This situation is more in keeping with the re- ality of a small institution, and again radically different from the standard within, for example, a licensing body.’

We have, however, included our nonfaculty employee costs because most institutions embarking in OSCEs will have to face them. This position contrasts with that of Carpenter, ’ ’ who suggests that all staffing costs should be absorbed by the university and can, therefore, be left out of the equation.

Another area of significant differences is in the cost of the 3’s. While the hourly rates paid were similar, we, like oth- ers, ‘z have found that these skilled amateurs performed ex- cellently after an average of only 2, rather than 5, hours of training per station. This was at least in part because the SF’s’ skills were kept up continuously through their involve- ment with other health sciences OSCEs. More importantly, compressing the examination trme to 1 (long) evening rather than 1 l/2 days resulted in a significant saving in SF’ fees. This shortened examination time was achieved through shorter, &minute stations. While reliability data are not available for our examination, we expect them to fall within the general range for this examination.” We have specifically not noticed a decline in the quality of the examination because of the shortened testing time. With regard to the SF’s, we too have been able to run the “ob- served” stations with one SP in each, thereby again reduc- ing cost, without observing any significant fatigue effects.

Finally, we have found that most administrative expenses can be significantly reduced. These include printing, cater- ing (replaced by simple refreshments), artifacts and other supplies, and examination support staff (students can act as excellent examination helpers for setup and cleanup). Fol- lowing the examination, statistical analysis can be per- formed by the faculty using commonly available software packages without great difficulty.

A listing of our recommendations for potential OSCE cost-saving areas in presented in Table II.

Comparisons between expenses of running OSCEs in sep- arate institutions are plagued by the limitations previously mentioned related to examination format, purpose, atten- tion to “hidden costs,” and availability of voluntary faculty support. As such any conclusions from our study need to be critically evaluated in the context of the reader’s educa- tional environment and goals. We have endeavored to show the feasibility of running low-cost OSCEs, not the expectation that all OSCEs are inexpensive. The infor- mation presented will be unlikely to reduce drastically the cost of high-stakes examinations already setup and run- ning-but it should guide the prospective educator in mak- ing wise choices in the setup of these examinations, and maybe point out some areas were the budget can be “thinned.” “ Shoe-string budget” OSCEs are not for all, but

540 THE AMERICAN JOURNAL OF SURGERYa VOLUME 173 JUNE 1997

(OBJECTIVE STRUCTURED CLINICAL EXAMINATION ON A TIGHT BUDGETIPOENARU ET AL 1

no one should dismiss OSCEs based on their expense before carefully considering the above budget considerations

REFERENCES 1. Petrusa ER, Blackwell H, Rogers LP. An objective measure of clinical performance. Am J Med. 1987;83:34-42. 2. Cohen R, Reznick RK, Taylor BR, et al. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990;160:302-305. 3. Reznick R, Smee S, Rothman A, et al. An objecrlve structured clinical examination for the licentiate: report of the pilot project of the IMedical Council of Canada. Acad rued. 1992;67:487-494. 4. Sloan DA, Donnelly MB, Johnson SB, et al. Assessing asurgical residents’ and medical students’ interpersonal skills. J Surg Res. 1994;57:613-618. 5. Sloan DA, Donnelly MB, Schwartz RW, et al. Assessing medical students’ and surgery residents’ clinical competence in problem solv- ing in surgical oncology. Ann Surg On&. 1994;1:204-212. 6. Sloan DA, Donnelly MB, Schwartz RW, Strudel WE. The ob- jective structured clinical examination: the new gold standard for

evaluating postgraduate chnical ptrfi~rmance. Ann Surg. 1995;2223735-742. 7. Sloan DA, Donnelly MB, Johnscm SB, et al. Use of an objective structured clinical examination (OSCE) to measure improvement in clinical competence during the surgical internship. Surgery. 1993;115:343-351, 8. Schwartz RW, Donnelly MB, Sloan DA, et al. Assessing senior residents’ knowledge and performance: an Integrated evaluatton program. Surgery. 1994;116:634-638.

i 9. Frye AW, Rrchards BF, Philp EB, Phrlp JR. Is it worth it! A look at the costs and benefits of an OSCE for second-year medical stu- dents. Med Teacher. 1989;11:291-293. 10. Resnick RK, Smee S, Baumhrr JS, e t al. Guidelines for esti- mating the real ctlst of an objective structured clinical examination. Acad Med. 1993;68:513-517. 11. Carpenter JL. Cost analysis of objective structured clinical ex- aminations. Acad Med. 1995;70:828-833. 12. Cusimanu MD, Cohen R. Tucker W. et al. A comparative anal- ysis of the costs of administration of an CSCE. Acad Med. 1994;69:573-576.

THE AMERICAN JOURNAL OF SURGERYE VOLUME 173 JUNE 1997 541