Embed Size (px)
Citation preview
edd(aeisadldbrd1iclr
ei
Ects
RA
The Journal of Emergency Medicine, Vol. 31, No. 4, pp. 437–440, 2006Copyright © 2006 Elsevier Inc.
Printed in the USA. All rights reserved0736-4679/06 $–see front matter
doi:10.1016/j.jemermed.2006.04.015
Education
TEACHING BAYESIAN ANALYSIS TO EMERGENCY MEDICINE RESIDENTS
David C. Grant, MD, Samuel M. Keim, MD, and Janet Telfer, MS
Department of Emergency Medicine, University of Arizona College of Medicine, Tucson, ArizonaReprint Address: Samuel M. Keim, MD, Department of Emergency Medicine, University of Arizona College of Medicine, P.O. Box
245057, Tucson, AZ 85724-5057
iipaOpslsbs
Ti(d(pcstTalufflbc
Abstract—Our objective was to determine if a briefidactic would improve Emergency Medicine (EM) resi-ent performance at using a key evidence-based medicineEBM) concept. We used a prospective, before and after,ssessment of EM resident estimates of post-test pulmonarymbolism (PE) probability for a defined pre-test probabil-ty, computed tomography (CT) and D-dimer results. Theurvey provided test sensitivity, and specificity for D-dimernd CT. Three months later, residents attended a briefidactic conference on how to use Fagan’s Nomogram and
ikelihood ratios (LRs) to calculate post-test probability ofisease. The accuracy of estimates of post-test PE proba-ility was reassessed. The absolute percentage difference inesident estimates from the true post-test PE probabilitiesecreased from 14.5% (95% confidence interval [CI] 9.7%–9.9%) to 4.5% (95% CI 2.0–6.8%) after the educationalntervention. This 10% effect size was statistically signifi-ant, p � 0.002. The study demonstrates the efficacy of theecture method in teaching an EBM concept to EMesidents. © 2006 Elsevier Inc.
Keywords—Bayes’ Theorem; teaching; education, med-cal; EBM; evidence-based medicine
INTRODUCTION
vidence-based medicine (EBM) is one philosophy oflinical problem-solving (1). Explicit EBM didactic lec-ures are a common tool to teach residents the requisitekills for accessing, interpreting and translating scientific
Education is coordinated by Stephen R. Hayden,Diego, California
ECEIVED: 10 January 2005; FINAL SUBMISSION RECEIVED:
CCEPTED: 11 April 2006437
nformation into clinical practice (2–6). However, thisnstruction format has not been demonstrated to be su-erior to the traditional residents’ journal club approacht acquiring appraisal skills of the medical literature (7).ther methods for learning statistics and critical ap-raisal certainly exist and include research modules,elf-directed programs, lecture series, and problem-basedearning. Some have challenged the plausibility that aingle educational intervention can produce learningenefits, and whether EBM is even a teachable set ofkills (3).
Several themes are common to EBM curricula.hese include understanding statistical principles of
nvestigation, results interpretation and applicability8 –10). The principles range in complexity and resi-ent physicians possess varied comfort learning them2,10 –12). Bayesian analysis is a fundamental princi-le of most EBM curricula and can be used by clini-ians to revise the odds or probability of disease in apecific patient after a diagnostic test result is ob-ained. Originally developed in the 18th century byhomas Bayes, the analysis generates a post-test prob-bility of disease as the product of the disease preva-ence and the likelihood ratio of the diagnostic testsed (13). The likelihood ratio (LR) is a stable per-ormance characteristic of a diagnostic test that re-ects its strength (14). The LR can be calculated foroth negative and positive tests and informs the clini-ian by how much (the magnitude) a test result
the University of California San Diego Medical Center, San
gust 2005;
MD, of
25 Au
cpsc
srSi
TmpvstEIFyuanducFmtieapwsttTaea
Tm(o9Lstp
twtittmatpA8
F
F
T
5555222
438 D. C. Grant et al.
hanges the likelihood of disease (Figure 1). Althoughossessing potentially far more value to clinical deci-ion-making than simple sensitivity or specificity, theoncept may seem complex to some physicians.
Our objective was to determine whether a singlecenario-based EBM didactic is efficacious in teachingesidents the skill of applying likelihood ratios (LRs).ubsequent inquiry to evaluate clinical effectiveness
s a logical follow-up to this study.
MATERIALS AND METHODS
his study was a prospective, before and after, assess-ent of Emergency Medicine residents’ estimates of
ost-test PE probability after a simple educational inter-ention. The Human Subjects Committee of our Univer-ity exempted the study from formal review. Study par-icipants were residents from a university-basedmergency Medicine program in winter-spring of 2003.
nformed consent was obtained from participants. Inebruary 2003, resident physicians completed an anon-mous survey of their estimates of PE probability basedpon a set of selected test results and pre-test PE prob-bilities (Figure 2). Residents were assigned subjectumbers. In May 2003, one of the authors (D.C.G.)elivered a 30-min didactic presentation, during sched-led residency conference time, on LRs and their appli-ation to calculating post-test disease probability viaagan’s Nomogram (14). Laminated versions of the No-ogram (Figure 3) were distributed to attendees, and at
he conclusion of the lecture the residents were surveyeddentically to the first examination except that they werencouraged to use the Nomogram cards. There was nodditional structured educational intervention on theost-test estimation of disease likelihood, and residentsere unaware that they would be evaluated until the
econd examination. Individuals participating in the firstest administration were eligible to complete the secondest if they attended the educational conference on LRs.he tests were proctored and collected by one of theuthors (D.C.G.). The measured outcome was the differ-nce in residents’ estimates of PE from the exact prob-
Bayes’ theorem: Post-test odds = pretest odds × LR
LR+ = sensitivity/1 specificity
LR = 1 sensitivity/specificity
–
– –igure 1. Bayes’ Theorem and Likelihood Ratios.
bility of disease as predicted by Fagan’s Nomogram.2
his was determined for eight different scenarios. Esti-ates of spiral chest CT and D-dimer sensitivity for PE
60% and 90%, respectively) were used in the calculationf LRs. Corresponding specificity for CT and D-dimer of0% and 60% was used (15,16). Positive and negativeRs were calculated for PE, rapid-ELISA D-dimer, andpiral chest CT. Post-test probabilities were drawn fromhese and Fagan’s Nomogram. The derived post-test PErobabilities are depicted in Table 1.
Seventeen residents completed both test administra-ions and were included in the analysis of results. Erroras defined as the absolute value of the difference be-
ween resident estimates and true post-test PE probabil-ties. Analysis was performed on the average error acrosshe eight presented scenarios for each resident. Becausehe distribution of pre-/post-intervention differences inean error was skewed, improvement in estimation was
ssessed using the non-Parametric Wilcoxon signed-rankest. This study was powered at 97.8% to detect a meanost-test PE probability estimate improvement of 10%.nalysis was conducted using STATA software, version.0 StataCorp, College Station, TX).
Please complete this survey without assistance, except a calculator. Your answers may be reported anonymously in published form. Please indicate the last 4 digits of your social security number across the top of this page. Estimate how you think various test results affect the probability of disease.
Imagine two patients that you suspect may have a pulmonary embolus (PE). You estimate the chances of patient X having a PE at 5% and patient Y at 20% (pre-test probabilities).
D-dimer or spiral CT are tests used to diagnose PE. Assume that the D-dimer has 90% sensitivity and 60% specificity for PE, and the CT has 60% sensitivity and 90% specificity for PE.
Please estimate the post-test probability of PE by completing each underlined space below. Post-test probability of PE Patient X (5% pre-test prob) with an abnormally high D-dimer............._______% or a normal D-dimer..................................._______% or a spiral CT indicating PE........................_______% or a spiral CT without evidence of PE........._______%
Post-test probability of PE Patient Y (20% pre-test prob) with an abnormally high D-dimer............._______% or a normal D-dimer..................................._______% or a spiral CT indicating PE........................_______% or a spiral CT without evidence of PE........._______%
igure 2. Survey.
able 1. The Conditional Post-test Probability ofPulmonary Embolus from Fagan’s Nomogram
Pre-test PE probability/test results Post-test PE probability
%/D-dimer abnormal 12%%/D-dimer normal 1%%/PE on CT 24%%/no PE on CT 3%0%/D-dimer abnormal 35%0%/D-dimer normal 3%0%/PE on CT 58%
0%/no PE on CT 11%
Tep
taw
EipBcfeouamltciuhscTsim
admrtt
F
Teaching Bayesian Analysis 439
RESULTS
he mean absolute percentage difference in residents’stimates of disease probability from true post-test PE
igure 3. Fagan’s Nomogram.
robabilities decreased from 14.5% (95% confidence in- F
erval [CI] 9.7%–19.9%) to 4.5% (95% CI 2.0%–6.8%)fter the educational intervention. This 10% effect sizeas statistically significant, p � 0.002 (Figure 4).
DISCUSSION
vidence-based medicine is a popular construct for clin-cal decision-making and graduate medical educationalrograms. Sackett, Guyatt and others from the Evidence-ased Medicine Working Group have presented andontinued developing an approach combining principlesrom clinical epidemiology, biostatistics, clinical experi-nce, ethics and socioeconomics that allows translationf the scientific literature to decisions regarding individ-al patient care (9,10). Emergency Medicine, like allreas of medicine, is facing an explosion of novel infor-ation and technology made accessible by the scientific
iterature. EBM is a construct of knowledge and skillshat addresses this rapid expansion by expecting clini-ians to search for the best evidence in the literature tonform their decisions. Many believe that patient caretilizing EBM will result in better individual and publicealth outcomes. Incorporating these knowledge andkills sets during residency training would therefore be arucial element in its integration into medical practice.he Bayesian analysis concepts and skills utilized in thistudy are core EBM concepts and are being taught withncreasing frequency in undergraduate and graduateedical education curricula (9,17–19).We demonstrated that a simple traditional lecture was
n efficacious instruction method within an EM resi-ency for one core EBM skill. Likelihood ratios provideore useful clinical information than the traditionally
eported sensitivity and specificity. The positive LR ishe ratio of an expected test result in patients with diseaseo patients without the disease (20). Fagan’s Nomogram
igure 4. Pre- and post-didactic Mean Error of PE estimates.
poaiocbf
LtitcPk
Fosptgpjtm
csictpe(lpv
Etddtbi
1
1
1
1
1
1
1
1
1
1
2
2
22
2
2
2
2
440 D. C. Grant et al.
rovides a quick estimate of how the pre-test probabilityf disease changes for a given test result. The EBMpproach emphasizes the utility and importance of LRsn patient care because they assist with the interpretationf laboratory and imaging results. Although seeminglyomplex, we believe this and other EBM knowledge cane effectively taught using the simple traditional lectureormat.
Pulmonary embolus is a useful condition to illustrateR utility to Emergency Medicine residents. The iden-
ification of this dangerous condition relies on carefulnterpretation of diagnostic tests. Many studies related tohe diagnostic strategies for PEs have also emphasizedentral EBM concepts (21–25). This might recommendE as a useful condition to study whether acquired EBMnowledge affects clinical outcomes.
Several limitations to our study design are evident.irst, the educational intervention was not standardizedr compared to other conventional teaching modalitiesuch as journal club, problem-based learning, self-pacedrograms, or a lecture series. Subjects’ long-term reten-ion was not tested and they were allowed to use nomo-ram cards that might not be as available in actualractice. It is also possible, although unlikely, that sub-ects conducted individual learning related to the educa-ional intervention. If this occurred it could reduce theean error in post-test estimates of PE probability.Although not the intent of this study, an evaluation of
linical benefit, and not only improvement in EBM testcores, is needed for EBM as a whole. Whether increas-ng post-test estimates of PE accuracy by 10% is clini-ally significant is also unknown. Finally, the assumptionhat pre-test disease probability is consistent or known isroblematic. Wide variability in clinicians’ pre-test dis-ase estimates undermine the utility of Bayes’ Theorem26). In the specific case of PE, a Canadian group pub-ished “validated” criteria for estimation of pre-test PErobability (27), but a Norwegian team was unable toalidate the same criteria.
Our study demonstrates the efficacy of teaching anBM concept to EM residents with the traditional didac-
ic presentation method. The resident error in estimatingisease probability improved significantly after a simpleidactic intervention. Further investigations are neededo determine whether this enhanced performance couldenefit patient care and whether the method is reproduc-ble in other institutions.
REFERENCES
1. Haynes R, Sackett D, Gray J, Cook D, Guyatt G. The role ofclinical care research evidence in clinical decisions. ACP J Club
1996;125:A14–6.2. Hardern R. Teaching and learning evidence-based medicine skillsin accident and emergency medicine. J Accid Emerg Med 1999;16:126–9.
3. Hatala R. Is evidence-based medicine a teachable skill (comment)?Ann Emerg Med 1999;34:226–8.
4. Jadad A, Cepeda M. Evidence-based emergency medicine. Tenchallenges at the intersection of clinical research, evidence-basedmedicine, and pain relief. Ann Emerg Med 2000;36:247–52.
5. Jouriles N, Cordell W, Martin D, Wolfe R, Emerman C, Avery A.Emergency medicine journal clubs. Acad Emerg Med 2000;3:872–8.
6. Newman D, Wyer P, Kaji A. Evidence-based medicine. A primerfor the emergency medicine resident. Ann Emerg Med. 2002;39:77–80.
7. Bazarian J, Davis C, Spillane L, Blumstein H, Schneider S. Teach-ing emergency medicine residents evidence-based critical ap-praisal skills: a controlled trial. Ann Emerg Med 1999;24:148–54.
8. Guyatt G, Rennie D. Users’ guides to the medical literature. JAMA1993;270:2096–7.
9. Guyatt G, Rennie D. Users’ guides to the medical literature.Chicago, IL: AMA Press; 2002.
0. Sackett D, Rennie D. The science of the art of the clinical exam-ination. JAMA 1992;267:2650–2.
1. Gill S, Levin A, Djurdjev O, Yoshida E. Obstacles to residents’conducting research and predictors of publication. Acad Med2001;76:477.
2. Jerant A, Lloyd A. Applied medical informatics and computingskills of students, residents, and faculty. Fam Med 2000;32(4):267–72.
3. Hall GH. The clinical application of Bayes’ theorem. Lancet 1967;2:555–7.
4. Fagan TJ. Letter: Nomogram for Bayes theorem. N Engl J Med1975;293:257.
5. Drucker E, Rivitz S, Shepard J, et al. Acute pulmonary embolism:assessment of helical CT for diagnosis. Radiology 1998;290:235–41.
6. Kruip M, Slob M, Schijen J, van der Heul C, Buller HR. Use of aclinical decision rule in combination with D-dimer concentrationin diagnostic workup of patients with suspected pulmonary embo-lism. Arch Intern Med 2002;162:1631–5.
7. Lewis R, Wears R. An introduction to the Bayesian analysis ofclinical trials. Ann Emerg Med 1993;22:1328–36.
8. Whitcomb ME. Why we must teach evidence-based medicine.Acad Med 2005;80:1–2.
9. Coon TP, Miller MA. Teaching evidence-based medicine. AcadEmerg Med 2005;12:675; author reply 675.
0. Hayden S, Brown M. Likelihood ratio: a powerful tool for incor-porating the results of a diagnostic test into clinical decisionmak-ing. Ann Emerg Med 1999;33:575–80.
1. Kline J, Wells P. Methodology for a rapid protocol to rule-outpulmonary embolism in the emergency department. Ann EmergMed 2003;42:266–75.
2. Colton T. The “power” of sound statistics. JAMA 1990;263:281.3. Stein P, Hull R, Patel K, et al. D-dimer for the exclusion of acute
venous thrombosis and pulmonary embolism: a systematic review.Ann Intern Med 2004;140:589–602.
4. Perrier A, Howarth N, Didier D, et al. Performance of helicalcomputed tomography in unselected outpatients with suspectedpulmonary embolism. Ann Intern Med 2001;135:88–97.
5. Clinical policy: critical issues in the evaluation and management ofadult patients presenting with suspected pulmonary embolism. AnnEmerg Med 2003;41:257–70.
6. Phelps M, Levitt A. Pretest probability estimates: a pitfall to theclinical utility of evidence-based medicine? Acad Emerg Med2004;11:692–4.
7. Wells P, Anderson D, Rodger M, et al. Derivation of a simpleclinical model to categorize patients’ probability of pulmonaryembolism: increasing the models utility with the SimpliRed D-
dimer. Thromb Haemost 2000;83:416–20.
