Research Forum Abstracts

their evaluation process gave “excellent insight into their performance” increased from39% (11 respondents) in the first questionnaire to 93% (28 respondents) in thesecond questionnaire (p�.01). The percentage of respondents who either “somewhatagreed” or “strongly agreed” that their current evaluation process “assisted theirprofessional growth and development” increased from 36% (10 respondents) in thefirst questionnaire to 87% (26 respondents) in the second questionnaire (p�.01).Similarly, statistically significant increases in resident satisfaction were seen in theanalysis of responses to each subsequent question addressing resident insight into his/her performance level and professional growth along each of the six specific ACGMEcore competencies (p�.01) with the new evaluation process.

Conclusions: A consensus faculty generated summary document based in the sixACGME core competencies provided more valuable insight and offered greateropportunity for professional improvement than did standard written monthlyevaluations and biannual assessments in the opinion of our residents. Our studysuggests that, for our residents, this evaluation process was an effective means ofproviding resident evaluation and feedback as well as communicating clearimprovement goals and continual assessment of resident progress towards theirachievement.

288 Assessment of Inter-Rater Reliability In Evaluationof Emergency Medicine Resident Core CompetencePerformance In Trauma Simulation

Gupta A, Preblick-Salib C, Sandoval M, Stoller M/Beth Israel Medical Center,New York CIty, NY; Beth Israel Medical Center, New York City, NY

Study Objective: Simulation is an important teaching and evaluation tool foremergency medicine (EM). The Accreditation Council for Graduate MedicalEducation (ACGME) expects residency programs to evaluate the performance ofcritical actions and to assess proficiency in specific core competencies (CC). EMprograms use CCs in both the creation of curricula and in resident evaluation. In thisstudy, we measure inter-rater agreement of EM faculty in measuring traumasimulation performance by third year residents.

Methods: During 11 standardized trauma simulations testing 3rd year EMresidents, 3 trained attending EM faculty categorized resident performance in real-time using a standardized form that assessed 5 core competencies. One additional EMattending directed the simulation in real time, then categorized the resident’sactivities using the same 5 CC measures immediately after the simulation. Inter-rateragreement was assessed using Fleiss free-marginal kappa coefficients.

Results: 11 residents were evaluated by 4 EM attending raters into 5 categories(1-5 scale) for each of 5 CCs. Fleiss free-marginal kappa coefficients for the corecompetencies were: medical knowledge (k � 0.26), systems-based practices (k �0.05), interpersonal & communication skills (k � 0.38), patient care (k � 0.28),professionalism (k � 0.39). Values of kappa (k) can range from �1.0 to 1.0, with�1.0 indicating perfect disagreement below chance, 0.0 indicating agreement equalto chance, and 1.0 indicating perfect agreement above chance. Kappa values � 0.70are acceptable for basic research purposes and values � 0.80 are considered necessaryfor high-stakes decisions. Limitations of our study include small sample size andconfounding variables such as preconceived impressions of the residents by theattending raters.

Conclusion: Our study found low inter-rater agreement of EM core competenciesin a trauma simulation setting. Although this study was limited by sample size and asingle EM training program, it indicates a need for more training of raters and abetter understanding of the CCs and their application in the evaluation process.Future studies should include larger sample sizes of both subjects and raters, andshould consider utilizing unbiased raters without previous interaction with testsubjects.

289 Orientation Program In Emergency Medicine forIncoming Emergency Medicine Residents:Improving Medical Knowledge and ResidentReadiness of EM1 (PGY2) Residents

Kirkland P, Phillips WJ, Lerant A, Jones J/University of Mississippi MedicalCenter, Jackson, MS

Background: The University of Mississippi Medical Center’s emergency medicine(EM) residency has been a PGY 2,3,4 program with residents completing their PGY1year in other departments. Despite a 3-day orientation in July, residents often began

their EM clinical duties with varying degrees of skill and knowledge, reflecting the

Volume , . : September

differences in their PGY1 experience. To address this variation in intern preparation,a full one-month orientation program in emergency medicine (OPEM) was designedto standardize readiness for the clinical practice of EM.

Study Objectives: This study had two objectives: 1) To assess the improvement inmedical knowledge over the course of the OPEM, and 2) To assess participantsatisfaction with the OPEM. Data gathered are being used to improve upcomingtraining.

Methods: This study, at a tertiary referral center, involved six trainees whoparticipated in the consolidated OPEM during the final rotation of the PGY1 year.To assess changes in medical knowledge at the end of the training month, weconstructed three formative exams: a multiple choice test addressing the procedurestaught, a short-answer emergency medicine imaging test and a short-answer clinicalanatomy practical test. Each test was administered on the first day of the trainingmonth (pre-test) and on the last day of the month (post-test) to the six participants.Additionally, we analyzed the results of the annual in-training exams and comparedthe EM 1 results of OPEM participants to the scores of EM 1 residents prior to theimplementation of the OPEM. Participant satisfaction was measured at the end of themonth with a 60-question, 5-point Likert-scale questionnaire addressing organizationand didactic content. The OPEM included training modules in emergency patientdiagnostics, airway management, vascular and spinal access, and therapeuticinterventions. The modules integrated critical clinical anatomy, emergency imaging,pathophysiology, pharmacology, and hands-on training in procedural skills.Emergency medicine scenarios written for high-fidelity full body simulators and EMshadowing experiences were also used to train residents in patient management,communication and documentation skills.

Results: Our formative evaluations suggest that the residents’ medical knowledgein clinical anatomy, imaging interpretation, and procedures significantly improvedduring the OPEM. Also, after the implementation of OPEM, residents achievedsignificantly higher in-training exam scores than prior to the program implematation.Participants agreed or strongly agreed to 86% of the positive statements in thesatisfaction survey.

Conclusions: This study indicates that the OPEM was very successful instandardizing intern orientation for beginning the EM1 experience. Collected datawill allow further modification and enhancement of this training platform.

290 The Case for Teaching Face-To-Face (Tomahawk,Ice-Pick or Inverse) Intubation

Venezia D, Wackett A, Remedios A, Tarsia V/Stony Brook University, StonyBrook, NY

Study Objectives: Determine the feasibility of endotracheal intubation using face-to-face technique in both reclining and sitting positions as compared to standardintubation and to ascertain the technique students felt was easiest to successfullyperform.

Methods: Study Design: Randomized prospective trial. Subjects: 34 paramedicstudents and 21 medical students. Setting: EM Teaching Lab. Interventions:Intubations were performed using a McIntosh blade on a standard full bodymannequin placed on a paramedic stretcher. Medical students were taught standardintubation (SI) and face-to-face sitting (FFS) intubation only. In addition to SI andFFS techniques, paramedic students were taught face-to-face reclining (FFR)technique. Each student performed procedures during testing in random order whilebeing videotaped. Raters measured the time on video from the moment they touchedthe mannequin with the laryngoscope until the final time they passed theendotracheal tube. Inter-rater agreement was 0.99, p �.001. Surveys were completedto ascertain which method students thought was easiest and any previous intubationexperience the student might have had. Outcomes: % failures to intubate and %successful intubations at 15, 20 and 30 seconds. Failure was defined as the inability toplace the tube in the trachea. Data Analysis: Paired statistical methods were used toevaluate the different intubation techniques. McNemar and Cochran’s Q tests wereused to compare outcomes (intubation failure, completion within 15, 20 and 30seconds).

Results: Of the 55 students, 53 (33 paramedics and 20 medical students)completed all three trials on video. All intubations were successfully accomplishedexcept 5 failed standard intubations (1 paramedic student failed to intubate within 2min and 4 med students placed tube in esophagus, P � .025). At times 15 and 20seconds, the rates of successful intubation were higher for the FFS than the SI formedical students alone (55% versus 20%, P � .04 and 70% versus 25%, P � .004)and for the combined student groups (81% versus 66%, P � 0.04 and 89% versus

68%, P � .001), respectively. There was no statistical difference in success rates at 15

Annals of Emergency Medicine S95