1. Cheng A, Grant V, Dieckmann P, Arora S, Robinson T ,Eppich W. Faculty Development for Simulation Programs; Five Issues for the Future of Debriefing Training. Simul Healthc 2015; 4:217-22.

2. Bromley LM The Objective Structured Clinical Exam- practical aspects. Curr Opin Anaesthesiol. 2000 Dec;13(6):675-8

o Assess the ability of simulation scenarios to evaluate ACGME milestones both objectively and systematically for Pediatric Critical Care and Pediatric Anesthesiology fellows.

o Create replicable scenarios with which to test the development of professionalism and communication tools throughout fellowship.

o Unique scenarios were created with these specific objectives to capture the milestone rubrics. All scenarios were tailored to be specific for the targeted specialty.

(1) Additionally, an attending may not have the opportunity to observe each fellow in an equal clinical setting to accurately assess and compare their professionalism growth. The use of simulation via objective structured clinical exams (OSCEs) is a well-developed tool for learning new techniques in undergraduate medical education. However, there is limited data on the use of OSCEs at the fellow level to assess professionalism. (2)

Combined survey data was analyzed from all scenarios. The surveys were based on a 5-point Likert scale. (1= strongly disagree, 5= strongly agree). Data is reported as means (M) and range (Min:Max).

Data regarding inter-rater reliability and comparison of faculty evaluation is on-going.

Background Example Grading Sheet

Objectives Results

Methods

References

9th International Pediatric Simulation Symposium and Workshops 2017

1-3 June, Boston, MA, USA

Simulation-Based Evaluation of Professionalism Milestones in Pediatric Sub-Specialty Residents

Melissa Smith, MD *1, Benny L. Joyner, Jr., MD, MPH1, Eva Waller, MD2, Rebecca Smith, MD1, Sofia Aliaga, MD, MPH2, Gene Hobbs, CHSE1, Robert Isaak, DO3, Kimberly Blasius, MD 3

1

University of North Carolina 1Department of Pediatric Critical Care Medicine, 2Department of Neonatology, 3Department of Anesthesiology

PO.ID10-5

o OSCE scenarios were designed to target difficult to assess milestones. The following categories were chosen to be assessed bi-annually in both the Pediatric Critical Care and Pediatric Anesthesiology fellowship programs:

o A checklist of tasks and global score mapped to the ACGME Subspecialty Milestone was completed during the simulations by 2 faculty per station (see grading rubric). Post-scenario surveys were completed by all faculty and fellows.

o The grading sheet includes 2 separate scores. The first is a Global Score, which is how the rater subjectively feels the the fellow performed. The second is a calculated score, completed by the study staff based on the specified tasks completed during the scenario.

o The observed fellow also has the opportunity to complete a grading sheet on their own performance.

o Post-OSCE surveys are completed anonymously by all fellows and faculty members.

Ped ICU Milestones:

o Transfer of Care with Seamless Transitions

o Working in Inter-professional teams

o High Standards of Ethical Behavior

Ped Anesthesiology Milestones:

o Interdisciplinary and Transition of Care

o Receiving and giving feedback

o Communication with Patients and families

Survey of Faculty (n=21) M RangeIt is important to have a milestone assessment program 4.7 3: 5

It was useful to observe these scenarios specifically 4.66 3: 5

The case was appropriate for the objective 4.65 4: 5The grading rubric was easy to use 4.39 3: 5

Survey of Fellows (n=10) M RangeThese scenarios helped my understanding of potential communication issues 4.53 4: 5

The scenarios were believable 4.75 4: 5I understood the purpose of this activity prior to participating 4.33 3: 5

Conclusions

o The Milestone simulation exercise was well received by all participants.o Faculty agreed it was useful to observe fellows objectively with the intention of

milestone evaluations. o All fellows within the same discipline were evaluated in a consistent manner,

allowing for an equal comparison of skill and knowledge across identical scenarios and situational variables.

o Plans to validate the grading sheets to ensure scenarios are objective and reproducible.

o We are currently in the process of adapting scenarios and rubrics to other pediatric subspecialties.

The Accreditation Council for Graduate Medical Education (ACGME) began Resident Subspecialty Milestones in 2015 to assess personal development of subspecialty residents (fellows). Many of the advanced level training milestones are targeted at professionalism and team communication, which are often very challenging to assess on a daily basis due to limitations such as clinical demands and time. As a result, feedback may be sporadic or overlooked.

Contact InformationMelissa Smith, MD

UNC Pediatric Critical Care, Chapel Hill NCMelissa_Smith@med.unc.edu

Scenario Objectives

Milestone Competency

Level Milestone Rubric Specific Action or Behavior (Y/N) Evaluation Comments

1. Accurately assesses situation

2. Brainstorms with the team all management options

3. Is able to request assistance for a difficult airway.

Working in inter-professional teams to enhance patient safety

1 Does not seek input from inter-professional team members.

1. Summarizes out loud2. Asks for suggestions of stridor management

1.

2.

2 Understands unique knowledge base of each team member but is unlikely to agree to their management suggestions.

1. Shares ideas and asks for suggestions throughout the scenario2. Acknowledges patient is a difficulty airway3. Requests needed equipment

1.

2.

3.3 Seeks input of other

professionals and is an excellent team player.

1. Calls for Anesthesia back up and gives concise summary of events2. Continues to remain active after anesthesia takes the airway

1.

2.

4 Excellent team leader who considers all team member suggestions, enhancing patient quality

1. Asks individually for RN, RT, or anesthesia suggestions for management2. Will enact other’s suggestions

1.

2.

5 Superb leader. Fully engages all members of the inter-professional team

1. Engages and supports all members of the team

1.

Calculated Score_____________________GlobalScore_____________________