Upload
sherman-garrison
View
226
Download
4
Tags:
Embed Size (px)
Citation preview
Nancy Piro, PhD – No conflicts of interest to disclose
Kim Walker, PhD - No conflicts of interest to disclose
SES:096Voice of the Director: Your experience on what works and doesn’t work with CCC meetings
Nancy Piro, PhDEducation Specialist/Program ManagerStanford University Medical Center
Kim Walker, PhDInstructional DesignerSchool of Medicine, Stanford University
Confidential – For Discussion Purposes Only
Session Overview
Introduction and background on multi-institution survey
Report on survey results from Program Directors at multiple institutions on the CCC process and outcomes
− Emergent themes on directors’ experiences and their roles in the CCC
Quantitative outcomes
Qualitative outcomes
− What directors tell us was ineffective in the CCC process
− Best practices for coordinators supporting a successful CCC
Action planning for implementation of best practices
Confidential – For Discussion Purposes Only
Objectives & Session Focus
This session will focus on understanding and discussing the results from a multi-institution survey on CCC practices from the Program Directors’ perspective.
At the conclusion, participants will be able to:
− Review their own practices in light of the survey outcomes
− Identify best practices that best suit their program or institution needs
− Develop and take home an action plan enabled by these best practices to better facilitate the work of their CCCs
Confidential – For Discussion Purposes Only
Clinical Competency Committee
How the CCC does its work is decided by the Program Director
Confidential – For Discussion Purposes Only
Survey – Fun Facts
What:
− A 14 Question* web-based survey with both quantitative and open ended questions was developed to assess the program director and coordinator experience so far with respect to CCCs.
When and To Whom:
− Between November 12, 2014 and January 15, 2015, the survey link was distributed to a wide sample of directors and coordinators (both within Stanford and across the country to the 15 largest institutions, and to PC networks in OB/GYN, Anesthesia, and Surgery)
Analytic Methods Used – Both Qualitative and Quantitative Analyses were used.
Response: 135 Program Directors and 325 Program Coordinators responded to our survey by January 15, 2015
* Questions detailed on the following slide
Confidential – For Discussion Purposes Only
Survey Questions
1. What's your program?
2. Are you a voting member of your CCC?
3. During your CCC meeting, how much time (on the average) per resident was spent on the milestone assessments?
4. What actions has your program taken to better align your training program with your milestones (e.g., changed curriculum, added didactics, linked current evals to milestones, used spreadsheets to aggregate evaluation data, added new evaluation tools)?
5. What data was used in your CCC meeting for trainee assessment?
6. What data was more valuable in your overall milestone assessments?
7. Please describe any aspects of the milestone evaluation process that you thought were particularly beneficial.
Confidential – For Discussion Purposes Only
Survey Questions (2)
8. Please describe any aspects of the CCC process that were particularly cumbersome.
9. To what extent do milestone evaluations provide for a more thorough semi-annual evaluation?
10.To what extent do milestone evaluations increase our documentation of trainee strengths?
11. Milestone evaluations increase our documentation of trainee weaknesses/areas for improvement.
12.The CCC process for trainee milestone evaluations is worth the time/effort.
13. If not worth the time and effort, why not?
14.Lastly, what did we miss? Are there any other areas of concern or suggestions?
Confidential – For Discussion Purposes Only
How to Apply to Your Institution
CURRENT PRACTICE (Starting State)
POSSIBLE PRACTICE(End State)
CCC Membership and Size
Faculty Pre-work
Alignment of Milestones
Evaluation Tools
CCC Process
Challenges
Confidential – For Discussion Purposes Only
Q1. Distribution of Programs Surveyed – 60 programs responded
Anesthesiology
Cardiovascular disease
Emergency medicine
Internal medicine
Neonatal-perinatal medicine
Nephrology
Obstetrics and gynecology
Orthopaedic surgery
Otolaryngology
Pathology-anatomic and clinical
Pediatric cardiology
Pediatric infectious diseases
Pediatric nephrology
Pediatrics
Physical medicine and rehabilitation
Urology
Other (1 response)
Other (2 responses)
0.00% 10.00% 20.00% 30.00% 40.00%
3.31%
3.31%
3.31%
4.96%
2.48%
3.31%
3.31%
3.31%
2.48%
2.48%
3.31%
3.31%
2.48%
3.31%
2.48%
2.48%
20.75%
29.70%
Confidential – For Discussion Purposes Only
Q2. PDs - voting member of your Clinical Competency Committee (CCC)?
Yes No0%
25%
50%
75%
100%
83.97% 16.03%
Confidential – For Discussion Purposes Only
PC Survey: Who is on your program's CCC?
Program Director
Associate Program Direc-tor
Chief Resident(s)
Rotation Director
Program Faculty Member
Program Director from other program
Attending who does sign out
Nurse
Case Manager
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
88.93%
69.74%
7.01%
25.83%
89.30%
7.38%
21.03%
8.86%
2.21%
Confidential – For Discussion Purposes Only
Q3. How much time (on the average) per resident was spent on the milestone assessments?
1-10 minutes
11-20 minutes
21-30 minutes
31-40 minutes
41-60 minutes
>60 minutes
0.00% 10.00% 20.00% 30.00% 40.00% 50.00%
23.08%
41.54%
18.46%
9.23%
5.38%
2.31%
Confidential – For Discussion Purposes Only
Q4. Alignment of milestones with training: Emergent Themes
18
New forms of evaluation (n=82) New curriculum/didactics (n=26)
Confidential – For Discussion Purposes Only
Q4. Alignment of milestones with training: Emergent Themes
New direct observation tools
“Started surgical skill evaluations”
“Created a computer based tool to provide instant feedback and document milestone progress.”
“…ramped up eval tools over time (mini CEX, tech skills assessment)…”
Confidential – For Discussion Purposes Only
Q4. Alignment of milestones with training: Emergent Themes
20
Dashboard Spreadsheets
“Spread sheets, color coded resident measures such as conference attendance”
“created spreadsheet to make sure our evaluation tools capture all the milestones”
Confidential – For Discussion Purposes Only
Q5. Data used in CCC meetings for trainee assessment
Aggregate Direct Milestone Evalua-
tions
Aggregate Rotation Evaluations
Staff or patient (360) Evaluations
Technical Skills Assessments/Mini-
CEX's
Scholarly Activities
Case Volumes
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
65.38%
93.85%
76.92%
44.62%
63.08%
36.92%
Confidential – For Discussion Purposes Only
Aggregate milestone evaluation scores
Board scores
Conference Attendance
Continuity clinic chart review
Curriculum Vitae
Didactic attendance
Duty Hour Summary
Duty Hour reporting compliance
Evaluation completion rate
In service exam scores
Learning modules completion
Medical records outstanding OP reports
Medical student evaluations
Mock oral results
Narrative evaluation comments
Peer Evaluations
Presentations
Previous evaluation
Procedure logs
Research projects
Self assessment scores
Spider diagrams with comparison data
Continuity clinic attendance
Data included in the CCC Review: Overall Emergent Themes
Confidential – For Discussion Purposes Only
Q6. What data was more valuable in your overall milestone assessments?
Quantitative ag-gregate evaluation
scores
Qualitative comments and direct observa-tions by CCC
members
Qualitative and quantitative data
were equally valuable
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
12.40%
46.51%
41.09%
Confidential – For Discussion Purposes Only
Q7. Beneficial aspects of the milestone evaluation process
24
“Group discussion to clarify our own thinking and understanding of the
Milestones and their meaning”
Committee-based review and discussion
Confidential – For Discussion Purposes Only 25
“…recognition of deficits in our teaching”
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
Confidential – For Discussion Purposes Only
Committee-based review and discussion
26
“Multiple opinions strengthen the rigor of the process.”
Q7. Beneficial aspects of the milestone evaluation process
Confidential – For Discussion Purposes Only 27
“…provides a much more comprehensive look at an
individual resident's performance.”
Q7. Beneficial aspects of the milestone evaluation process
Committee-based review and discussion
Confidential – For Discussion Purposes Only
Q7. Beneficial aspects of the milestone evaluation process
“
“common language”
“more objective evidence”
“more detailed evaluation”
More objective and focused reviews
Confidential – For Discussion Purposes Only
Identification of trainee strengths and areas for improvement
Q7. Beneficial aspects of the milestone evaluation process
Confidential – For Discussion Purposes Only 30
Q7. Beneficial aspects of the milestone evaluation process
Identification of gaps in education (curriculum)
Confidential – For Discussion Purposes Only
Q8. Cumbersome aspects: Emergent Themes
Milestone interpretation
"what are they really getting at
here?"
“contrived”
“nebulous”
“uncalibrated”
Confidential – For Discussion Purposes Only
Q8. Cumbersome aspects: Emergent Themes
33
Acquisition of sufficient data
Another 50 milestone evals due tomorrow?!
I can’t take it anymore!!!
Confidential – For Discussion Purposes Only
Q9. Milestone evaluations provide for a more thorough semi-annual evaluation.
Very Strongly Agree
Strong Agree
Agree
Disagree
Strongly Disagree
Very Strongly Disagree
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
6.87%
19.08%
41.22%
16.79%
9.92%
6.11%
Confidential – For Discussion Purposes Only
Q10. Milestone evaluations increase our documentation of trainee strengths.
Very Strongly Agree
Strong Agree
Agree
Disagree
Strongly Disagree
Very Strongly Disagree
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
6.11%
21.37%
38.93%
20.61%
7.63%
5.34%
Confidential – For Discussion Purposes Only
Q11. Milestone evaluations increase our documentation of trainee weaknesses/areas for improvement
Very Strongly Agree
Strong Agree
Agree
Disagree
Strongly Disagree
Very Strongly Disagree
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
5.34%
24.43%
40.46%
18.32%
9.16%
2.29%
Confidential – For Discussion Purposes Only
Q12. The CCC process for trainee milestone evaluations is worth the time/effort.
Very Strongly Agree
Strong Agree
Agree
Disagree
Strongly Disagree
Very Strongly Disagree
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
6.92%
16.92%
40.00%
21.54%
6.15%
8.46%
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
“already doing it in other ways”
“CCC does not tell me more about my residents than I already know.”
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
Cumbersome and Esoteric Milestones
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
No prior research, testing for validity
40
Confidential – For Discussion Purposes Only
Q12. Why not worth the time: Emergent Themes
PDs need educational mission support.
Confidential – For Discussion Purposes Only
Other thoughts from Coordinators: Emergent themes
Approaches to CCC faculty review process
Recognizing time-consuming nature of work and need for support
“Food for thought” for ACGME from program coordinators
Confidential – For Discussion Purposes Only
Overall Results Summary
Overall Improved Evaluation and Documentation
− Semi-annual evaluations improved
− Documentation of trainee strengths and weaknesses
Time Consuming
− Need leadership support
− Administrative Resources
− Ample planning time
CCC Membership Size
− Share the burden!
− Value of diverse perspectives
Faculty Roles
− Responsibilities
Pairing faculty with trainees
Assigning faculty to become “milestone” experts
Confidential – For Discussion Purposes Only
Lessons Learned / Best Practices
It’s your turn and your voice that counts!!!
How might you change your practice in light of these findings?
Confidential – For Discussion Purposes Only
Contact Information:
Nancy Piro, Program Manager/Education Specialist
Kim Walker, Instructional Designer