Embed Size (px)
DESCRIPTION
Evaluation of Surgical Skill Skills that are uniquely surgical Evidenced in OR Surgical patient management Observed during training events Clinical (e.g. OR) Laboratory Occurs throughout residency Progression of skills
Citation preview
EVALUATING SURGICAL SKILL AND OR PERFORMANCE
Neal E. Seymour, MDBaystate Medical Center
Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab
2010 APDS Annual Meeting, San Antonio, Texas
DisclosuresNo disclosures
Evaluation of Surgical Skill Skills that are uniquely surgical
Evidenced in ORSurgical patient management
Observed during training eventsClinical (e.g. OR)Laboratory
Occurs throughout residencyProgression of skills
Improved Task Performance
0
50
100
150
200
1 3 5 7 9 11 13 15 17
Tota
l Sco
re
Trial Number
VR Laparoscopic Suturing and Knot-Tying
Residents
Experts
Effect of VR Training on Laparoscopic Suturing Performance: Video Analysis
0
10
20
30
40
50
60
70
Pre-VR Post-VR
Video Score
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
Effect of VR Training on Laparoscopic Suturing Performance: Time
0
50
100
150
200
250
300
350
400
Pre-VR Post-VR
Time (sec)
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
Proficiency-Based Lab Training
p > 0.006*
Skills Transfer: “VR to OR” Study
0
1
2
3
4
5
6
7
8
9
10M
ean
num
ber o
f erro
rs
VR Trained Standard Training
*
Assessment of Performance in OR
Procedure specific instruments GOALS OSATS Analysis of video recorded tasks Differences between investigative
instruments and assessment in frequent, routine use.Limitations on obtrusive methodsResource intensive
Design of New Assessment Tool
User-friendly in routine use, valid, reliable.Opportunity for delivery of feedbackLeverage technology to manage large
volumes of data generated.Generate useful reports on resident
performance.
Resident OR Performance Assessment at Baystate Medical Center
Objective: Develop OR performance assessment tool for routine use
OpRate: Dept. of Surgery and IS project:Multi-item global assessment of skillAttending surgeon raterEase of use = compliance with use.Interfaced to other databases
Evaluated Items
3 preop (preparedness) questionsPatient, technical and disease-specific knowledge
6 open technical skills questionsTissue, instrument handling, knot tying, efficient
planning and ability to function independently 6 laparoscopic technical skills questions 4 point frequency-based responses for
specific technical performance behaviors
Assessment of Residents with OpRateJuly 2004 - June 2006
15 29 21 36 3863
146101
237
113
050
100150200250
Appen
decto
my
Ventra
l Herni
a
Colon R
esec
tion
Cholecy
stecto
my
Inguin
al Hern
ia
Case
Num
ber
2004-5 2005-6
799 OpRate evaluations logged
579 cases reviewed 29 residents
33 attending surgeon evaluators◦ 6 contributed 75% of
assessments
Preop Preparedness
0
20
40
60
80
100
120
140
160
PGY1 PGY2 PGY3 PGY4 PGY5
Training Year
Num
ber o
f Cas
es
3.5
3.6
3.7
3.8
3.9
4
Perfo
rman
ce S
core
Cases
Performance
Cronbach’s α (3 test items) = 0.80
Laparoscopic Appendectomy
Cronbach’s α (4 test items) = 0.80
0
5
10
15
20
25
30
35
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4
Perfo
rman
ce S
core
Cases
Performance
Laparoscopic Cholecystectomy
Cronbach’s α (4 test items) = 0.80
05
101520253035404550
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
22.22.42.62.833.23.43.63.84
Perfo
rman
ce S
core
Cases
Performance
Colon Resection
Laparoscopic (n = 37): Cronbach’s α (4 test items) = 0.82 Open (n = 36): Cronbach’s α (6 test items) = 0.84
0
5
10
15
20
25
30
35
40
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f cas
es
2.5
2.7
2.9
3.1
3.3
3.5
3.7
3.9
Perfo
rman
ce S
core
CasesPerformance
VR laparoscopic gall bladder excision (time) vs laparoscopic OR performancep = 0.0066, r2 = 0.4450 (Spearman correlation)
J Surg Res 2009;154:177-178
Conclusions Experience with OpRate confirms predictions of
experience-dependent resident performance increases, but does not validate attending rating behaviors.
Still Required:Systematic rater trainingInter-observer agreementConcurrent validity studies (lab and OR)Mandatory compliance
OpRate Build 2 CQL Server database Edit redundant items Add important items
Bimanual skillsAbility to assistDepth perception in laparoscopic cases
Status of Use Remediation decisions 2 SD below mean for peer group
High likelihood of performance problems Compliance to be augmented
Variable compensation linked to useCompliance monitored and posted monthlyExpectation that residents will advocate for
use
Patient Management and Teamwork Skills
Electrical Burns VTE Septic Shock Hemorrhagic Shock Spinal shock Post-Op MI Anaphylaxis
FPS Curriculum Thoracic trauma Hyperkalemia Intraabdominal
catastrophe Pediatric trauma Pheochromocytoma
Recognition of importance of teamwork orientation to effective clinical careStrong foundation principlesAlignment with growing body of work on “non-
technical skills” Effort to take advantage of simulated patient
care, simulated hospital environments (e.g. OR)Limited precedent in surgery
ACS-APDS Phase III Curriculum
Unique communications rating instrument
Checklisted trauma assessment SPIKES protocol (delivering bad news) NOTECHS Ottawa Global Scale Need to strengthen and consolidate
assessment
Assessment
Conclusions Assessment of surgical skills and OR
performance:Opportunities
Expanded uses:Identification of performance outliers.Definition of areas where focused training is required
for specific residents.Measurement of effects of educational interventions.
Measurement of curriculum effectiveness
