Implementing an Implementing an Effective Academic Effective Academic

Remediation Program: Remediation Program: Results and Results and

ConsequencesConsequences

Karen SharrockKaren SharrockYork Hospital, York, PAYork Hospital, York, PA

Illustration by David Harbaugh1

He never recovered from the most recent site visit

WebsterWebster’’s Dictionary:s Dictionary:EffectiveEffective: producing a decided, decisive, or : producing a decided, decisive, or desired outcomedesired outcomeAcademicAcademic: based on formal study especially at an : based on formal study especially at an institution of higher learninginstitution of higher learningRemediationRemediation: concerned with the correction of : concerned with the correction of faulty study habits and the raising of a pupil's faulty study habits and the raising of a pupil's general competence.general competence.

Background: Standards to Demonstrate Background: Standards to Demonstrate Medical Knowledge CompetencyMedical Knowledge Competency

York Hospital Surgery Residency Program considers poor York Hospital Surgery Residency Program considers poor performance on the ABSITE examination to be an performance on the ABSITE examination to be an indication that a resident has not acquired an adequate indication that a resident has not acquired an adequate knowledge baseknowledge base

A minimum total test score of 67% (on the ABSite) is A minimum total test score of 67% (on the ABSite) is required by each resident to remain in good academic required by each resident to remain in good academic standing. Those residents who score less than standing. Those residents who score less than 67%67%correct as the total test (TT) score will be placed on correct as the total test (TT) score will be placed on academic remediation, which includes a concentrated academic remediation, which includes a concentrated course of study devised by the program director.course of study devised by the program director.

Inclusion CriteriaInclusion Criteria

The total test score value of 67% was chosen The total test score value of 67% was chosen becausebecause

Previous compilation of remediation limits in the Previous compilation of remediation limits in the literatureliterature22

Reports of an increased failure rate on following Reports of an increased failure rate on following ABSite exams for those scoring less than the 30ABSite exams for those scoring less than the 30thth

percentilepercentile3.3.

New RRC requirements New RRC requirements

20052005--2006:2006:All residents: Basic Science Curriculum, as well as other All residents: Basic Science Curriculum, as well as other

interdisciplinary conferences and mock oral conferences.interdisciplinary conferences and mock oral conferences.No remediation plan No remediation plan

20062006--20072007As a result of Jan 2006 ABSite results, a remediation plan was As a result of Jan 2006 ABSite results, a remediation plan was

developed in February 2006 and implemented for 4 residents developed in February 2006 and implemented for 4 residents out of 15 categoricalout of 15 categorical

20072007--20082008The success of the previous plan year indicated that we should The success of the previous plan year indicated that we should

modify the basic science curriculum for all residents to modify the basic science curriculum for all residents to incorporate remediation plan componentsincorporate remediation plan components

Residents (4 of 15) who did not meet the minimum cutoff Residents (4 of 15) who did not meet the minimum cutoff score were assigned to the full remediation program. score were assigned to the full remediation program.

Basic Science CurriculumBasic Science CurriculumCY 2005 (ABSite 2006)CY 2005 (ABSite 2006)

•• This conference covers the material present in a This conference covers the material present in a programprogram--chosen basic science textbookchosen basic science textbook

•• WeeklyWeekly

•• 5 question quiz based on the reading administered to 5 question quiz based on the reading administered to assure that the residents have reviewed the material. assure that the residents have reviewed the material.

•• 100% attendance expected.100% attendance expected.

Results 2006Results 2006

Departmental mean = 60.80, departmental median = 63

Remediation CurriculumRemediation Curriculum

Those residents on remediation will: Those residents on remediation will: Complete Basic Science CurriculumComplete Basic Science Curriculum

In addition, supplemental reading will be required In addition, supplemental reading will be required from three additional textsfrom three additional texts

SESAP 13 must be purchased by the residentSESAP 13 must be purchased by the resident

Supplemental monthly exams.Supplemental monthly exams.

Remediation Requirements:Remediation Requirements:

Must receive a Must receive a ““passpass”” on all rotationson all rotations

In House Tests/Quizzes average >80%In House Tests/Quizzes average >80%

Next YearNext Year’’s ABSite score must be above 67% s ABSite score must be above 67% total test score.total test score.

Remediation Requirements (contRemediation Requirements (cont’’d)d)During this period of academic remediation, the During this period of academic remediation, the resident will be offered the following at no cost:resident will be offered the following at no cost:

General medical examinationGeneral medical examinationPsychiatric evaluationPsychiatric evaluationPsychological testingPsychological testingTestTest--taking skills (learning style) assessment taking skills (learning style) assessment and coaching.and coaching.

Consequences of Remediation:Consequences of Remediation:

Lost PrivilegesLost Privileges

Vacation Scheduling Priority deferredVacation Scheduling Priority deferred

Basic Science CurriculumBasic Science CurriculumCY 2006CY 2006

•• This conference covers the material present in a This conference covers the material present in a programprogram--chosen basic science textbook chosen basic science textbook

•• WeeklyWeekly

•• 5 question quiz based on the reading administered to 5 question quiz based on the reading administered to assure that the residents have reviewed the material assure that the residents have reviewed the material

•• 100% attendance expected.100% attendance expected.

Results: Results: CY 05 (ABSite 06) vs. CY 06 (ABSite 07):CY 05 (ABSite 06) vs. CY 06 (ABSite 07):

40

45

50

55

60

65

70

Res A Res B Res C Res D

ABSite 2006ABSite 2007

Residents on remediation in 2006Residents on remediation in 2006

Basic Science CurriculumBasic Science CurriculumCalendar Year 2007Calendar Year 2007

Basic Science Curriculum was assigned to all Basic Science Curriculum was assigned to all residents in the programresidents in the program

Residents on remediation: monthly 20 question Residents on remediation: monthly 20 question examexam

Addition: biAddition: bi--monthly 20 question exam given to monthly 20 question exam given to all residents.all residents.

404550

5560

6570

758085

Res E Res B Res F Res G

ABSite 2007ABSite 2008

Results: Results: CY 06 (ABSite 07) vs. CY 07 (ABSite 08):CY 06 (ABSite 07) vs. CY 07 (ABSite 08):

Residents on remediation in 2007Residents on remediation in 2007

If all requirements are met If all requirements are met ANDAND the resident the resident scores at or above a scores at or above a 67% TT score67% TT score on the next on the next

ABSITE, then s/he will be removed from ABSITE, then s/he will be removed from academic remediation.academic remediation.

HoweverHowever……

Consequences of NonConsequences of Non--compliance:compliance:

Probation or DismissalProbation or Dismissal

Although the ABSITE examination will not be used Although the ABSITE examination will not be used as the sole measure of promotion, substandard as the sole measure of promotion, substandard scores, scores, despite attempts at remediationdespite attempts at remediation, combined , combined with poor performance in other areas will lead to with poor performance in other areas will lead to academic probation and may be grounds for academic probation and may be grounds for dismissal from the program dismissal from the program

(YH Residency Handbook).(YH Residency Handbook).

From the ACGME:From the ACGME:Surgery RRC, Program Requirements,

Section V.1.A.1.e

Assessment should specifically monitor the resident's knowledge by use of a formal exam such as the American Board of Surgery In Training Examination (ABSITE) or other cognitive exams.

(ABSite) Test results should not be the sole criterion of resident knowledge, and

should not be used as the sole criterion for promotion to a subsequent PG level.

Have the resident repeat a year?Have the resident repeat a year?

DIO approval DIO approval -- fundingfunding

Needs RRC approval Needs RRC approval

ABS ABS ““encourages encourages retention of residents in a program as long as is required to achieve the necessary level of qualifications (ABS booklet, 2008).

Program ComparisonProgram Comparison

2008 2007 2006median 64.50 64.00 63.00average 66.08 64.33 60.80

Comparison by PGYComparison by PGY

n/an/a

n/an/a

n/an/a

52.7552.75

58.558.5

20052005

n/an/a

n/an/a

55.355.3

64.2564.25

59.559.5

20062006New BS New BS

curriculumcurriculum

7676n/an/a11

6767676722

60.3360.3365.365.333

707060.560.544

666665.565.555

2008200820072007New New

remediationremediation

PGY PGY (as of 2008)(as of 2008)

Overall FindingsOverall Findings

All residents on remediation demonstrated All residents on remediation demonstrated improvement, but some still did not meet the improvement, but some still did not meet the minimum score of 67%minimum score of 67%Previous literature showing documented success Previous literature showing documented success in other programsin other programsd/td/t improved individual weekly reading effort, improved individual weekly reading effort, curriculum change, fear of consequences?curriculum change, fear of consequences?Newly discovered Newly discovered -- PGY3 dip.PGY3 dip.

Results 2006Results 2006

Departmental mean = 60.80, departmental median = 63

Results 2007Results 2007

50

55

60

65

70

75

PGY5 PGY4 PGY3 PGY2 PGY1

AVG

Median

Departmental mean = 64.33, departmental median = 64

Results 2008Results 2008

Departmental mean = 66.08, departmental median = 64.5Departmental mean = 66.08, departmental median = 64.5

50

55

60

65

70

75

80

PGY5 PGY4 PGY3 PGY2 PGY1

AVG

Median

Goals:Goals:

Initial purpose of ABSiteInitial purpose of ABSiteMeasure knowledgeMeasure knowledgeIdentify deficiencies and potential weaknessesIdentify deficiencies and potential weaknesses

Ultimate effectiveness of remediation should be Ultimate effectiveness of remediation should be determined by pass rate on QE.determined by pass rate on QE.

Review of ABS QE Pass RateReview of ABS QE Pass RateFirst attempt pass rateFirst attempt pass rate

67%

100% 100%

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

2005 2006 2007

Correlation of Competencies:Correlation of Competencies:

Deficiency in medical knowledge affects other Deficiency in medical knowledge affects other competencies as well:competencies as well:

Patient carePatient careICSICSProfessionalismProfessionalismSystems Based PracticeSystems Based PracticeEvidence Based MedicineEvidence Based Medicine

Going forwardGoing forward……

More stringent reliable/valid tests (online surgical More stringent reliable/valid tests (online surgical database of texts/questions)database of texts/questions)

Continue to monitor 1Continue to monitor 1stst time pass rate, ABS QEtime pass rate, ABS QE

Perhaps development of transitional curriculum (PGY2 Perhaps development of transitional curriculum (PGY2 to PGY3)to PGY3)

Continue remediation program.Continue remediation program.

ReferencesReferences

HarbaughHarbaugh, D. , D. Residency Program Alert . Residency Program Alert . HC Pro HC Pro 2007. HC Pro. 30 March 2008 2007. HC Pro. 30 March 2008 http://www.hcpro.com/content/204657.pdfhttp://www.hcpro.com/content/204657.pdf

HarthunHarthun NL. NL. SchirmerSchirmer BD, BD, SanfeySanfey H. Remediation H. Remediation of low ABSITE scores. of low ABSITE scores. CurrSurgCurrSurg. 2005;62:539. 2005;62:539--542542

1.1. American Board of Surgery. American Board of Surgery. Interpretation of Results of Interpretation of Results of the American Board of Surgery Inthe American Board of Surgery In--Training/Basic Science Training/Basic Science Examination.Examination. Philadelphia, PA: American Board of Philadelphia, PA: American Board of Surgery.; 2005Surgery.; 2005

Questions?Questions?

HUMOR FOR THE DAY HUMOR FOR THE DAY ……