Curriculum Change & Student Learning Outcomes: InternshipCurriculum Change & Student Learning Outcomes: Internship

Prof Tony Chung, Associate Dean (Clinical)

Faculty of Medicine, Chinese University of Hong Kong

Prof Tony Chung, Associate Dean (Clinical)

Faculty of Medicine, Chinese University of Hong Kong

Research Objective

Impact of curriculum changes in terms

of student learning outcomes

Measured Outcomes

Curriculum changes assessed by:

1. Summative assessments during clinical years

2. Surveys of final year students re: instructional quality, curriculum coverage & clinical skill development

3. Interns’ self-assessment of their generic skills

4. Internship performances as rated by clinical supervisors

Historical Controls

Compared Interns from new curriculum (2006) with

Interns from 1995 – 2005 (all interns from old curriculum)

Interns from 2002 – 2005 (senior classmates trained under old curriculum)

Measured Outcomes

Preceptor Ratings of 11 Intern Performance Criteria– Professional Knowledge– Clinical Skill– Clinical Judgment– Use of Medical Language

– Attitude to Patients – Attitude to Medical Staff – Willingness to Learn – Organizational Ability – Attendance at Educational Activities – Communication Skill – Sense of Responsibility

attributes given increased emphasis in new curriculum

Additional Measures

Additional assessments from Internship

- Content analyses of clinical supervisors’ comments

compared 2003/04 - 2005/06 (old) & 2005/06 (new)

- Frequency of medical incident reports related to interns

compared 2005/06 (old) & 2006/07 (new)

Statistical Adjustments

Covariates

Type of hospital (Teaching, Teaching affiliated)

Unit (Med, Surg, Paed, Psy, OBG, Ortho)

Gender (Female / Male)

Rotation (1st to 4th)

Initial graduating status (Pass/ Fail)

Inflation in supervisor ratings over 11 yrs

(all significant polynomial terms for trends)

Results

Results

In terms of room for performance improvement in reference to all interns educated & trained in the old curriculum (1995 to 2001)

improvements by new interns ranged from 2.3% to 39.5% across the 11 performance attributes

If controls restricted to last 4 cohorts trained in the old curriculum

improvements by new interns ranged from 2.7% to 51.8% across the 11 performance attributes

Three attributes statistically significant after covariate adjustment (function of statistical power/sample size)

Curriculum Changes & Performances % Gained within Region of Performance

Attribute Effect p that Could be Improved Given Previous Interns

(1995 - 2005) (2001 - 2004)

Attitude to medical staff + 0.316 0.007 39.5% 51.8%

Organization ability + 0.259 0.018 20.2% 23.3%

Communication skill + 0.230 0.037 20.1% 23.7%

Use of medical language + 0.157 0.103 13.1% 15.1%

Willingness to learn + 0.112 0.358 12.5% 16.2%

Clinical judgment + 0.151 0.134 12.1% 14.0%

Attendance at educ. act. + 0.132 0.218 11.1% 13.4%

Attitude to patients + 0.091 0.421 10.4% 13.2%

Sense of responsibility + 0.076 0.532 9.1% 11.2%

Professional knowledge + 0.030 0.751 2.3% 2.7%

Clinical skill + 0.029 0.772 2.3% 2.7%

Covariates: gender, hospital, unit, P/F status, rotation & polynomial terms for trends

Content Analysis of Supervisors’ Comments

~ 2/3’s of the supervisors include written comments

compared cohorts from old curriculum, comments

regarding the 2005/06 interns (new curriculum) were

– more detailed (+10.5%)

– more complimentary & more interns described as good/better

Rotation Curriculum None Poor Average Good

1st New 26.8% 1.4% 4.2% 67.6% * Old 33.3% 1.4% 16.0% 49.3%

2nd New 33.1% 0.7% 9.2% 57.0% * Old 35.4% 2.1% 27.8% 34.7%

3rd New 37.3% 1.4% 7.0% 54.2% *

Old 35.4% 3.5% 18.1% 43.1%

4th New 42.3% 2.8% 10.6% 44.4% *

Old 45.1% 6.9% 23.6% 24.3%

NNew = 142 *

c21st rotation = 14.86 (p=0.002); c2

2nd rotation = 22.23 (p<0.001)

NOld = 144 c23rd rotation = 10.04 (p=0.018); c2

4th rotation = 18.12 (p<0.001)

Nature of Clinical Supervisors’ Comments Concerning CUHK Interns

2006/07 (New) : 2005/06 (Old)

Proportion of Interns Whose Performance was described as Good

0

10

20

30

40

50

60

70

80

90

100

1st Rot. 2nd Rot 3rd Rot 4th Rot

2003/04

2004/05

2005/06

New Curric

2003/04 – 2005/06 (Old) vs 2006/07 (New)

%

No trend of increasingly positive comments across rotation & years. There is a statistically significant trend during 4 rotations of more new interns being described as good compared to any of the last three cohorts from the old curriculum.

Among Clinical Supervisors’ Comments: Proportion Indicative of good Intern performance

0

10

20

30

40

50

60

70

80

90

100

1st Rot. 2nd Rot 3rd Rot 4th Rot

Old Curric

New Curric

2005/06 vs 2006/07

Example Comments for ‘Good’

• “Strong sense of responsibility; professional knowledge, skills and clinical judgment are excellent for her seniority; humble, quick and eager to learn. Overall excellent performance”

• “Excellent attitude to patients; appreciation letters received from patients she cared for”

• “Demonstrated impressive clinical acuity in psychiatry, astute and logical clinician, sensitive intuition, appropriate empathy and scientific objectivity; identified relevant research questions in his daily clinical encounters”

• “Did a fantastic job at her first rotation, keen to learn and willing to take responsibility, no reservation in recommending her to extend her future training in oncology and in medicine”

• “Particularly appreciated the clinical records and documentation written by Dr. xxx, which were very neat, well organized and comprehensive”

• “One of the very best interns we ever had. He has sound general medical knowledge and can practically apply this to patients; hard working, reliable & totally committed to his work; well liked by his colleagues. He is someone with great potential!!”

Example Comments for Good (cont’d)

3rd Outcome from Supervisors:

Medical Incident Reports

11 hospitals, 58 specialty units & 4 rotations in 2003-6

(old curriculum)– Average 20 interns had at least one medical incident report– Average 26 medical incident reports were filed

In 2006/07 (new curriculum)– 10 interns had at least one medical incident report (50% reduction)– 14 medical incidents were filed (46 % reduction)

Frequency of medical incident reports significantly less

Discussion

Interns’ performance as a measure

Interns assessed since 1995– many attributes matched additional learning goals emphasized in new

curriculum

– comparisons during medical school limited to assessments used in old & new curriculum

– usually did not include the generic & attitudinal measures

Third party assessment (purchaser)– less potential for bias

Assessments related to professional work as a front-line doctor – not restricted to within-school clinical tasks

Limitations

• Historical controls

• Blinding not uniform across clinical supervisors

• Sample size of intervention group small (i.e., one year of interns)

• 1st cohort of a new curriculum often does better than subsequent cohorts

Conclusion

• Little evidence to support knowledge of the new cohort has significantly changed as seen in exam results comparing cohorts in years 4 & 5

• Preliminary evidence supports new graduates have some enhanced generic skills and professionalism (a goal of the new curriculum)

Conclusion

Importance of Establishing Evidence

Ongoing monitoring and evaluation

Draw firmer conclusions when more new graduates assessed