Annual Meeting 2008 Creating a Better Tomorrow GSA Transition Forum: Is the MSPE Dead? AAMC Annual Meeting San Antonio, Texas November 1, 2008

Annual Meeting 2008

Creating a Better Tomorrow

GSA Transition Forum:

Is the MSPE Dead?

AAMC Annual MeetingSan Antonio, TexasNovember 1, 2008

Creating a Better Tomorrow2008 Annual Meeting

Chair

Molly Osborne, M.D., Ph.D.

Associate Dean for Student Affairs

Oregon Health & Science University

Chair-Elect, AAMC Group on Student Affairs


Speakers

Richard P. Wheeler, M.D.

Executive Associate Dean for Academic Affairs, College of Medicine

University of Arkansas for Medical Sciences

Lanny Garth Close, M.D.

Professor and Chair, Department of Otolaryngology

Columbia University College of Physicians & Surgeons


Speakers

Neil H. Parker, M.D.

Senior Associate Dean, Student Affairs & GME

University of California Los Angeles Geffen SOM

Angela Nuzzarello, M.D., MHPE

Associate Dean, Student Programs & Professional Development

Northwestern University Feinberg School of Medicine


Speakers

Maria C. Savoia, M.D.

Vice Dean for Medical Education

Professor of Medicine

University of California San Diego School of Medicine


Question # 1

Which of the choices below is your primary affiliation?

1. Student Affairs (GSA)

2. Undergraduate Medical Education (GEA)

3. Graduate Medical Education (Program Directors)

4. Student (OSR)

5. Resident (ORR)

6. Other


Question # 2

If you are involved with GSA what part are you primarily involved with?

1. MAS

2. COSR

3. COSA

4. COSFA

5. OSR

6. All


Question # 3

How long have you worked in this position?

1. < 1 year

2. 1-5 years

3. 5-10 years

4. >10 years


Question # 4

Are you the sender, recipient or subject of the MSPE?

1. Sender

2. Recipient

3. Subject

4. None of the Above


Richard P. Wheeler, M.D.

Executive Associate Dean for Academic Affairs, College of Medicine

University of Arkansas for Medical Sciences


History of the Dean’s Letter

AKA – M S P E

AKA – MISSPEEEEEEEE

AKA – “It’s not time yet to do those darn things again this year, is it?”


National Library of Medicine

4th or 5th Century AD








This was too damn hard. Have the Associate Dean do it from now on!!











Henry M. Sondheimer, M.D.November 1, 300


No rules or standardization.

Personal letters written to individual program directors.

Many times a real letter of recommendation rather than

reference.



UAMS Dean’s Letters = 1 to 2 pages


1989




November 1 release date approved through a complex

process.

Intended to slow down the process - and it worked. Few interviews before November.


ERASSeptember 1 release established by ERAS Committee but not

through complex approval process like the November 1 release date

1995



Camel


2002




TODAY


OR


Lanny Garth Close, M.D.

Professor and Chair, Department of Otolaryngology

Columbia University College of Physicians & Surgeons


THE PROGRAM DIRECTOR’S POINT OF

VIEW

The Medical Student Performance Evaluation


MSPE

2004 Survey of Oto/HNS PD’s

Rank Order of ERAS Candidate Information

• USMLE step 1 score

• Performance as sub-I

• LOR from fellow Chairs/PD’s

• AOA status

• Candidate’s Personal statement

• Dean’s Letter (MSPE)


MSPE

The PD’s dream MSPE………

“ Josephine Smith’s class rank at the completion of the third year of medical school is **/***.”


MSPE

The PD’s nightmare MSPE………..

“Although the Dean’s Letter Advisory Committee

of the AAMC recommends that the MSPE

contain comparative performance statements,

………..’s evaluation system was not designed to

provide information comparing one student to

another. For that reason, we encourage review

of this evaluation in it’s entirety.”


MSPE

Why is the non-revealing “stealth” MSPE a problem for the PD in a highly competitive specialty for which large numbers of highly qualified candidates are applying?

450 applicants

4 positions

40 interview slots

One PD


MSPE

What does the PD want from the MSPE

Earlier release

Clearly stated, uniform performance scoring

Clearly stated and illustrated performance comparisons for

• All subjects (rotations) for all 3 years

• An overall ranking of performance at the end of the third year (at least by quartiles)

• As few adjectives and adverbs as possible


MSPE

Where does the PD turn when the MSPE does not provide a meaningful comparison of performance for the candidate?

Other schools

USMLE scores

AOA status (when available)

LOR from colleagues

College transcript


MSPEExample of scored/illustrated

performance by course

Nu

mb

er

of

Stu

den

ts


MSPE

Example of scored/illustrated overall

performance at completion of third year

%

Stu

den

ts


MSPE

A Review of 2007 MSPEs from

62 U.S. Medical Schools


MSPE

SCORING CRITERIA

3 CATAGORIES

MS performance scored

MS performance compared

MS performance comparison illustrated


MSPE

MS Performance Scored

0- not done

1- poorly/incompletely done

2- average (acceptable)

3- well done

4- uniformly done (excellent)


MSPE

MS Performance Compared

0- not done



3- well done



MSPE

MS Performance Comparison Illustrated

0- not done



3- well done



MSPE

Overall Score for Institution

0- No attempt to follow AAMC guidelines

6- Acceptable presentation of MS performance

12- Outstanding (Gold Standard)


MSPE

SCORES

# o

f In

stit

uti

on

s


MSPE Study Results

20/62 (32%) U.S. Medical Schools failed to present MS Performance in acceptable format

42/62 (68%) U.S. Medical Schools present MS Performance in acceptable format

6/62 (10%) U.S. Medical Schools present MS Performance in outstanding format


Timing of MSPE

Present date is November 1st

• Applications reviewed from mid-September until mid-October

• First batch of interview invitations go out 1st week of October

• Interview slots filled by 2nd to 3rd week of October

• MSPE used by PD to help rank candidates after interviews


Timing of MSPE

Proposed date is October 1st

• Applications reviewed mid-September until mid-October

• Interview slots filled by 2nd or 3rd week of October

• MSPE used by PD to select resident applicants for interview


Student Affairs Dean Prospective on the MSPE





Student Affairs Dean Prospective on the MSPE





MSPE – One Dean’s PerspectiveAccurate assessment of student’s performance

Perspective over three plus years

Preclinical coursework minor component and not very useful

Clinical coursework major component and should be very useful

Summary provides perspective on student, potential and overall assessment

Predictive of success as a houseofficer


MSPE – Unique CharacteristicsProvides insight into what makes this student different, special, unique….

Documents the Road Travelled

Different from Student’s Activity and Background sections of ERAS as seen through the eyes of the Student Affairs Dean and Educator

Non-cognitive variables important for choosing residents - Programs three to seven years in length


MSPE – Education DemographicsImportant for years of school

Leaves of absence important but very variable in how schools represent leaves and reasons

Academic difficulty and disciplinary issues

Predictive of future problems in licensure, etc


MSPE – Years One and TwoToo short to be meaningful

All students about the same

Little information from course chairs

Can be left out without / not very different from transcript grades

Trumped by USMLE Step 1 score


MSPE – Clinical Years

Not same as clinical transcript

Critical information on student’s performance in required clerkships

Best predictor of abilities, attitudes and future potential and problems

This is the meat of the MSPE

Essentially same as evaluations of residents by faculty in programs

Needs faculty and resident development


MSPE – SummaryShort and puts residency applicant into perspective

Not specialty oriented

Not a recommendation

Would be better if was recommendation and uniform as to prediction of success as a resident


MSPE – Graphs & AppendixIn Pass Fail Schools, totally useless

What does it predict

Adds multiple pages that make MSPE seem long

Appendix E – Does anyone read in selection or only when there is a problem in residency


MSPE – One Dean’s PerspectiveWritten by Student Affairs Dean

169 Students

Follow AAMC MSPE recommendations

Use Automated System for Processing

Senior reviews for accuracy

1 ½ - 2 hours per student


MSPE – Production Time / CostAt UCLA, production of MSPE reduced by 50% through automation

Students complete web-based survey on their activities, background and other unique characteristics – 2+ hours

Individual meeting with students 20-30 minutes, SAD

Production of Unique characteristics and Summary – SAD ( Student Affair Dean – and is sad) 1 – 1 1/2 hours

Electronic formulation of clerkships information from evaluation system – HAL

Cost $300-400 per student x 169 students = $ 45,000 +


TimingStudents complete questionnaire on Activities, Background and Characteristics - August

Letters written between August and October

Students review letter for accuracy between September and mid October

Uniform release November 1

This allows for all the Third Year electives and first Sub I’s of fourth year to be received and place in the MSPE


MSPEGeffen School of Medicine

Pass/Fail all Years

Nu

mb

er

of

Stu

den

ts

0

20

40

60

80

100

120

140

160

180

PASS FAIL


MSPE - ProblemsPD’s and faculty increasingly busy with clincial & research productivity and ACGME requirements

Extremely useful but not appreciated by PD’s

Decisions on Interviews often decided before MSPE released

Undervalued

Clerkship evaluations require 6-8 weeks to collect

Senior electives evaluations difficult to collect and late


MSPE – Perspective Cont.MSPE really only 2-3 Pages

5 -6 pages of graphs, Appendices, etc

Written comments 2-3 pages

Has the MSPE gone the way of the Physical Exam and the Medical Write-up?

Should it be reduced to Consumer Report style?

Is complete, valuable information overruled by need for bottom lines and byte size information?


MSPE – Thoughts from the Weird sideConsider component release – everything short of Senior electives by October 1st

Hold ERAS process till two weeks after release of MSPE – lite

No interviews scheduled till after MSPE lite

Senior Elective evaluations supplement Dec 1st

Transcripts folded into MSPE’s


MSPE - From the Far SideDon’t change, they will come around

Go back to letter of recommendation

Clerkship evaluation development

Do away with MSPE

Just send unabridged clerkship evaluations

Develop electronic MSPE that PD’s can sort for the parts they want


Are we dealing with MillennialsNot the Students but PD’s – younger and have tenure of 3 years on average

Want everything electronic

Information must be in Byte size

Want it their way and don’t except No as an answer

Multitaskers who don’t have time to read 8 pages

Just give me the bottom line


Angela Nuzzarello, M.D., MHPE

Associate Dean, Student Programs & Professional Development

Northwestern University Feinberg School of Medicine*

*As of January 1, 2009 – Oakland University William Beaumont School of Medicine


What are we duplicating?

Unique Characteristics

• Leadership, research, community service, already in personal statement, CV, and LoRs

Academic History

• Grades already on transcript

Summary

• If summary statements are included, information has already been presented in LoRs or unique characteristics


What information is unique to the MSPE?

Explanation of any extension of time

Recipient of any adverse action

Assessment of professional performance

Clerkship narratives

Comparative performance (ranking, or quartiles, etc)

Grade percentages for each clerkship


The Problem with Creating MSPEs

Information from multiple sources required a significant amount of “cutting and pasting”

Students previewed MSPE as PDF and suggested changes through email or in person

Changes to document were required every time a new grade was finalized

Setting up comparative charts and tables was time consuming

Creating each student’s graphical record was also time consuming.


Solutions

Web-based system allows students easy access to MSPE document

Can make suggestions for changes through web interface which decreases number of emails

Any information that is stored in a database can be put in to the MSPE template

New information (i.e. grades) continually updated


Unique Features of Web-based MSPE

Email notification to students when the draft is ready; when student comments have been addressed

Forum-style system for collaboration on various sections of MSPE letter

Real-time clerkship narrative and grade inclusion

Dynamic chart and table generation

MS Word document generation

Allows tracking of progress of individual MSPEs


Medical Student Performance Evaluation for

John Smith November 1, 2008

Identifying Information [Comment] # of Comments: 0

John is a fourth-year student at Northwestern University Feinberg School of Medicine in Chicago, Illinois.

Unique Characteristics EDIT [Comment] # of Comments: 0

Over the course of his medical training, John has participated in a variety of activities and gained experiences that will add to his strengths as a physician. Although all of his pursuits have been listed elsewhere on his ERAS application, several pursuits have been particularly meaningful to his development. John has done……..

Academic History EDIT [Comment] # of Comments: 0

Date of Expected Graduation from Medical School: May 2009

Date of Initial Matriculation in Medical School: July 2005

Please explain any extensions, leave(s) of absence, gap(s), or break(s) in the student’s educational program.

Not applicable

For transfer students:

Date of Initial Matriculation in Prior Medical School: Not applicable

Date of Transfer from Prior Medical School: Not applicable

For dual/joint/combined degree students:

Date of Initial Matriculation in other degree program: Not applicable

Date of Expected graduation from other degree program: Not applicable

Type of other degree program: Not Applicable

Was this student required to repeat or otherwise remediate any coursework during his/her medical education?

No

Was this student the recipient of any adverse action(s) by the medical school or its parent institution?

No

Professional Performance [Comment] # of Comments: 0

We have assessed all students’ capabilities to meet standards for: accountability, self-improvement and adaptability, appropriate relationship with patients, relationship with healthcare team, initiative, professional demeanor, and behavior under stress. Unless otherwise noted, all students have met the stated objectives for professionalism at Feinberg.

Academic Progress EDIT [Comment] # of Comments: 0

John earned grades of Pass in all preclerkship courses and recorded a score of 239 on Step I of the USMLE exam.

Preclinical/Basic Science Curriculum [Comment] # of Comments: 0

(Details are provided in Appendix E.)

Core Clinical Clerkships EDIT [Comment] # of Comments: 0

Following are the unedited narrative evaluations of John’s performance on the successive clerkships and electives. Appendix B summarizes this information graphically. NEUROLOGY: PASS 7/2/2007 - 7/27/2007

John did a very good job during his neurology rotation. This was his first clinical clerkship. He demonstrated good history and examination skills. His histories were thorough and he was able to pay attention to detail. He also demonstrated appropriate techniques in the neurological examination. His oral presentations were well organized and clear. He appeared to have a good fund of knowledge based on his clinical interactions and discussion; he scored below the mean on his written shelf examination. He was motivated and eager to learn. He was hard working, organized and responsible in his work. He had great interpersonal skills and interacted very well with his team members and patients. For his overall performance on this rotation, he is awarded the grade of PASS.

PRIMARY CARE: HONORS 7/30/2007 - 8/24/2007 John is a bright medical student who did an outstanding job in all aspects of the Primary Care Clerkship. He has excellent interactions with patients and staff alike. His history and physical exam taking skills are on par with a 4th year medical student. His oral case presentations are concise and well organized. He has a solid knowledge base and very good clinical judgment. In Weekly Report he was an active participant with frequent insightful comments. His Learner Centered Learning Goal presentation on depression screening was outstanding. The preceptor wrote, "Not

APPENDICES

Appendix A: Comparative Performance in Preclinical/Basic Science Coursework [Comment] # of Comments: 0

During the first and second years of the curriculum, the only grades assigned to students are Pass or Fail. Students must receive a grade of “Pass”, in each preclinical course before beginning the clerkships.

Appendix B: Graphic Representations of Comparative Performance in Core Clinical Clerkships [Comment] # of Comments: 0

Grades in Required M3 Clerkships/Courses

John’s grade for each clerkship is indicated inside the bracket beside the clerkship name.

Table 1- Grade Percentages for Each Required M3 Clerkships/Courses

IDM3 MED NEURO OBGYN PEDIA PRIMCARE PSYCH SURG

H 25 36 34 22 23 31 24 31

HP 16 27 30 34 29 26 40 18

P 59 37 36 43 46 43 36 48

F 0 0 0 1 2 0 1 2


Comments from students

“I think it worked well. The ability to send e-comments was very helpful and response time was quick.”

“I think it was great that we could view our MSPEs and any comments I had were immediately addressed.”

“I feel that the bi-directional feedback and electronic format made this quite efficient.”

“I liked that everything was web based.”


Maria C. Savoia, M.D.

Vice Dean for Medical Education

Professor of Medicine

University of California San Diego School of Medicine


““BOTH SIDES NOWBOTH SIDES NOW””

A VIEW ABOVE THE FRAYA VIEW ABOVE THE FRAY&&

A REVIEW OF THE RELEVANT A REVIEW OF THE RELEVANT LITERATURELITERATURE


Do MSPEs Accurately Reflect Do MSPEs Accurately Reflect Medical School Performance?Medical School Performance?

Are they based on accurate data?

Are data accurately reflected in letters?


Based On Accurate Data?Based On Accurate Data?

Evaluation often done by the least experienced

› Experience important in preceptor’s ability to assess cognitive levels Taylor C., Lipsky MS Fam Med 1990 Jul-Aug;

(2294)296-8



For data to be reliable, either many observations or many observers

For data to be valid, they have to measure what they say they measure

› “Knowledge and skill” in a Psychiatry clerkship correlated with personality characteristics and not with NBME Subject exam or OSCE scores Chibnall JT, BlaskiewiczRJ. Acad Psychiatry 2008

May-Jun;32(3); 199-205



“Objective” data often maligned

Although many recent studies have not demonstrated gender or ethnic biases… it is better to be good looking and thin & to work closely with the person who evaluates you

› Supervisors grade higher than control graders McKinstry BH, et. al. BMC Medical Education 2004,

4;28


Based On Accurate Data?Based On Accurate Data?The great group of “Very Good”

› In Pass/Fail curriculum, how do you distinguish among those who excel &those who scrape by?

› P/F grading puts students at a disadvantage in applying to General Surgery programs Dietrick JA, et. al., Am J Surg 1991 Jul;162(1);63-6

› In a survey of 110 program directors across specialties, only 8 favored a P/F system Provan JL, Cuttress L, Can Med Assoc J Oct.1, 1995;

153 (7)


1992 55%

2001 65%

2005 75%

“Adequate MSPEs”

Are Data Accurately Reflected?Are Data Accurately Reflected?

Hunt DD, et. al., Acad Med 2001 Jul;76(7): 727-33

Shea JA, et. al., Acad Med 2008 Mar; 83(3):284-91



We live in a “fantasy land” where all students “will make a fine physician,” & are “above average” “with excellent interpersonal skills.”

Dean’s Letter for Adolph Hitler: “A natural leader…good communication skills…assisted in the development of technological advances…likes to find solutions to problems”

Friedman RB, NEJM 1983 308;651-3



Negative information found on transcript not mentioned in Dean’s Letter 34% of time.

› Conclusion: “Some deans suppress negative information in their letters and potentially obfuscate the residency selection process.” Edmond M, et. al. 1999 Acad Med 74(9);1033-5.



Excellent: of superior merit, exceptionally good

56/75 schools used “Excellent” as a category.

For >50% of schools, a student described as excellent might be in bottom half of class. For 2 schools, all in “Excellent” category were in the bottom half of the class

Naidich JB, et. al., Acad Rad 2007 14; 1121-6.



Professionalism comments evaluated in 293 MSPEs for 2005 graduates.

70% mentioned professionalism, with fewer than 1% of comments that could be considered negative.

› Conclusion: “Most professionalism comments in MSPE’s are generic and somewhat bland…” Shea JA, et. al., Acad Med 2008 Oct. 83(10

Suppl);S1-4


What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?

Univ. Rochester looked at 2 graduating classes, divided into:• “Outstanding” (upper quartile),• “Excellent” (second quartile),• “Very Good” (lower 2 quartiles)• “Good” (lowest few percentiles)

Found “Dean’s Letter rankings were a significant predictor of later performance.”

Lurie SJ, et. al., Teach Learn Med 2007 19(3);251-6



At the Uniform Services University, appearing before Standings & Promotions Committee (for any reason) was predictor of below average performance during internship.

Durning SJ, et. al., Teach Learn Med 2008 20(3);267-72



At the Medical University of South Carolina, GPA (preclinical + clinical) + interpersonal score on Step 2CS best predicted internship performance but only accounted for 30% of variance.

Taylor ML, et. al., Acad Med 2005 80(5); 496-501



Otolaryngology: “excellent academic performance in medical school”

Calhoun KH, et. al., Otolaryngol Head Neck Surg 1997 116(6 pt1); 647-51

Emergency Medicine: medical school attended, MSPE and “distinctive factors”

Hayden SR, et. al., Acad Emerg Med 2005 12(3); 206-10



Radiology: “Noncognitive factors were as important as cognitive factors,” “Objective measures, such as NBME scores, failed to adequately predict residents’ performance.”

Wood PS, et. al., Invest Radiol 1990 25(7);855-9

Radiology: Success on ABR exam predicted by grades and NBME scores. No good predictors of clinical performance.

Boyse TD, et. al., Acad Radiol. 2002 9(4);437-45



PM&R: 205 residents at the University of Washington. Clinical residency performance predicted by clerkship honors grades (p=.0001). Probation was predicted by failing a basic science course (p=.0001)

Amos DE, Massagli TL, Acad Med 1996 71(6);678



Orthopedics: “Academic score,” using only objective measures, made screening process more objective but didn’t correlate with outcomes of the training program.

Dirschl DR, et. al., Clin Orthop Relat Res 2006 449;44-9

Surgery: “Dean’s letter seems to give the most accurate picture of applicant’s suitability for residency training.”

Naylor RA, et. al., Arch Surg 2008 143(7); 647-52


PD’s: Ignore at your own risk!With an edge of annoyance, Hunt asked (Barrows) about Swango. “What kind of guy did you send us?”

Barrows said that Hunt should have seen plenty of warning flags in Swango’s dean’s letter. “Well,” Hunt retorted, “I don’t read dean’s letters.”

….Soon after, Hunt called back: he’d found the letter

“Oh, my God,” Hunt said. “You’re right. You did tell me.”

Stewart, JB, Blind Eye , 1999, pps. 59-60



Surgery: Only Step 2 scores (and not Step 1, overall GPA, AOA status, or 3rd year clerkship grade) predicted internship performance for 87 graduates from Wash U.

Andriole DA, et. al., Am J Surg 2004 188;161-4

Anesthesia: Best predictor of academic performance number of years spent in another specialty. Step 1 score inversely correlated with in service exam score

Warrick SS, Crumrine RS, J Med Ed 1986 61(7); 591-5


What Factors Accurately Predict What Factors Accurately Predict Residency Performance?Residency Performance?Anesthesia: “Selection committee score,” which correlates with Step 1 score, doesn’t correlate with resident performance.

Metro DG, et. al., Anesth Analg 2005 100;502-5

Pediatrics: In the case of 69 residents at UVA, medical school grades, performance on standardized exams, interviews and match list rankings did not predict clinical performance during residency.

Borowitz SM, et. al., Arch Pediaatr Adolesc Med 2000 154;256-60


In Conclusion:In Conclusion:

We need better measures of performance during medical school.

Multiple types of evaluation may provide a more accurate picture of performance.

It’s about the Match. The best program may not be the best program for a particular student.

Honesty is the only policy.



More categories of grading may help students in the Match

The MSPE is as accurate a measure as any in predicting residency performance.

NBME Step 2 but not NBME Step 1 scores have occasionally been found to predict clinical performance




Question # 5

Do you favor continuing the MSPE in some form?

1. Yes

2. No

3. Unsure


Question # 6

If we keep the MSPE, do you favor changing the current format of the MSPE?

1. Yes

2. No


Question # 7

Would you remove the unique characteristics section of the MSPE?

1. Yes

2. No


Question # 8

Would you remove the clerkship narratives from the MSPE?

1. Yes

2. No


Question # 9

Do you favor changing the MSPE release date?

1. Yes

2. No


Question # 10

If you favor changing the MSPE release date, which would be the optimal date?

1. September 1st

2. September 15th

3. October 1st

4. October 15th

5. November 1st

6. January 11th


Question # 11

Would you favor a single date for MSPE release and the start of ERAS upload?

1. Yes

2. No


Question # 12

Would you favor a uniform release date of

Oct 1st for transcripts?

1. Yes

2. No


Question # 13

Do you think the GSA steering committee should propose a change in the MSPE, develop a survey on the MSPE, or should we a new advisory group be convened?

1. Propose an MSPE change

2. Develop a survey

3. Convene a new advisory group


Thank you for coming!!