Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Predicting Student Professionalism in Medical School: A Situational
Judgment TestAdmissions and Selection Program
November 10, 2019
Dana M. Dunleavy, PhD, DirectorRebecca L. Fraser, PhD, Manager
7 years of rigorous research
2012Comp.Model
2014Critical
Incident Study
2015SJT
Prototype
2016Field Test
2017Validity Study
Launch
2019Field Test
The AAMC SJT is supported by comprehensive research.• Reliable and valid• Lower group differences than
academic tests• Correlates with medical school
performance
The AAMC SJT could add value.• Easy to use for examinees and
admissions officers• Offer a balance to existing academic
metrics during pre-screening to facilitate holistic review
• Help identify applicants who may be at risk of professionalism issues or have strong professionalism skills
• Help broaden the diversity of applicants considered for interviews
2012Comp.Model
2014Critical
Incident Study
2015SJT
Prototype
2016Field Test
2017Validity Study
Launch
2019Field Test
2020 SJT Operational
Use (Limited)
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
Applicants need more than academics to be ready for medical school• Medical school curricula include instruction and training about
academics, interpersonal skills, clinical care, and professionalism
• Medical school and practice “derailers” are often related to a lack of professionalism
• It’s harder to remediate students with professionalism issues than academic issues
Medical schools want to identify students who are prepared to:• Work in a diverse learning environment• Serve the health needs of underserved populations• Serve the health needs of local communities• Serve as next generation of medical leaders• Demonstrate pre-professional competencies
Admissions officers believe the AAMC SJT can add value to pre-screening
Hardest thing to remediate is professionalism
SJT scores could be used to:• Expand the applicant pool in pre-screening• Evaluate applicants in the “gray zone” • Help sift through high application volumes• Compare with in-person interview evaluations
Desire for evidence:• Reliability, validity• Coaching effects• Outcomes• No disadvantage to under-
represented applicants
Fall 2017, 3 focus groups with admissions officers representing 21 schools
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT was developed in partnership with medical schools
Competency Modeling
Critical Incident Writing
Item Writing
Bias & Sensitivity Review
Healthcare Experience Review
Scoring Key Development
• Item Writing Guidelines
• Subject Matter Experts
• Inclusion criteria
• Subject Matter Experts
• Inclusion criteria
• Subject Matter Experts
• Inclusion criteria
• Subject Matter Experts
• Subject Matter Experts
Items must pass subject matter expert review at every step to be included on the test.
AAMC SJT is designed to measure
Pre-Professionalism
Service Orientation
Social Skills
Cultural Competence
Teamwork
Ethical Responsibility
to Self and Others
Reliability & Dependability
Resilience & Adaptability
Capacity forImprovement
Examinees’ understanding of effective
pre-professional behaviors across eight
core competencies
Sample SJT Scenario and ItemsWhile viewing one of your classmate’s social media profile, you notice that your classmate has made negative comments about treating a recent patient. Your classmate describes the patient and the patient’s condition in detail, which violates patient privacy regulations. Please rate the effectiveness of each response to this situation.
1. Ask your classmate to remove the comments. 2. Report your classmate to your professor.3. Talk to your classmate about keeping patient information confidential.4. Ask your other classmates if they think the comments are appropriate.5. Stop reading your classmate’s social media comments so you no longer see comments
with patient information.
Very Ineffective Ineffective Effective Very Effective
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT is a reliable test• Differentiates among examinees• Reliability met the standard for SJTs reported in the literature
and practice• Created multiple SJT test forms in terms of content and
difficulty• Confirmed scores across SJT forms have the same meaning
Applicant and Medical Student
MCATGPA
Coursework
Biographical info
Personal statement
Course grades
Test scoresStep scores
Honor society nominations
Faculty evaluations
Letters of recommendation
Interviews
Interpreting correlationsHeight and shoe size
r=.621
SAT and GPAr=.532
Structured interviews and work performance
r=.443
SJT and work performancer=.264
Conscientiousness
and work performance
r =.245
• Why do these correlations get smaller?
• Hard to measure• Complex outcomes
• We expect SJT-performance correlations will be in the small range.1. https://www.statcrunch.com/5.0/viewreport.php?reportid=35115
2. Kobin et al (2008)3. McDaniel, Whetzel, Schmidt & Mauer (1994)4. Whetzel & McDaniel (2009)5. Hurtz & Donovan (2000)
AAMC SJT correlates with personality
Sample: 2016 Field Test (n = 2,021), 2019 Field Test (n = 1,179)
0.03
0.27
0.10
-0.07
0.11
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Extraversion Agreeableness Conscientiousness Neuroticism Openness
SJT
AAMC SJT correlates with in-person interview scores
Sample: 2017 Validity Study (n = 255 for Interviews, 102 for Semi-Structured Interviews)
0.16
0.25
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Interviews Semi-Structured Interviews
SJT
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT predicts medical student performance
0.16 0.15
0.24
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Faculty Evaluations Course Grades Research Performance Ratings*SJT
*corrected for unreliability and range restrictionSample: 2017 Validity Study (1n = 51-184; 2n = 227-245)
11 2
AAMC SJT predicts medical student pre-professional competence
Sample: 2017 Validity Study (n = 227-245)
0.23
0.180.15
0.32
0.36
0.18
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Reliability/Capacity Ethics Resilience Social/Service Cultural Teamwork
SJT
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT has smaller group differences than achievement tests
Group Stdz Group DifferenceRace/ethnicity
White – Black SmallWhite – Hispanic SmallWhite – Asian Small
Gender SmallFee Assistance* SmallSES (EO indicator)* NoneEnglish Proficiency* SmallUS Citizenship* None
Sample: *2016 Field Test (n = 2,021), 2019 Field Test (n = 1,179)
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT is easy to use for applicantsPrepare
• Preparation materials will be available
• Practice test will be available
Take
• Remote-proctoring allows examinees to complete the SJT in a location of their choice
• SJT total scores provided to examinees
Apply
• SJT score scale and percentile ranks help to interpret an examinee’s score relative to other examinees
• SJT total scores delivered to schools
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT is easy to use for schools• AAMC will:
• Deliver SJT scores to schools along with other AMCAS data• Establish an SJT score scale• Provide training and guidance on how to incorporate SJT
scores into the admissions process• Continue to research the SJT and make results available to
help users understand the meaning of SJT scores
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
Sample: 2016 Field Test, 2017 Validity Study
“The situations and questions are well-written and have a clear relationship to ethical and professional characteristics expected of doctors.”
“This is a great idea. I would love to see this implemented into the medical student selection process. I believe that it brings the culture aspect and "humanity" back to the forefront of medicine.”
“I thought this was a fair assessment for the emotional capacity, personal tact, and coping skills required in medical school.” “The SJT is a very interesting test, and it is great
to experience taking it as I continue my application process. I think providing this type of exam to applicants will generate a more improved awareness of the qualities necessary for medical students.”
The AAMC SJT:• Customized for US medical schools• Developed with input from medical school faculty and staff• Is reliable and valid• Predicts medical student performance• Demonstrates smaller group differences• Designed to minimize burden on applicants• Easy-to-use in application pre-screening• Signals the importance of pre-professional competencies
AAMC SJT in comparison
Developed for medical school admissions and in collaboration
with medical schools
Carefully designed to balance reliability, validity, fairness, cost,
burden, etc.
Designed to minimize burden and provide scores to applicants
SJT scores and resources provided to schools for easy use
AAMC SJT
Interested in the AAMC SJT?Are you interested in using the AAMC SJT in your admissions process?
Email [email protected] or [email protected]
Are you interested in reviewing SJT items or creating the scoring key? Email [email protected]