IIME Workshop on IIME Workshop on AssessmentAssessmentMeeting SummaryMeeting Summary

Chengdu, ChinaChengdu, China

February 27, 2003February 27, 2003

MCQ SummaryMCQ Summary

• Hardest working group!!• 150 Item test translated into Chinese• Translation of each question reviewed by the

entire group for clarity• Examination details reviewed

– Location– Receiving and returning examination materials– Rules for how to proctor the examination

• Leader of MCQ team to recruit assistant proctor

OSCE SummaryOSCE Summary

• 15 Station (10 room) OSCE– 5 Long SP, 5 Short SP, 5 Non-SP Cases

• Blueprint of exam finalized• 1-day examination (3 sessions of 2 ½ hours)• Schools with 60 or 90 students can share

patients and/or repeat day 1 examination• CMB support provided through Association for

Chinese University Medical Colleges

Med 0201: Coronary heart

Med 0701:Asthma

Surg 0401:Acute choly

ER 0101: Low abdo pain

Psych 0601: Depression

Psych 0604:Panic disorder

Peds 0801: Malnutrition

Peds 0301: Fever + cough

OBG 0502:High risk preg

OBG 0501: Ectopic preg

OBG paper case, pelvic model, mcq

Surgery gowning or sterile technique,

tube, or CPR

Literature Search: computer

Medicine paper case, lab report, mcq,

auscultion

Medicine paper cases, Xray, ECG,mcq

Long SP cases Short SP cases Non-SP cases

2003 OSCE Day 1

Observation/LogbookObservation/Logbook Summary Summary• Scales finalized for all 17 GMER • Faculty at this meeting trained to use the

observation form• IIME to create evaluation form and send final

version before April 30, 2003• Faculty development plan workshops at your home

schools• Observations before the examination• Submit forms back to IIME for scoring and analysis

Tasks for March 2003Tasks for March 2003

• Leader for IIME project from each school will be primary contact for all information

• IIME will create spreadsheet for student registration process

• Leader registers students for examination– Spreadsheet returned to IIME with student

identification numbers– Additional data on students for comparison to your

school-based examinations and grades

• IIME creates draft of faculty observation forms

Tasks for April-June 2003:Tasks for April-June 2003:Faculty ObservationsFaculty Observations

• IIME creates final evaluation forms and sends to each school

• Schools begin faculty development workshops to teach how to use the evaluation forms

• Begin faculty observations in June or July for 3 months

• At least 3 evaluations for each student

Tasks for April – June 2003:Tasks for April – June 2003:Information for Students and TeachersInformation for Students and Teachers

• IIME will develop a brochure in Mandarin to educate students about the IIME project:– MCQ– OSCE– Faculty Observations

• Also provide brochure to teachers

• Draft score report to deans and presidents

Tasks for July:Tasks for July:Standardized Patient DevelopmentStandardized Patient Development

• Possible meeting in July (Shenyang and/or Chengdu)

• 1-day workshop

• OSCE Coordinator, Standardized Patient Trainer, OSCE team (1-2 people/school)

• Details of how to hire, train, and work with standardized patients.

Tasks for August – September Tasks for August – September 20032003

• Final room arrangements for MCQ, receive test books

• Final preparations for OSCE, complete training of SPs, recruit faculty

• Complete faculty observations in clerkships and prepare forms to return to IIME

October 2003!!!October 2003!!!

• MCQ Administration– Secure receipt, administration, and return of

test materials

• OSCE Administration– Score reports created on-site and email to

IIME

• Observations– Forms sent to IIME by secure mail

Timeline: 2003Timeline: 2003

MCQ OSCE OBSERVATION

March Student Lists

Final Stations

Draft Evaluation Forms

April Faculty Workshops

May Faculty Workshops

June Recruit Proctors

Recruit Faculty

Begin Assessments

Timeline: 2003Timeline: 2003

MCQ OSCE OBSERVATION

July Reserve Exam Room

Additional Meeting

Student Assessments

August Sample Material

Training of SPs

Student Assessments

September Receipt of Test Books

Test Center Preparation

Student Assessments

October Examination Examination Forms submitted

Quality Assurance ProgramQuality Assurance Program• MCQ examination

– Statistical review for quality of questions– Comparison with your home school examinations

• OSCE examination– Anonymous videotapes sent to review and score at

other schools

• Observations– Statistical review for quality of scores reported by faculty

• Student and faculty assessment of examination– How realistic?– Fair coverage of domains?

Score ReportsScore Reports• Data security in the IIME office• Score reports created by GMER item, by

discipline, and by student to return to IIME leader at your school.

• Comparison of scores with other Chinese students WILL BE anonymous and combined ONLY.

• Areas of success and improvement will be identified in this report

• Comparison of these tests with your school examinations can be provided

Score Report: Major DomainsScore Report: Major Domains

• Professional Values, Attitudes, Behaviors and Ethics

• Scientific Foundation of Medicine• Clinical Skills• Communication Skills• Population Health and Health Systems• Management of Information• Critical Thinking and Research

Score Report: ProfessionalismScore Report: ProfessionalismProfessional values which include excellence,

altruism, responsibility, compassion, empathy, accountability, honesty and integrity, and a commitment to scientific methods;

  Self-regulation and a recognition of the need for continuous self-improvement with an awareness of personal limitations including limitations of one’s medical knowledge;

Recognition of the moral obligation to provide end of life care, including palliation of symptoms;

 Ability to effectively plan and efficiently manage one’s own time and activities to cope with uncertainty, and the ability to adapt to change;

Personal responsibility for the care of individual patient.

Score Report: Scientific Score Report: Scientific FoundationsFoundations

01020304050

60708090

100

My School All Schools

My School

All Schools

Mean 74 72

5th Percentile 66 68

95th Percentile 82 78

Students Below Standard

2% 10%

Score Report: PediatricsScore Report: Pediatrics

01020304050

60708090

100

My School All Schools0

1020304050

60708090

100

My School All Schools

MCQ OSCE

Score Reports: Correlation of Score Reports: Correlation of Overall MCQ ExaminationOverall MCQ Examination

Correlation Significance

Entrance Examination r=0.45 p<0.001

Year 2 Examination r=0.52 p<0.001

Internal Medicine Clerkship r=0.31 p<0.01

Surgery Clerkship r=0.22 p<0.05

Pediatrics Clerkship r=0.28 p<0.05

Psychiatry Clerkship r=0.15 p<0.08

IIME 2003IIME 2003

• Security and confidentiality

• Pilot Study

Why We Will SucceedWhy We Will Succeed

• International Standard Outcomes

• International Standard Assessment Tools

• International Quality Assessment Process

• International Quality Leaders, Faculty and Assistants working on the IIME project at EVERY school!