Standard setting Determining the pass mark

- OSCEs

The Old Way (1)

……..I think that ..I think that the pass mark the pass mark in this is exam in this is exam is probably is probably about here…..about here…..

there is a there is a natural break I natural break I can see can see dividing the dividing the bottom from bottom from the restthe rest

The Old Way (2)

……..The pass ..The pass mark is 60%.mark is 60%.

Assessments

Assessments

Match OutcomeObjectives

Assessments

Match OutcomeObjectives

Integrated

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Progressivetesting

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Progressivetesting

Housestyle

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Progressivetesting

Housestyle

From approvedlist

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Progressivetesting

Housestyle

From approvedlist

Studentinformation

Assessments

Match OutcomeObjectives

IntegratedClinical Competenciestested throughout

Progressivetesting

Housestyle

From approvedlist

Studentinformation

External Examiners

Standard setting

Knowledge Skills

Angoff

Hofstee

Borderline method

Knowledge Skills

Standard setting

Borderline procedureBorderline procedure

Used for clinical (OSCE) examinations

Borderline procedureBorderline procedure

Used for clinical (OSCE) examinations

Examiners score the students’s performance at the station e.g 17/20

Examiners judge the overall performance

clear pass / borderline / clear fail

Mark sheets rated borderline identified and the scores of borderline students averaged

Process repeated for each station

Calculate the median borderline score for all stations

Item I Clinical Problem

Type of station/question

Max marks available

Median Borderline score

Station1 Cranial nerves Examination 40 28.75

Station2b Patient education

Communication 20 13.38

Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00

Overall borderline score

510 297.89

Overall borderline score %

100

58.41

Borderline Method

1 2 3

Pass/Fail

Station 15 (static) Oral Lesions

0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00

Score

0

10

20

30

40

50

Co

un

t

Borderlinegroup

pass

fail

Median Borderline score 58.41

Borderlinegroup

pass

fail

Median Borderline score 58.41 How wide should

this band be?

Borderlinegroup

pass

fail

Median Borderline score 58.41 How wide should

this band be?

+/- 1 - standard deviation- standard error- or what?

The Standard Error of MeasurementThe Standard Error of Measurement

• depends on the reliability of the test (R)

• depends on the standard deviation of the test (SD)

• SEM = SD 1 - R• acts as a confidence interval in high stakes situations

The Standard Error of MeasurementThe Standard Error of Measurement

• depends on the reliability of the test (R)

• depends on the standard deviation of the test (SD)

• SEM = SD 1 - R• acts as a confidence interval in high stakes situations

R = Variance Variance x Error

Borderlinegroup

Pass

Fail

Median Borderline score 58.41

Pass Score60.93

} - 1 SEM

Fail score 55.88

} + 1 SEM

Number of students

Mean Score (%)

Overall Borderline Score (%)

Standard deviation

Standard Error of Measurement

Pass mark (%)

Reliability

04 195 73.25

58.28 5.50 2.40 60.68

0.81

03 215 74.4 58.41 5.77 2.52 60.93

0.81

02 214 73. 0 59. 81 4.95 2.98 62.79

0.64

01 195 71.85

58.90 5.86 3.21 62.11

0.70

Internal Reliability of the ExamCronbach’s Alpha

Cronbach’s alpha

Shows whether randomly split halves of the exam by item vary together

Split halfCorrelations

(SH)

Item / Stations

Population

(students)

Borderline

Grading System

Borderline

Fail

Pass

Good Pass

Top Mark

Borderline Mark

Distinction

Pass Mark

Pass

Fail

Reducing measurement errorReducing measurement error

1. Increase reliability of exam

2. Compromise with feasibility/cost

* Content specificity

* More items in the exam

* Blueprinting

* Quality assurance of item writing

* Training examiners/standardised patients

* Feedback from exam performance