What are we to do?

Jim Crossleyfor Maastricht 2012

Outline

• A crisis in WPBA

• A brief history in lenses

• Construct-aligned scales

• Some implications

Outline

• A crisis in WPBA

• A brief history in lenses

• Construct-aligned scales

• Some implications

Crisis

UK Academy of Medical Royal Colleges

2010“The profession has rightly

lost all confidence in Workplace Based

Assessment”

Outline

• A crisis in WPBA

• A brief history in lenses

• Construct-aligned scales

• Some implications

Getting a clear picture

First lens - constructs

?

Getting a clear picture

Getting a clear picture

Domain of Interest

Test Sample

Test Sample

Sampling

Second lens - sampling

?

Getting a clear picture

Outline

• A crisis in WPBA

• A brief history in lenses

• Construct-aligned scales

• Some implications

Surgical skills study

• 3 instruments• 1635 assessments• 437 cases• 15 index procedures• 5 specialities• 85 surgical trainees• 155 judges

Modelled reliability with

3 cases & 2 observers

• PBA, G=0.81

• NOTTS, G=0.59

• OSATS, G=0.65

PBA response scale• Level 1 Unable to perform the

procedure, or part observed, under supervision

• Level 2 Able to perform the procedure, or part observed, under supervision

• Level 3 Able to perform the procedure with minimum supervision (needed occasional help)

• Level 4 Competent to perform the procedure unsupervised (could deal with complications that arose)

• Very Poor

• Poor

• Good

• Very good

educ

atio

nal

anch

ors

slid

ing

clin

ical

anc

hors

ACAT 35 12CBD 8 3mini-CEX 7 3raters to reach G=0.7

It generalises

Getting a clear picture

Third lens – judge alignment

?

Outline

• A crisis in WPBA

• A brief history in lenses

• Construct-aligned scales

• Some implications

Understanding development

Please join in!

judgement and obscurity

Now we see as through a glass darkly; then we shall see face to face. Now I know in part; then I shall know fully, even as I am fully known.1 Corinthians 13:12