Ottawa 2010 Errors

8/8/2019 Ottawa 2010 Errors

1/16


2/16


3/16

Self assessment

Self assessment often inaccurate

Why? Do students give the same error attribution

as their tutors?

I. Colthart, et al. (2008). The effectiveness of self-assessment on the identification of learner needs, learneractivity, and impact on clinical practice: BEME Guide no. 10'. Medical Teacher 30(2):124-145.

Score = True score + Error component


4/16

Aims

Establish a taxonomy of common types oferrors

Determine ability to detect gold standarderrors

Encourage reflection in learners

Student engagement in assessment


5/16

Methods

Pilot study 67 students included

of 84 Failed one or both

semester exams

Student consent

Mock exam 10 SAQ questions

SAQExam

Self-

marked

Tutor-

marked

Errorscoded

Errorscoded

ComparisonStandardisedcoding sheet


6/16

Errors (overall)

Tutor mark= maxmark?

Studentmark = max

mark?

N

Studentmark = max

mark

Y

TruePositive

FalsePositive

FalseNegative

TrueNegative

Error No Error

E r r o r

N o

E r r o r

Tutor

S t u d e n

t

False Positive

True Negative

N

Y

False Negative

True Positive

Y

N


7/16

Errors (specific)

Tutormark =

maxmark?

Tutorcode = X*

N

Exclude

Y

An error has occurred

*Where X is the test

Y

Studentcode = X*

N

Studentcode = X*

TruePositive

FalseNegative

Y

N

FalsePositive

TrueNegative

Y

N


8/16

Results


9/16

Results

Cronbachs Alpha 0.81Tutor: 29.45Student: 44.29p < .001Eta 2 .608


10/16

Error Detection (overall)

2107 Errors* Students detected

74%

Error prevalence 0.88 Sensitivity 0.74

True Positive1560

False Positive102

False Negative547

True Negative173

Error No Error

E r r o r

N o E r r o r

Tutor

S t u d e n t

*Instances where full marks were not achieved for a part of a question


11/16

Error Type DetectionComparison of tutor and student error attribution

Sensitivity Specificity

A 0.016 0.919B 0.012 0.908C 0.166 0.723D 0.032 0.837


12/16

Summary

Students are able to identify errors withhigh sensitivity

There is a difference in error attributionbetween students and their tutors


13/16

Future work

Is there a difference in error attributionbetween first, second and third yearstudents?

What factors are associated withcorrelation between student and tutor errorattribution?

Do interventions to help students attributeerrors improve correlation and improveacademic performance?


14/16

Acknowledgements

Professor Stewart Petersen, Professor of Medical Education, University of Leicester

Dr Heather Crick, Lecturer, University of Leicester

Dr Gordon French, Deputy Postgraduate Dean,East Midlands Healthcare Workforce Deanery Dr Adrian Hastings, Senior Clinical Educator,

University of Leicester

Dr David Matheson , Lecturer in Medical Education, Medical Education unit, University of Nottingham & East Midlands Healthcare Workforce Deanery


15/16

http://tinyurl.com/hebbes


16/16

References

Documents

Ottawa 2010 Errors