Feedback credibility in a formative postgraduate OSCE: Effects of examiner type

Lynfa Stroud, Matt Sibbald, Denyse Richardson Heather McDonald-Blumer, Rodrigo B. Cavalcanti Oct 1, 2016

I do not have an affiliation (financial or otherwise) with a

pharmaceutical, medical device or communications organization.

Je n’ai aucune affiliation (financière ou autre) avec une

entreprise pharmaceutique, un fabricant d’appareils

médicaux ou un cabinet de communication.

2

Background

• Greater emphasis on the perspective of the feedback recipient

• Importance of credibility of feedback provider

• Longitudinal relationships

Ericsson KA. Acad Med. 2004 / Archer JC. Med Educ. 2010 / Sargeant J. Med Educ. 2011 / Eva KW. Adv in Health Sci Educ. 2012 /

Watling C. Med Educ. 2014 / Watling C. Medical Educ. 2012 / Eva K. Acad Med. 2010.

3

Background

• Formative OSCEs as learning opportunities

• Prior exploratory study:

Faculty examiners rated more credible than SP examiners

Generalists more credible than specialists

4

Study Question

What factors affect residents’ perceptions of the credibility of

their examiners during a formative OSCE?

Faculty examiners versus SP examiners

Faculty examiners who are:

generalists

specialty-congruent with station content

specialty-incongruent with station content

5

Methods

• University of Toronto

• Internal Medicine Program

• 192 residents

• 2015

6

Formative OSCE

• 5 stations x 12 minutes each

Clinical cases x 2 – faculty

Physical exam – faculty

Communication – standardized patients (+faculty)

• 5 point global rating scale (blind)

• 4 minutes for formative feedback end of each station

7

192 RESIDENTS

OSCE Track 1

Clinical and PE: Faculty Mix

Communication: SP Feedback

107 RESIDENTS

OSCE Track 2

Clinical and PE: Faculty Mix

Communication: “Faculty” Feedback

85 RESIDENTS

Post-OSCE Questionnaire

Study Design

Post-OSCE Questionnaire

• Demographics

• Overall, how would you rate the:

CREDIBILITY of the examiner providing feedback on this station?

9

Analysis

• Resident and examiner demographics

• Univariate analysis

• Multivariable analysis

• Credibility of examiner as dependent variable

10

Results

11

Resident Demographics

12

Examiner Demographics

13

Univariate Analysis

RESIDENTS CREDIBILITY p

Gender M 6.25

0.03 F 6.41

Level

PGY 1 – Faculty 6.46

0.002

PGY 2 – Faculty 6.27

PGY 3 - Faculty 6.16

PGY1 – SP 5.68

0.132 PGY2 – SP 5.39

PGY3 - SP 5.00 14

EXAMINERS CREDIBILITY p

Gender

M - Faculty 6.45 <0.001

F – Faculty 6.19

M - SP 5.61 0.08

F - SP 5.13

Type Faculty 6.27

<0.001 SP 5.37

Faculty

Generalist 6.48

<0.001 Specialist – Congruent 6.56

Specialist - Incongruent

5.96

Familiarity Yes 6.43

0.45 No 6.32 15

Multivariable Analysis – Faculty Only

VARIABLE MEAN F p

PGY Level

1 = 6.32

5.3 0.005 2 = 6.30

3 = 6.07

Examiner Gender M = 6.35

11.7 0.001 F = 6.11

Examiner Congruity

Generalist = 6.39

32.5 <0.001 Congruent = 6.44

Incongruent = 5.86

Examiner Familiarity Y = 6.43

1.4 0.26 N = 6.32 16

Multivariable Analysis - Communication

VARIABLE MEAN F p

PGY Level

1 = 5.98

1.85 0.16 2 = 5.84

3 = 5.58

SP Gender M = 5.96

3.15 0.08 F = 5.65

Examiner Type SP = 5.36

25.44 <0.001 Faculty = 6.24

17

Discussion – Main Findings

18

• Residents rated the credibility of all examiners highly in a

formative OSCE

• Some characteristics of about learners and examiners may

influence credibility judgments in small, but significant ways:

Training level

Gender

Discussion – Main Findings

• Greater influence on credibility judgements by examiner

congruency with station and by examiner type:

Specialty-congruent versus specialty-incongruent

» Effect of expertise?

Faculty versus SP

» Unlikely content issue

» Unlikely domain issue

» More acceptable from familiar?

19

Acknowledgments

Co-investigators

Matthew Sibbald

Denyse Richardson Heather McDonald-Blumer Rodrigo B. Cavalcanti

Research Assistance Lisa St. Amant

Funding University of Toronto, Department of Medicine,

Education Scholarship & Research Grant

20

Thank you & Questions

Lynfa Stroud

lynfa.stroud@sunnybrook.ca

21

• Download the ICRE App,

• Visit the evaluation area in the Main Lobby, near Registration, or

• Go to: http://www.royalcollege.ca/icre-evaluations to complete the session evaluation.

Help us improve. Your input matters.

• Téléchargez l’application de la CIFR

• Visitez la zone d’évaluation dans le hall principal, près du comptoir d’inscription, ou

• Visitez le http://www.collegeroyal.ca/evaluations-

cifr afin de remplir une évaluation de la séance.

Aidez-nous à nous améliorer. Votre opinion compte!

You could be entered to win 1 of 3 $100 gift cards.

Vous courrez la chance de gagner l’un des trois chèques-cadeaux d’une valeur de 100.

22

Extra Slides

23

Example Exam

Tues May 26 PGY2

TRACK 1 TRACK 2

Hemolytic Anemia Hematologist Generalist

Hyper-parathyroid Geriatrician Endocrinologist

Diplopia Neurology Gastroenterology

Error Disclosure Standardized Patient Faculty

Simulator NA NA

24

Limitations

1. Single institution, single program

2. Ceiling effect

3. Unclear if / how affected utilization of feedback

25

Feedback Characteristics

Faculty only Stations Faculty

Specific 5.86

Quality 6.03

Credibility of Examiner 6.31

Communication Stations Faculty SP

Specific 5.84 5.33

Quality 5.99 5.40

Credibility of Examiner 6.27 5.37

26

Effect of Gender Congruency

Resident

Examiner F M Grand Total

F 6.28 6.11 6.19

M 6.56 6.39 6.45

Grand Total 6.41 6.25 6.31

27 p=0.37

Examiner vs SP Score - Communication

MEAN SD RANGE r p

Examiner Score 3.43 0.98 1-5

0.39 <0.001

SP Score 3.99 0.80 2-5

28

Examiner vs Self Score – Clinical & PE

MEAN SD RANGE r p

Examiner Score

2.93 0.92 1-5 0.35 <0.001

Self Score 3.49 0.83 1-5

29