Seeing Things Differently: Better Performance in Resuscitation-Based Simulation OSCEs is Associated

With Specific Gaze Patterns

Adam Szulewski, Gerhard Dashi, Nick McGraw, Rylan Egan, Andrew Hall,

Damon Dagnone, Dan Howes

Seeing Things Differently: Better Performance in Resuscitation-Based Simulation OSCEs is Associated With Specific Gaze Patterns

Adam Szulewski BSc, MD, FRCPC, MHPE

Assistant Professor, Dept of Emergency Medicine, Queen’s University

PhD Candidate, Maastricht University

Emergency, Trauma and RACE Team Physician, KGH

I do not have an affiliation (financial or otherwise) with a

pharmaceutical, medical device or communications organization.

Je n’ai aucune affiliation (financière ou autre) avec une

entreprise pharmaceutique, un fabricant d’appareils

médicaux ou un cabinet de communication.

3

Why resuscitation medicine?

http://www.cedars-sinai.edu/Patients/Programs-and-Services/Trauma-Program/Trauma-Program-Continuum-of-Care.aspx

Why resuscitation medicine?

• The stakes are high

• It is a complex environment

Numerous medical decisions

Team management, leadership and other elements of crisis resource

management (CRM) are key

• Little is known about expertise development in this setting

• Expert resususcitationists are easy to identify, but it is very

difficult to explain what exactly makes them expert

http://www.cedars-sinai.edu/Patients/Programs-and-Services/Trauma-Program/Trauma-Program-Continuum-of-Care.aspx

6

Why simulation?

7

• Common and effective method to teach CRM in resuscitation

medicine

• Allows for feedback, practice, curriculum integration

Why simulation?

Assessment challenges

• The milestones for the

development of competence (and

expertise) in resuscitation medicine

are not fully known.

• Expertise is tacit

8

http://www.innerbody.com/careers-in-health/psychology-degrees.html

Gaze tracking and expertise

• Getting inside the expert’s head

• There are measurable visual

differences between novices and

experts in numerous domains

(sports, driving) that can be

quantified using eye-tracking

9

Source: Frits Ahlefeldt / Wikimedia Commons

Basic premise: Assessing competence and expertise

expertise development

http://scrubs.wikia.com/wiki/My_First_Day https://dontfierme.files.wordpress.com/2008/01/jdiz.jpg

competence expertise

Question

• In medical simulation resuscitation-based OSCEs, how do

residents’ gaze patterns differ with respect to expert-rated

performance?

• A pilot study

Methods

12

84 cases

August OSCE 2 cases

February OSCE 2 cases

QSAT score by blinded external

assessor

Gaze-tracking metrics

QSAT

13

Gaze tracking

14

Diabetic ketoacidosis case

The amount of time it took to first check vital signs was negatively correlated with performance

The number of times the participant looked at the respiratory therapist was negatively correlated with

performance

High performers seem to rapidly recognize that the RT did not have much new (or relevant) information to

provide and check the vital signs early.

These high performers tend to treat the DKA algorithmically (the treatment of DKA is well described). As a

result, they don’t tend to focus their visual attention on the RT (as they have no relevant information to

provide).

Variables 1 2 3 4

Diabetic ketoacidosis (N = 11)

1. Average Score -

2. Number of RT views (First 60 Seconds) -.65** -

3. Time until participant checked patient’s

Vitals

-.70** .48 -

Ruptured abdominal aortic aneurysm case

The total time spent looking at the lab values and the ECG were negatively

correlated with performance

The number of times a participant looked at the RN was negatively correlated

with performance

The key to this scenario was quick diagnosis and then focusing attention on

getting the patient to an operating room.

The lab values, ECG and RN did not provide extra clues to the high performers.

Ruptured abdominal aortic aneurysm

1. Average Score -

2. Total time spent looking at lab values (First

60 Seconds)

-.50** -

3. Total time spent looking at ECG (First 60

Seconds)

-.39** .16 -

4. Number of RN views (First 60 Seconds) -.46** .54*** .117 -

Variables 1 2 3 4

Ethylene glycol toxicity

Ethylene glycol toxicity

1. Average Score -

2. Number of medication list views -.48** -

17

The number of medication list views was negatively correlated with performance The medication list did not provide any relevant information as to the eventual diagnosis in

this scenario. High performers looked at this list, analyzed it once, and then discarded it. They focused on

basic resuscitation priorities for an undifferentiated patient with altered mentation Poor performers seemed to come back to these pieces of information to try to determine

what was happening in the scenario as the diagnosis of ethylene glycol poisoning was not immediately apparent (reaching for clues?)

Variables 1 2 3 4

Conclusions

• Specific gaze patterns were correlated with objective

performance.

• These gaze behaviours were different from scenario to

scenario.

We hypothesize that this is because each scenario had different

“relevant” vs. “irrelevant” cues (or visual endpoints)

18

Conclusions

• Overall, higher performers were able to decide what was

relevant or irrelevant efficiently.

Irrelevant information was appropriately discarded and relevant

information was focused upon by the high performers.

Poor performers seemed to have difficulty deciding how to

appropriately divide their attention between relevant and

irrelevant information.

19

Next steps

• This was a pilot study

• Systematic (and partially automated) data analysis method

• Creating new scenarios with appropriate “task relevant” and

“task redundant” stimuli to test the hypotheses generated by

this pilot study.

20

expertise development

http://scrubs.wikia.com/wiki/My_First_Day https://dontfierme.files.wordpress.com/2008/01/jdiz.jpg

competence expertise

Assessing competence and expertise

Concluding thoughts

• This data provides a new way to think about our learners’

progress on the novice/expert continuum.

• Can we intervene earlier if we identify learners whose

expertise development is not progressing as expected?

• Implications for CBME

Would an examiner/program director/Royal College committee

want to know whether a candidate is novice-like or expert-like in

their cognitive and gaze behaviour characteristics (beyond

traditional assessment metrics)?

22

http://www.google.ca/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=lY7CX_2xV5n8-M&tbnid=cMAEykDATlx7lM:&ved=0CAQQjB0&url=http%3A%2F%2Fwww.jeffkorhan.com%2F2011%2F08%2Fask-your-customers-the-tough-questions.html&ei=mb5NU7e7CpLW2wX_n4GAAg&psig=AFQjCNG_RWJqyPaBUpl-VS0jkMACUJVD8w&ust=1397690285007845

• Download the ICRE App,

• Visit the evaluation area in the Main Lobby, near Registration, or

• Go to: http://www.royalcollege.ca/icre-evaluations to complete the session evaluation.

Help us improve. Your input matters.

• Téléchargez l’application de la CIFR

• Visitez la zone d’évaluation dans le hall principal, près du comptoir d’inscription, ou

• Visitez le http://www.collegeroyal.ca/evaluations-

cifr afin de remplir une évaluation de la séance.

Aidez-nous à nous améliorer. Votre opinion compte!

You could be entered to win 1 of 3 $100 gift cards.

Vous courrez la chance de gagner l’un des trois chèques-cadeaux d’une valeur de 100.

24