Script Concordance Test Jean Paul Fournier Département de Pédagogie Médicale Faculté de Médecine de Nice Sophia Antipolis, France

Script Concordance Test

Jean Paul Fournier

Département de Pédagogie MédicaleFaculté de Médecine de Nice Sophia Antipolis, France

Tempus ProgramMarch 12, 2012

Clinical reasoning

Patient’s complaint

Problem representation

Early generationof relevant hypothesis

Hypothesis check

Clinical data interpretation

Quest for newclinical / paraclinical data

Early problem identification(non analytic process)Compiled knowledge

Nendaz M. Pédagogie Médicale 2005

Pellacia Th. Annales Françaises de Médecine d’Urgence 2011

InstancesPrototypes

Semantic networks

Script activation

AnalyticprocessMicro judgements

(slots)


Clinical reasoning (2)

No hypothesis

Systematic approach

Diagnosis

No semantic network High complexity case

EpidemiologyPrevalence

Pathophysiology…

Forwardreasoning




Non analytic process Description

Pattern recognitionIntuitive identification of a typical signs configuration (= pattern), correspounding to diagosi(e)s

InstancesIntuitive identification of a déjà vue situation and immediate diagnosi(e)s generation




Analytic process Description

Hypothetico deductive process

The physician consciously tries to confirm / infirm his early hypotheses, with an oriented quest of clinical / paraclinical signs

Forward reasoning

The physician goes from clinical / paraclinical data to the problem solving, through application of causal or conditionnal rules



A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She suffers an ovarian carcinoma, in palliatie care. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. Last creatinine level was 150 μmol/l. She has venous disease sequellae and bilateral pitting edema. You think of a possible pulmonary embolism.

Venous ultrasound

Femoral vein thrombosis

D dimers assay unesuseful in such a

context

CT pulmonary angiogram

Bilateral proximal pulmonatry embolism

D dimers assay

D dimers > 5000 g/ml

With an high pre test probability and D dimers >

4000 : pulmonary embolism probability : 88 p. cent

CT pulmonary angiogram benefit/risk ?

Heparin

Physician 1 Physician 3D dimers assay

unesuseful in such a context

Physician 2


What happened ?

3 different pertinent approaches but with the same result : heparin therapy ;

3 different pertinent approaches for the resolution of the same problem

Impact of new data in initial hypotheses (micro judgements).

Uncertainty : D dimer results interpretation in an old wman with ovarian cancer, CT pulmonary angiogram risk/benefit in an old woman with chronic kidney impairment ;

Script activation

Clinicalreasoningvariability

Ponderation

Uncertainty


Script theory

Script : Compiled knowledge network (clinical, biological, imaging,…) used for a clinical problem solving

Scripts are elaborated from early patients encounters ;

The more frequent and diverse are encounters, the richest are scripts ;

Scripts cannot be transfered ;

They vary from a physician to anaother.

Charlin B. Acad Med 2000


Script Concordance Test Standardized assessment tool of clinical reasoning through the presence

and functionnality of scripts in simulated clinical situations in an uncertainty context

Standardized assessment tool ; Clinical situations ; Ill-defined situation and context : no consensus, no evidence-based medicine ; Assessment of compiled clinical reasoning ; Reference to an expert panel (aggregate scoring).

SCT : Principles

Comparing the attributed impact of a new information on initial hypothesis between experts and students ;

Experts take the same test as students (same questions, same timing) and serve as the reference panel ;

All experts answers are accepted (with the restriction of discordant answer or

expert) ;

The closest to the experts’ choice is the students’ choice, the most performing are students.


The SCT

If you were planing to prescribe…

And then you find… The impact in your hypothesis becomes…

A D-dimer assay An ongoing ovarian carcinoma in palliative care

-2 -1 0 +1 +2

A venous ultrasound A swollen painful left calf -2 -1 0 +1 +2

A CT pulmonary angiogram Créatinine level of 150 µmol/l -2 -1 0 +1 +2

-2 : Very unuseful-1 : Unuseful

0 : Neither more or less useful +1 : Useful+2 : Very useful

A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. You think of a possible pulmonary embolism.

Short clinical vignette

Pertinent hypothesis New relevant information

Anchor

Question

Likert scale : assessmentof the new information

on the initial hypotheses(Micro judgement)

Item

SCT vs rich-context MCQ


Is SCT an efficient assessment tool ?

A valid tool ? Face validity : depends on test construction (blueprint) ;

Construct validity :

identification of juniors vs seniors ;

prediction of scores for tests exploring the same domain ;

A reliable test ? Cronbach alpha from 0,79 to 0,82.

MCQ : 0,76 à 0,93 (2 to 8 hour testing)

Patient Management Problem : 0,69 à 0,82 (4 to 8 hour testing)

Key Feature Case : 0,66 à 0,79 (4 à 8 hour testing).


SCT among other assessment tests ?

MCQ ;

Clinical cases for ECN.



9523Clinical ED situations

Recommandations NBME

RecommandationsB. Charlin 2002

60 MCQ 60 SCT

Each situation appeararing in the 2 formats

Test taken by 16 EM residents




SCT vs true/false MCQ

Collard A. Med Educ 2009

106 medical students from 3rd to 6th year ;

Endocrinology tutorial ;

SCT vs true/false MCQ :

Significant correlation in 3rd and 4th year ;

Correlation for the 4 years with the knowledge level estimate (true vs false answers).


SCT vs ECN clinical case

18 medical students from 4th to 6th year ;

ECN clinical case ;

26 questions SCT.

Gibot S. Pédagogie Médicale 2008


SCT vs ECN clinical case

R = 0,55

Gibot S. Pédagogie Médicale 2008


SCT vs other assessment tools

Significant correlation, but low clinical significance : : some common points (assessment domain), but the SCT assesses some other aspects ;

Combining several assessment formats is a key-point in assessment.


Usual assessment tools

Knowledge

Knowledgeutilization

Competences

Action

True/false MCQ, essay-questions

Rich context MCQ, SCT, PMP, key-features

Standardized oral examination, OSCE, mini CEX simulation, SCT

Direct observation, peers, video, audits

Miller Acad. Med. 1990Compétence

Performance


How is it accepted ?

Bounouffe C. Am J Pharm Educ 2010


Technical aspects

Blueprint ;

Preparation ;

Anchor ;

Experts panel ;

Quality control.

Scoring ;


Blueprint4 points :

What do we do ?

Situations selection ?

Questions formats

Items and questions number

Certification ? Formative aspect ? Difficulty level ?

Prevalence ?

Severity ?

Diagnosis

Treatment

Management

Validity

Reliability

Feasibility


Blueprint

Cardio Pneumo Neuro Gastro Metabol. Rheumato Geriatr. Gyneco Psy Traumato

Dg X X X X X X X X X

Trt X X X X X X

Management X X X

Test developers + experts


Blueprint

Cardio Pneumo Neuro Gastro Metabol. Rheumato Geriatr. Gyneco Psy Trauma

Dg Chest pain

Acute shortness of

breath

Headache Abdominal pain

Mono arthritis

Altered general

condition

Ectopic pregnancy

Confusion Neck trauma

Trt CHF Pulmonary embolism

Seizure Gastric bleeding

Dehydrat. Agitation

Management ACS COPD exacerbation

Stroke

Test developers + experts


How many cases, how many questions ?

Case specificity ;

Many cases with one question ?

1 case with n questions ?

20 cases and 60 questions : Cronbach alpha ≥ 0,70

Formathomogeneity

Few cases with many questions ?



Gagnon R. Adv Health Sci Educ 2008



Gagnon R. Adv Health Sci Educ 2008


Preparation : situations, questions

Principles :

Clinical situations selection, fitted to students level ;

Relevant hypothesis selection in that context ;

New relevant informations selection, in that context

Page G. Academic Medicine 1995

Test developers+ experts

Uncertainty


Clinical vignette and questions preparation

Vignette : relevant informations :

Age, gender ;

Complaint, adlission, hospitalization reason ;

Context : ED, office, ward,…

Clinical data,…

1 to 2writers



Relevant new informations :

Clinical data ;

Biological test ;

Imaging ;

Video,…

1 to 2writers



Vignette, hypothesis and new informations :

Sweeping the five points of the Likert scale ;

Well-balanced variability level ;

All cases are independant ;

All questions are independant ;

One format by case (diagnosis, treatment,…).

Charlin B. Medical Education 2006


Question discrimination power and variability

Charlin B. Medical Education 2006


Lickert scale - Anchor

-1 : Unlikely-2 : Very unlikely

0 : Neither more or less likely

+1 : More likely+2 : Very likely

Recommandations :

Explicit the significance of each point of the Lickert scale, particularly the neutral point (« 0 ») ;

Keep the same presentation.



Lickert scales :

If you were thinking of the following diagnosis…

And then you find… This new information makes your diagnosis…

Diagnosis option New information -2 -1 0 +1 +2

-1 : Unlikely -2 : Very unlikely

0 : Neither more or less likely

+1 : More likely+2 : Very likely

Diagnosis anchor :



Lickert scales :

If you were considering the usefulness of the following

test…

And you find… This new information makes the following test…

Investigative option New information -2 -1 0 +1 +2

-1 : Useless-2 : Less useful

0 : Neither morer or less useful +1 : Useful+2 : Very useful

If you were considering the benefit/risk ratio of the

following test…

And you find… This new information makes the test…


-1 Contraindicated-2 : Strongly contraindicated

0 : Neither more or less indicated

+1 : Indicated+2 : Strongly indicated

Anchor for test(utility) :

Anchor for test(benefit/risk) :



Lickert scales :


test…

And you find… This new information makes the following test…




If you were considering the benefit/risk ratio of the

following test…

And you find… This new information makes the test…





Anchor for test(utility) :

Anchor for test(benefit/risk) :

Inadequate prescription

Non appropriate option Adequate prescription



Likert scales :


treatment…

And you find… This new information makes the following treatment…

Treatment option New information -2 -1 0 +1 +2



If you were considering the benefit/risk ratio of the following treatment…






Treatment anchor (utility) :

Treatment anchor (benefit/risk) :



Échelles :

Si vous considériez l’utilité du traitement suivant

Et qu’alors vous trouvez Cette nouvelle information rend le traitement




If you were considering the benefit/risk ratio of the following treatment…






Treatment anchor (utility) :

Treatment anchor (benefit/risk) :

Inadequate prescription

Non appropriate option Adequate prescription



Lickert scales :

Prognosis item New information The prognosis becomes

Prognosis item New information -2 -1 0 +1 +2

-1 : Worsened-2 : Very worsened

0 : Never more or less modified +1 : Improved+2 : Very improved

Prognosis anchor :


Experts panel

Roles :

Test preparation ;

Clinical vignette and questions validation ;

Scoring.

2 differentgroups


Experts panel

Quality : experts in the considered domain ;

Number : 10 to 20 minimum.

Gagnon R. Medical Education 2005


Mean reliability following the experts number

Gagnon R. Medical Education 2005


Expert panel composition

Teaching physicians vs physicians ?

Charlin B. Medical Teacher 2007

SCT in community medicine ;

Comparison of scores ;

Panel : teaching physicians vs physicians in a CME program ;

Perfect concordance of the scores (R = 0.98) ;

Scores attributed by physicians more elevated (p = NS).


Scoring

Scoring :

Panel choice :

Modal transformation of the panel choice :

Residents’ choice :

Residents’ scores :


Scoring

Excel® scoring calculator available :

www.cme.umontreal.ca/tcs


Summary

Test preparation Blueprint Clinical vignettes Hypotheses New informations Scoring Pass/fail decision

Experts 1 Clinical vignettes Hypotheses New informations

Experts 2 Scoring (reference panel scoring)

Validation

Quality control

A priori quality control :

Caire F. Neurochirurgie 2004Tempus ProgramMarch 12, 2012


Quality control

A posteriori quality control : item analysis :

Difficulty index ;

Discrimination index ;

Question impact on the test reliability (Cronbach alpha) ;

Etc…

Thank you for your attention !

Documents

Script Concordance Test Jean Paul Fournier Département de Pédagogie Médicale Faculté de Médecine de Nice Sophia Antipolis, France