DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS:

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE

School of Medicine, Flinders University

Train the Trainer WorkshopOctober 28, 2003

The sit uati on of Moun t Lof ty was fo

f rom hence and from some ot her cr

beari ngs, to be 34¡ 59 ' south and 13

east. N o l and w as vi si bl e so f ar to th

nort h as where t h e trees appeare d a

t he horizon , w hi ch show ed the coas

be v er y low, and our soundings we

f ast dec reasing.

noon t o six o'cl ock we ran th irt y t o t he nor thward, sk irt ing a sandy at the d istance of fi ve, and t he nce t mi les; the dept h was then 5 s, and we dropped the anchor upon

om of sand , mi xed wit h p ieces of o ral.

CONSTRUCTED RESPONSE ITEMS:

MEQ & SAQ

Hong Kong International Consortium

CONSTRUCTED RESPONSE ITEMS

ASSUMPTIONS

• Higher cognitive processes

• Increased reliability through wider sampling

ARE THE ASSUMPTIONS JUSTIFIED?

MODIFIED ESSAY QUESTIONS: MEQs

• 1970s – 1980s

• RCGP (Hodgkin & Knox 1975)

• Newcastle Australia (Feletti 1980, Feletti and Smith 1986)

• Jefferson USA (Rabinowitz 1987, Rabinowitz and Hojat 1989)

• Learning objectives, concepts, deep structures

• Clinical scenario

• 7 – 10 questions

• Sequential

(information question information question ...)

• Integrated (content, media – Rabinowitz 1985)


HIGHER COGNITIVE ORDER(Bloom 1956, Buckwalter et al 1981)

1. Knowledge Level 1 Recognition Recall

2. Comprehension Level 2 ComprehensionInterpretation

3. Application

4. AnalysisLevel 3 Application

5. Synthesis

6. Evaluation

(Irwin and Bamber 1982)


LEVEL 1 RECOGNITION AND RECALL

Know

• What is …..?

• List …..

• Identify …..



LEVEL 2 COMPREHENSION, INTERPRETATION

Understand

• Give reasons ….

• Explain ….

• Interpret ….

LEVEL 3 APPLICATION

Apply, Utilise, Judge

• How does X assist in explaining Y?

• What are the components of X?How are they related?How do they affect Y?

• How does XYZ affect treatment and management?

• Assess, judge X in terms of ….


MODIFIED ESSAY QUESTIONS – VALIDITY

• CONTENT

(Blueprints, objectives, deep structures)

• CONSTRUCT

(Level 1, 2, 3 questions)


• CONSTRUCT

Norman et al 1987Effect of educational levelMCQ and MEQ not different skillsUEQ may be a different skillSmall effect of context

Feletti 1980More Level 1 and 2 in first year

undergraduate

Irwin and Bamber 1986More Level 3 in final year undergraduate

• PREDICTIVE

Irwin and Bamber 1986 Correlation with clinical examination 0.41

Rabinowitz 1987 MCQ better predictor NBME I & II

Rabinowitz and Hojat 1989 Higher correlation for MEQ than MCQ with overall clerkship ratings


• Inter-rater reliability (model answers)

Norman et al 1987Inter-rater reliability satisfactory

• Internal consistency (cumulative errors)

Feletti 1980Alpha(60) 0.57 – 0.91

Stratford and Pierce-Fenn 1985Alpha(60) 0.39

MODIFIED ESSAY QUESTIONS – RELIABILITY

• Administration

• Marking

• Higher order questions

MODIFIED ESSAY QUESTIONS – FEASIBILITY

• Model PBL process

• Determine objectives

• Construct scenario

MINI-CASES: THE FLINDERS MEQ

John Taylor (age 55) has been brought to the E.D. by his wife. John

woke up this morning (one hour ago) with nausea and vertigo. He

managed to walk to the bathroom but had to hold onto the furniture

and walls to steady himself. He vomited when he reached the

bathroom.

1. Give 3 hypotheses to account for John’s condition.

2. Explain the mechanisms which give rise to John’s condition.

3. What further information would you like to distinguish between hypotheses. Explain how this information will distinguish between hypotheses.

MINICASES: THE FLINDERS MEQ

• Provide further information

• Cumulative error cues

• Continue scenario


John has had no significant medical problems in the last few years. He spent the previous evening at home watching TV and did not drink any alcohol or take any other substance.

….. The left pupil is smaller than the right …..

….. John’s uvula deviates to the right …..

….. Movements of the left hand are clumsy.

MINICASES: THE FLINDERS MEQ

• Further questions

• Continue unfolding scenario

• Page breaks, cumulative cues

• Conclude with SAQs (Level 2/3)


• Face validity

• Construct validity

• Model answers



• Reform of RCGP MEQ– Swanson– Mullholand & McAleer

• Increase number (10 – 12 / 2 hours)– Reflect reality

• Increase independence

• Single marked – schedules

• SAQ

(Lockie et al 1990)

SHORT ANSWER QUESTIONS - SAQ

Evaluation: “The ability to make a value-judgement in relation to

relevant criteria”

Bloom 1956

contextual information + Q responsecriteria

Opportunity to target highest cognitive skills

• application - analysis - synthesis - evaluation

Flinders SAQ strategies

1. Provide multiple, detailed, relevant information

• Patient

• Problem

• Clinical care setting

• Your role


Targeting “evaluation” response

• Ron Harkin is a 32 year-old father of two and self-employed

gardener. He falls heavily. He is unable to stand and is in

considerable pain. He is taken to hospital where you are the Intern

in the Emergency Department. You note swelling and that his foot

appears to be at an unusual angle to his leg.

• A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild

fever and painful knee to a small rural hospital. Her mother has

brought her 50km to the hospital. You are the GP called to see

her.

Example scenario information

2. Pose question requiring value-judgement• focus the response

• assume criteria are understood

• relate to what you would do

3. Extend information / probe further (optional)


Targeting “evaluation” response

Example questions targeting “evaluation”

Rationalising the diagnosis

• What further information from the history do you require?

• On what aspects of the physical examination will you focus?

• What investigations would you undertake to establish the diagnosis?

• What are your differential diagnoses? Which do you consider most likely?

Formulating an appropriate management plan

• What would be your immediate management goals?

• Which treatments would you advise?

• For what potential complications would you monitor?

• What longer-term management strategy would you advise?

• What advice would you give in relation to this new development?

Example questions targeting “evaluation”

Mr Dawes, a 58 yo man, presents to the Emergency Dept with a 24h history

of vomiting and colicky abdominal pain. He has had no bowel movement for

24h. He had an appendicectomy at age 30 to remove a gangrenous

appendix. On examination you find he has a distended abdomen with no

focal tenderness. There is no palpable hernia. He has symptoms and signs

consistent with dehydration of about 10%. Fluid resuscitation is

commenced. A plain abdominal Xray shows distended loops of small bowel

with air fluid levels and normal calibre large bowel.

Q1. What is your most likely diagnosis for this patient? (2 marks)

Q2. What possible causes of this would you consider? (3 marks)

After 48h of conservative management, Mr Dawes’ condition is such that surgical intervention is warranted.

Q3. What clinical indications would suggest to you that Mr Dawes needs surgery to resolve his problem at this time?

(5 marks)

Fairly reward sound responses

• Model answer (checklist)

• Writers / markers who understand:

- curriculum content

- learner stage

• Ongoing item improvement

Flinders SAQ strategies

• Are the claims justified?

• Do they measure anything different?(Maquire, Skakun, Triska 1997)

• What role do they have in a balanced assessment program?


REQUIREMENTS

• Write to a blueprint

• Sample widely

• Increase level 2 and 3 questions

• Model answers


THE END

You see Rebecca and Robert who are residents at a local training

centre for 27 young mentally handicapped patients. They are

accompanied by their care helper. Rebecca and Robert have

scabies.

Discuss your management

• communication

• clinical management

• health in the community

• sensitivity to the mentally handicapped

• use of extended team

• ethical issues

Deep/Surface Structures

(Lockie et al 1990)


VALIDITY

Construct

Wright et al (1983) – Independent cognitive abilities

Predictive

Wass et al (2001) – correlate with clinical tests

RELIABILITY

Low to moderate

Des Marchais and Vu (1996)

Wakeford and Roberts (1984)

Good

Wright et al (1983) – marking schedules


FEASIBILITY

• Marking



• One clinical scenario – multiple questions

PAQ (Des Marchais and Vu 1996) – no sequence

SAQ (Flinders) sequence

• Level 1, 2 and 3 questions on one scenario

• Reliability

PAQ – low to moderate

• Feasibility

HYBRID APPROACHES

• Face validity

• Construct validity

• Model answers


THE FLINDERS HYBRID MODEL

Documents

DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE School of Medicine, Flinders University Train the Trainer Workshop October 28, 2003 CONSTRUCTED RESPONSE ITEMS: