Constructing flawless MCQs for assessment and learning.pdf

8/17/2019 Constructing flawless MCQs for assessment and learning.pdf

1/37

Constructing flawless MCQs

for assessment and learning

Jack W. Strandhoy, PhD


2/37

Students can escape from

bad teaching, but not frompoor assessment (Boyd, 1995)


3/37

Evaluation

Learning Objective Content

Integrated Components of Teaching

and Learning


4/37

How can we improve our MCQs?

•

Purpose: formative, summative

•

Focus: important concepts, common,

clear-cut, or potentially catastrophicclinical problems

• Format: question type, no technical flaws

•

Level: strive to test application ofknowledge, higher order thinking


5/37

From D. DaRosa


6/37

Fundamental knowledge item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d.

Pralidoxime *

e. Prazosin

Which of the following dephosphorylates AChE?


7/37

Applied basic science mechanism item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d.

Pralidoxime *

e. Prazosin

Reactivation of cholinesterase shortly after toxicexposure to an OP insecticide can be accomplishedwith:


8/37

Clinical task item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d.

Pralidoxime *

e. Prazosin

You are treating a farm worker in the ED who wasaccidentally poisoned with an OP. You treat parasym.symptoms with atropine. Which drug do you choose toreactivate cholinesterase?


9/37

A 13 year-old boy is brought to the ED 5 days

after a camping trip. He has a petechial rash

on his arms and legs that began on the wrists

and ankles and spread in the last two days.

He has a headache, malaise, fever and chills.

A skin biopsy and Ab titer are positive for a

rickettsial infection. He is prescribed anantibiotic that acts by which of the following

mechanisms?

A. Acts as a detergent on the rickettsial cell wall

B. Binds at the 30S ribosome to prevent attachment of tRNA

C.

Depolymerizes microtubules in the cytosol

D.

Inhibits DNA gyrase

E. Prevents ergosterol biosynthesis

Integrating information

Note: Info from several disciplines but question is focused on one topic.


10/37

Other formats

•

Video clips, sound files, radiographic

images, virtual microscopy

•

Much easier with computerized examsthan on paper

• Can test higher order thinking

•

Being incorporated into NBME exams


11/37

Type A one-best-answer items

•

Focus on important concepts; common,

clear-cut, or catastrophic clinical problem

•

Test application of knowledge rather thanfactual recall

• Question answer should be clear and

answerable from just the stem

•

Distractors should be homogeneous

• Avoid technical flaws


12/37

Questions with structural flaws

•

Which of the followingpairs has won the most

Abby awards?

• A. Jones & Smith

• B. Smith & Taylor

•

C. Smith & White

• D. White & Allen


13/37


•


Abby awards?



•

C. Smith & White


• Ans = C


14/37


•


Abby awards?



•

C. Smith & White


• Ans = C

•

Which of the followingdrug combinations is

most likely to produce

ototoxicity?

• A. tetracycline & HCTZ

•

B. gentamicin & furosemide• C. ampicillin & furosemide

• D. gentamicin & triamterene

• Ans = B


15/37


•

Grammatical clues – a and an

•

Longest answer is correct

•

Absolutes – always, never [usu. wrong]•

Vague, weasel words – usually, often,

frequently

•

Negatives, double negatives,except

or

not statements


16/37


•

Answers too long or complicated

•

Word repeats as clues

•

Inconsistent, awkward structure•

Terms like none of the above


17/37

Interpretation of imprecise terms

NBME, 2003


18/37


•

Stem should provide all of the information

necessary to answer the question

•

Flawed: – Which of the following is true about

pseudogout?

• Use longer stems and shorter options

rather than the other way around


19/37

Elementary School Quiz

•

True/False:

–

A shape that has 4 sides is called a square

–

An apple is a sphere – The numerical value of the term ! is

3.1415926585


20/37

Medical School Quiz

•

True/False:

–

The incidence of cystic fibrosis is 1:2000

–

Children with CF usually die in their teens – Males with CF are sterile


21/37

Be aware:

All of the following concerning H. pylori and peptic

ulcer dz (PUD)are correct EXCEPT: A.

Infection w/ this organism is seen in themajority of pts with PUD

B.

Infection enhances gastric acid secretion

C.

Only a minority of infected pts develop PUDD.

The organism secretes urease that breaksdown urea into NH3 and CO2, which forms thebasis of a diagnostic test for the presence ofthe organism.

= multiple T / F


22/37

Type A: One best answer

•

Which of the following most approximates

a sphere?

–

Apple –

Banana

–

Frisbee

–

Hula hoop

–

Pretzel


23/37

•

Which of the following most approximates

a sphere?

–

Apple –

Basketball

–

Egg

–

Charlie Browns head

–

Watermelon


24/37

Type R questions:

Extended Matching•

Theme: broad or narrow

•

Options list: homogeneous, alphabetical.

Words or short phrases•

Lead in statement: match how?

•

At least two stems (small vignettes). Make

similar in structure within a set

•

Make sure theres only 1 best answer and

at least 4 distractors/stem


25/37

A. gentamicin G. penicillin VB. hydrochlorothiazide H. prednisone

C.

lithium carbonate I. primaquineD. metronidazole J. vancomycin

E.

allopurinol F. morpine sulfate

For each pt. description, select the MOST likely drug that was administered.

A 9 yo boy is admitted with gram (+) sepsis. An antibiotic is quicklyinfused IV. The child complains of a flushed feeling and you note

redness around the face and neck.

A 35 yo woman psychiatry patient complains of excessive urination andthirst. Urine osmolality was 100 mOsm/Kg water.

A 35 yo woman with lupus nephritis is treated with chronic medication.She develops gastric irritation, altered fat deposition to her face and

back, and osteoporosis.

Type R questions


26/37

A. Acetaminophen P. Sulfasalazine

B. Amiodarone Q. Tetracycline

C. ACE inhibitors ….. R. Verapamil

A 56 yo man with recurrent ventricular arrhythmias

began taking an antiarrhythmic drug 5 mo ago. He nowhas progressive cough, dyspnea and low-grade fever.

ESR is increased. CXR shows diffuse interstitial

pneumonia. PFT show that diffusing capacity for CO is

decreased. {amiodarone}

A 62 yo man with COPD begins therapy with anantihypertensive drug. Two wks later, he has marked

worsening of dyspnea and clearly audible wheezing.

{propranolol}

NBME examples, Type R

For each pt, select the drug most likely to have caused adverse effect


27/37

Adjust the difficulty of type R

questions with the optionse.g., antibiotics vs cephalosporins


28/37

Available from NBME.org Constructing Written Test

Questions for the Basic and

Clinical Sciences 3rd ed.

Susan Case & David Swanson

181 pages


29/37

Template for clinical vignette

style questions•

Item stem (patient vignette): describe a patientwith a problem – Age, gender; Site of care (ED, etc.)

–

HPI: Presenting complaint; Duration – Pt. hx; family hx; Physical findings, studies, Dx, initial

treatments

•

Lead-in: e.g., Which of the following is the mostlikely diagnosis?

•

Options: List of diagnoses

•

Long stem, short options better than v/v


30/37

Additional Item Templates

•

Which of the following is the most likely

mechanism for (drug) in patients with (dz)?

•

A patient has (signs, symptoms). Which ofthe following (drugs) is best treatment?

Which drug should be avoided?

• A patient has (side effects). Which drug is

she most likely taking?


31/37

A 13 year-old boy is brought to the ED 5 days

after a camping trip. He has a petechial rashon his arms and legs that began on the wrists

and ankles and spread in the last two days.

He has a headache, malaise, fever and chills.

A skin biopsy and Ab titer are positive for a

rickettsial infection. He is prescribed an

antibiotic that acts by which of the following

mechanisms?

A. Acts as a detergent on the rickettsial cell wall

B. Binds at the 30S ribosome to prevent attachment of tRNA

C.

Depolymerizes microtubules in the cytosolD.

Inhibits DNA gyrase

E. Prevents ergosterol biosynthesis

Integrating information

Note: Info from several disciplines but question is focused on one topic.


32/37

Using Clinical Vignettes

No vignette:

The most likely renal abnormality in children withnephrotic syndrome and normal renal function is:

A. Acute poststreptococcal glomerulonephritis

B. Hemolytic-uremic syndrome

C. Minimal change nephrotic syndrome*D. Nephrotic syndrome due to focal and segmental

glomerulosclerosis

E. Schönlein-Henoch purpura with nephritis

A B C D E

1 0 99 0 0 TOP 10%

8 1 90 1 0 BOTTOM 10%


33/37

Clinical summary vignette:A 2-year-old boy has a 1-week history of edema.

Blood pressure is 100/60. There is generalized

edema and ascites. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/

dL Urinalysis shows 4+ protein and no blood.The most likely diagnosis is:

A B C D E

1 0 98 2 0 TOP 10%

5 2 82 8 1 BOTTOM 10%


34/37

Clinical observations vignette:

A 2-year-old boy has developed swelling of his ankles andaround his eyes for the past week. Blood pressure is

100/60, pulse 110/min, and respirations 28/min. In

addition to swelling around his eyes and 2+ pitting

edema of his ankles, he has abdominal distention with

a positive fluid wave. Serum creatinine is 0.4 mg/dL,albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis

shows 4+ protein and no blood. The most likely

diagnosis is:

A B C D E

0 1 98 1 0 TOP 10%

10 9 66 10 5 BOTTOM 10%


35/37

Research vignettes

•

Item stem – lab data, graphs, dose-

response curves

•

Lead in – – Most likely interpretation?

–

Next logical experiment?

–

Best statistical analysis of the data?

–

Best control experiment?

• Options -


36/37

Guidelines for good questions•

Focus on important concepts; dont wastetime testing trivial facts

•

Match exam question to objective incontent and application; encourage higher

order thinking

•

Screen for technical flaws, ambiguities

• Examine poor question stats for reasons

•

Explore multidisciplinary vignettes andquestions; require data analysis foranswers


37/37

Guidelines for good questions

•

You should be able to put your hand over

the answers, read the stem, answer it

based on the information, then choose the

answer from the list.

• Good questions should test more than just

knowledge – integration, synthesis,

judgement.

Documents

Constructing flawless MCQs for assessment and learning.pdf