Constructing flawless MCQs for assessment and learning Jack W. Strandhoy, PhD

Constructing flawless MCQs for assessment and learning

Jack W. Strandhoy, PhD

“Students can escape from bad teaching, but not from poor assessment” (Boyd, 1995)

Evaluation

Learning Objective

Content

Integrated Components of Teaching and Learning

How can we improve our MCQs?

• Purpose: formative, summative

• Focus: important concepts, common, clear-cut, or potentially catastrophic clinical problems

• Format: question type, no technical flaws

• Level: strive to test application of knowledge, higher order thinking

From D. DaRosa

Fundamental knowledge item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d. Pralidoxime *

e. Prazosin

Which of the following dephosphorylates AChE?

Applied basic science mechanism item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d. Pralidoxime *

e. Prazosin

Reactivation of cholinesterase shortly after toxic exposure to an OP insecticide can be accomplished with:

Clinical task item

a. Phenoxybenzamine

b. Phenylephrine

c. Physostigmine

d. Pralidoxime *

e. Prazosin

You are treating a farm worker in the ED who was accidentally poisoned with an OP. You treat parasym. symptoms with atropine. Which drug do you choose to reactivate cholinesterase?

A 13 year-old boy is brought to the ED 5 days after a camping trip. He has a petechial rash on his arms and legs that began on the wrists and ankles and spread in the last two days. He has a headache, malaise, fever and chills. A skin biopsy and Ab titer are positive for a rickettsial infection. He is prescribed an antibiotic that acts by which of the following mechanisms?

A. Acts as a detergent on the rickettsial cell wall

B. Binds at the 30S ribosome to prevent attachment of tRNA

C. Depolymerizes microtubules in the cytosol

D. Inhibits DNA gyrase

E. Prevents ergosterol biosynthesis

Integrating information

Note: Info from several disciplines but question is focused on one topic.

Other formats

• Video clips, sound files, radiographic images, virtual microscopy

• Much easier with computerized exams than on paper

• Can test higher order thinking

• Being incorporated into NBME exams

Type A one-best-answer items

• Focus on important concepts; common, clear-cut, or catastrophic clinical problem

• Test application of knowledge rather than factual recall

• Question answer should be clear and answerable from just the stem

• Distractors should be homogeneous

• Avoid technical flaws

Questions with structural flaws• Which of the following

pairs has won the most Abby awards?

• A. Jones & Smith• B. Smith & Taylor• C. Smith & White• D. White & Allen




• Ans = C




• Ans = C

• Which of the following drug combinations is most likely to produce ototoxicity?

• A. tetracycline & HCTZ• B. gentamicin & furosemide• C. ampicillin & furosemide• D. gentamicin & triamterene

• Ans = B

Questions with structural flaws

• Grammatical clues – a and an

• Longest answer is correct

• Absolutes – always, never [usu. wrong]

• Vague, “weasel words” – usually, often, frequently

• Negatives, double negatives, “except” or “not” statements


• Answers too long or complicated

• Word repeats as clues

• Inconsistent, awkward structure

• Terms like “none of the above”

Interpretation of imprecise terms

NBME, 2003


• Stem should provide all of the information necessary to answer the question

• Flawed: – Which of the following is true about

pseudogout?

• Use longer stems and shorter options rather than the other way around

Elementary School Quiz

• True/False:– A shape that has 4 sides is called a square– An apple is a sphere– The numerical value of the term π is

3.1415926585

Medical School Quiz

• True/False:– The incidence of cystic fibrosis is 1:2000– Children with CF usually die in their teens– Males with CF are sterile

Be aware:

All of the following concerning H. pylori and peptic ulcer dz (PUD)are correct EXCEPT:

A. Infection w/ this organism is seen in the majority of pts with PUD

B. Infection enhances gastric acid secretionC. Only a minority of infected pts develop PUDD. The organism secretes urease that breaks

down urea into NH3 and CO2, which forms the basis of a diagnostic test for the presence of the organism.

= multiple T / F

Type A: One best answer

• Which of the following most approximates a sphere?– Apple– Banana– Frisbee– Hula hoop– Pretzel

• Which of the following most approximates a sphere?– Apple– Basketball– Egg– Charlie Brown’s head– Watermelon

Type R questions:Extended Matching

• Theme: broad or narrow

• Options list: homogeneous, alphabetical. Words or short phrases

• Lead in statement: match how?

• At least two stems (small vignettes). Make similar in structure within a set

• Make sure there’s only 1 best answer and at least 4 distractors/stem

A. gentamicin G. penicillin VB. hydrochlorothiazide H. prednisoneC. lithium carbonate I. primaquineD. metronidazole J. vancomycinE. allopurinol F. morpine sulfate

For each pt. description, select the MOST likely drug that was administered.

A 9 yo boy is admitted with gram (+) sepsis. An antibiotic is quickly infused IV. The child complains of a flushed feeling and you note redness around the face and neck.

A 35 yo woman psychiatry patient complains of excessive urination and thirst. Urine osmolality was 100 mOsm/Kg water.

A 35 yo woman with lupus nephritis is treated with chronic medication. She develops gastric irritation, altered fat deposition to her face and back, and osteoporosis.

Type R questions

A. Acetaminophen P. Sulfasalazine

B. Amiodarone Q. Tetracycline

C. ACE inhibitors ….. R. Verapamil

A 56 yo man with recurrent ventricular arrhythmias began taking an antiarrhythmic drug 5 mo ago. He now has progressive cough, dyspnea and low-grade fever. ESR is increased. CXR shows diffuse interstitial pneumonia. PFT show that diffusing capacity for CO is decreased. {amiodarone}

A 62 yo man with COPD begins therapy with an antihypertensive drug. Two wks later, he has marked worsening of dyspnea and clearly audible wheezing. {propranolol}

NBME examples, Type R

For each pt, select the drug most likely to have caused adverse effect

Adjust the difficulty of type R questions with the optionse.g., antibiotics vs cephalosporins

Available from NBME.org

“Constructing Written Test Questions for the Basic and

Clinical Sciences” 3rd ed.

Susan Case & David Swanson

181 pages

Template for clinical vignette style questions

• Item stem (patient vignette): describe a patient with a problem– Age, gender; Site of care (ED, etc.)– HPI: Presenting complaint; Duration– Pt. hx; family hx; Physical findings, studies, Dx, initial

treatments

• Lead-in: e.g., Which of the following is the most likely diagnosis?

• Options: List of diagnoses• Long stem, short options better than v/v

Additional Item Templates

• Which of the following is the most likely mechanism for (drug) in patients with (dz)?

• A patient has (signs, symptoms). Which of the following (drugs) is best treatment? Which drug should be avoided?

• A patient has (side effects). Which drug is she most likely taking?

A 13 year-old boy is brought to the ED 5 days after a camping trip. He has a petechial rash on his arms and legs that began on the wrists and ankles and spread in the last two days. He has a headache, malaise, fever and chills. A skin biopsy and Ab titer are positive for a rickettsial infection. He is prescribed an antibiotic that acts by which of the following mechanisms?

A. Acts as a detergent on the rickettsial cell wall

B. Binds at the 30S ribosome to prevent attachment of tRNA

C. Depolymerizes microtubules in the cytosol

D. Inhibits DNA gyrase

E. Prevents ergosterol biosynthesis

Integrating information

Note: Info from several disciplines but question is focused on one topic.

Using Clinical Vignettes No vignette:The most likely renal abnormality in children with

nephrotic syndrome and normal renal function is:

A. Acute poststreptococcal glomerulonephritisB. Hemolytic-uremic syndromeC. Minimal change nephrotic syndrome*D. Nephrotic syndrome due to focal and segmental

glomerulosclerosisE. Schönlein-Henoch purpura with nephritis

A B C D E

1 0 99 0 0 TOP 10%

8 1 90 1 0 BOTTOM 10%

Clinical summary vignette:A 2-year-old boy has a 1-week history of edema.

Blood pressure is 100/60. There is generalized edema and ascites. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis shows 4+ protein and no blood. The most likely diagnosis is:

A B C D E

1 0 98 2 0 TOP 10%

5 2 82 8 1 BOTTOM 10%

Clinical observations vignette: A 2-year-old boy has developed swelling of his ankles and

around his eyes for the past week. Blood pressure is 100/60, pulse 110/min, and respirations 28/min. In addition to swelling around his eyes and 2+ pitting edema of his ankles, he has abdominal distention with a positive fluid wave. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis shows 4+ protein and no blood. The most likely diagnosis is:

A B C D E

0 1 98 1 0 TOP 10%

10 9 66 10 5 BOTTOM 10%

Research vignettes

• Item stem – lab data, graphs, dose-response curves

• Lead in – – Most likely interpretation?– Next logical experiment?– Best statistical analysis of the data?– Best control experiment?

• Options -

Guidelines for good questions

• Focus on important concepts; don’t waste time testing trivial facts

• Match exam question to objective in content and application; encourage higher order thinking

• Screen for technical flaws, ambiguities• Examine poor question stats for reasons• Explore multidisciplinary vignettes and

questions; require data analysis for answers

Guidelines for good questions

• You should be able to put your hand over the answers, read the stem, answer it based on the information, then choose the answer from the list.

• Good questions should test more than just knowledge – integration, synthesis, judgement.

Documents

Constructing flawless MCQs for assessment and learning Jack W. Strandhoy, PhD