Embed Size (px)
Citation preview
Item Writing Workshop
May 16, 2011
Sameano F. Porchea PhD
What is the purpose of a test?
To differentiate between students of differing knowledge/ability levels
That’s pretty much it as far as we’re concerned at this workshop.
2
In a Perfect World of Testing
0 100
Low Ability Students High Ability Students
You would get test scores something
like this…..
If the test items behave
properly.
But, 3
What if the items don’t “Behave Properly”?
0 100
You could get test scores something
like this…..
Low Ability
Student
High Ability
Student
Low Ability
Student
High Ability
Student
4
What do you mean “Behave Properly”?
If test items are “Behaving Properly”
1. Students with low ability will select the wrong answer more often than students with high ability and vice versa.
2. Students will select the correct option for one and only one reason:
They know the content!!Not because they are TEST-WISE.
5
What do you mean“Test-wise”?
You are test-wise if you can select the key based on clues given in the item without knowing the content.
Are you Test-wise?
6
1. The fribbled breg will ninter best with an
A. mors.B. ignu.C. derst.D. sortar.
Example 1
7
2. What probable causes are indicated when doss occurs in a compots?
A. The sabs foped and the doths tinzed.B. The kredges roted with the rots.C. The rakogs were not accepted in the sluth.D. The polats were thenced in the sluth.
Example 2
8
3. What is the primary purpose of the cluss in frumpaling?
A. remove cluss-prangsB. patch tremailsC. loosen cloughsD. repair plumots
Example 3
9
Components of Multiple Choice Items
• Multiple Choice Items have 2 parts.– Stem
• Question or opened ended statement– Options
• Key (correct answer)• Distractors (incorrect options)
10
What should be considered when writing items?
– Discrimination– Content Knowledge being assessed– Item Difficulty– Fundamental Item Flaws
11
Item Discrimination
An item should “discriminate” between high and low-ability students.
Better discrimination = better spread of test scores.
Better spread of test scores = more reliability.
12
Low Ability Students High Ability Students
Good Item Discrimination
(scores are spread out)
13
Low Ability
Student
High Ability
Student
Low Ability
Student
High Ability
Student
Poor Item Discrimination(very little spread in scores)
Leads to………..
Poor Test Reliability!!!!14
Poor Test Reliability
3
0 100
1
High Ability
Student
Low Ability
Student
0 100
High Ability
Student
Low Ability
Student
0 100
2
Low Ability
Student
High Ability
Student
15
Good Test Reliability
0 100
Low Ability
Student
1
High Ability
Student
High Ability
Student
0 100
2
Low Ability
Student
0 100
3
High Ability
Student
Low Ability
Student
16
How to Get Better Item Discrimination
Write items that assess only one thing – nursing ability.
17
How to Lose Item Discrimination
Write items that accidentally or deliberately attempt to assess multiple things (e.g., nursing ability, reading comprehension, common sense).
Write items that are too easy or too hard.
18
Test Wise-ness v.
Content Knowledge
Every test question should be a trick question! However,
Students should miss questions based on what they don’t know about nursing, not on how well they play the testing game.
19
Item Difficulty
How hard should an item be?
At least 30% should get the item correct.
At most 90% should get the item correct.
The “sweet spot” is 50 - 70% correct.
20
Item Difficulty Why 30 to 90%?
At less than 30% correct, it seems unlikely that the item is at a student level.
At more than 90% correct, the item is unlikely to contribute much to test score spread.
50% correct is the point at which the item has the most potential to spread test scores out.
21
Item Difficulty
“Shouldn’t we try to have some really easy items so poor students don’t get discouraged?”
A test is meant to assess, not to help with self-esteem.
“Throw away” items really won’t help with self-esteem much, but they will hurt the test.
22
0 100
Low Ability
Student
High Ability
Student
Low Ability
Student
High Ability
Student
23
Very Easy Item
Item Difficulty What about really easy items that assess
something critical to know?
This is okay for a classroom test if and only if... You take action to remediate any student
who misses the item.
You recognize that by having a really easy item overall test score reliability will be decreased.
24
Item DifficultyA nurse is providing oral care to a patient with dentures. Which of the following actions should the nurse perform first?
A assess the mouth and gums
B wash the patients face
C brush the dentures with tooth powder
D put on gloves
25
Item DifficultyWhich of the following is illegal when creating handouts for a class?
A. photocopying from a published book
B. creating the handouts from scratch
C. making extra copies of the handouts
D. using public domain resources
26
Item Difficulty “Shouldn’t we try to have some really hard
items to challenge the top students?”
If the items are kept within a 30 - 90% correct range, there will be sufficient challenge for the top students.
If they really are top students, they should have high scores.
27
0 100
Low Ability
Student
High Ability
Student
Low Ability
Student
High Ability
Student
28
Very Difficult Item
Item DifficultyAn older adult client is started on captopril (Capoten)
for hypertension. Which of the following gerontological considerations is most appropriate when initiating drugs for hypertension?
A Initial doses should be half that prescribed for younger adults and dosing intervals increased.
B Initial doses should be same as for younger adults but tapered until therapeutic effects are reached.
C Initial doses should be reduced initially, and then increased gradually over the next 6 months.
D Initial doses should be higher than for a younger adult.
29
Fundamental Item Flaws
Self evident keys Guess what I’m thinking! Trivial pursuit Preaching and teaching Controversy and the cutting edge Reading comprehension v. nursing content Definition questions
30
Self Evident Keys Test-wise students will be drawn to answers
like: “… within legal guidelines.” “…according to established safety
procedures.” “…use appropriate hand-washing
technique.” If an option simply reads like something
impossible to disagree with it is probably the key.
31
Self Evident KeysWhich of the following is appropriate when passing another car on a divided highway?
A. Pass when the center line is double yellow.B. Pass in compliance with posted speed limits
and traffic laws.C. Flash the headlights before passing.D. Close the gap between cars to one half of a
car length, then accelerate to pass.
32
Self Evident KeysA nurse in the staff development department at a hospital is orienting a group of new employees and is reviewing the manual on disaster planning. Which of the following should the nurse emphasize first to the group?
A The nurse's responsibility is to ensure client safety.B Employees should learn the code for disasters and
how to alert the hospital switchboard.C The employees should take direction from the charge
nurse on their assigned floor.D Disaster drills will be held periodically through the
year on all shifts.
33
Self Evident Keys
Dr. Porchea asks the nurse if it would be okay to continue drinking lots of pop and eating chocolate chip cookies every day. The proper response from the nurse should be
A. Sure, go ahead.B. Try to keep the soda drinking down to 12
a day.C. Stop that right now and eat your broccoli.D. Okay, but have a vegetable with that.
34
Guess What I’m Thinking! There should be no legitimate argument for any
distractor.
Make sure the key is universally accepted, not just how things are done at your institution.
It’s a good idea to have a specific reference supporting the item.
35
Guess What I’m Thinking!Which of the following is appropriate for sweeping a floor?
A. broomB. vacuumC. mopD. duster
36
Guess What I’m Thinking!A nurse is caring for a postoperative client. Which of the following findings should indicate to the nurse that the client has developed a deep vein thrombosis (DVT)?
A. Positive Homan’s sign of affected extremityB. Tenderness, warmth, erythema along affected veinC. Lower extremity muscle pain occurring with walking
and relieved with restD. Aching lower extremities after prolonged standing and
relieved with walking
37
Guess What I’m Thinking!Healthcare instructors teaching adult learners typically
A. have to work individually with students. B. are best when they practice in the field before teaching.C. have a tendency to be less demanding of
adults than children. D. do not have to be available to students
outside of class hours.
38
Guess What I’m Thinking! v.
Priority Setting Items “Which of the following steps should the nurse perform
first?”
“Which of the following actions is most appropriate for the nurse to take at this time?”
“Which of the following motions is most likely to cause injury?”
39
Trivial Pursuit
Remember that there are a limited number of items on the test. Is it worth spending one of them to ask about the topic?
Can students reasonably be expected to know about the topic?
40
Trivial Pursuit
A client goes to his primary care provider with reports of abdominal pain, steatorrhea, diarrhea and weight loss. He undergoes a barium enema to confirm a diagnosis of ulcerative colitis. When performing a focused assessment of the gastrointestinal system the nurse reviews the results of the barium enema. Which of the following findings should the nurse expect to find?
A. Cobblestone appearance of the intestinal mucosaB. Small outpouchings of the mucosaC. Intestinal mucosa that is pale and thinD. Shortening of the intestine
41
Don’t Preach, Don’t Teach A test should not be used to educate or to
emphasize issues of importance to the students.
Preaching and teaching within an item adds reading comprehension to the test, and the students aren’t likely to be paying attention anyway.
Write to assess; don’t try to impart information.
42
Don’t Preach, Don’t Teach A good nurse must be able to help patients process, or gain insight into their behaviors, desires and goals, as well as solutions to issues that bother them. This is often done in “one-to-one” (1:1) sessions. Which of the following is an approved technique for this process?
A. Encourage the patient to ask personal questions in order to build trust
B. Place the patient in restraints until the technician is comfortable with the situation
C. Make the introduction to the patient using a calm, friendly tone
D. Approach the patient in an authoritative, assertive manner
43
Don’t Preach, Don’t Teach A Milwaukee brace is used to treat a 13-year-old with scoliosis. In addition to changes in body image, which of the following can the nurse most likely expect to be a major problem?
A Skin irritationB Acceptance by peersC Interference with normal body growthD Alteration of clothing
44
Avoid Controversy and the Cutting Edge
Nurses deal with sensitive issues as part of their jobs, but avoid culturally or politically touchy subjects unless they are really central to the topic of the item.
Test the field of nursing as it exists now, not how it will be in the future.
• For classroom assessments this rule can be relaxed somewhat.
45
Reading Comprehension v.
Nursing Content If the item reads like a novel, it might be testing
reading comprehension more than nursing.
If students must ferret out the important words from the item, it might be testing reading comprehension more than nursing.
Avoid window dressing. Omit unnecessary text.
46
Reading Comprehension v. Nursing Content
It was a dark and stormy night. A community health nurse is visiting the home of a client whose preschooler has a low-grade fever and an earache. The client states that transportation is a problem and there is no one to care for the three younger children in the home if the preschooler is taken to the doctor. The client states, however, that the child will be taken to the clinic the next day after the other children go to a baby sitter. Which of the following nursing interventions is most appropriate for this situation?
A Call the clinic the next day to make sure the client kept the appointment.
B Ask the client how the preschooler will be cared for until morning.
C Make an appointment for the child, informing the client that the child needs immediate medical attention.
D Notify the clinic that the client will be bringing in a sick child.47
Definition Questions
Correctly responding to such items may require little job-related mastery.
Rather than ask students to define a term, make them interpret the term as a part of the larger problem.
48
Definition QuestionsA nurse is caring for a client with a low potassium level. The nurse correctly identifies that the client has
A. hyperkalemia.
B. hypokalemia.
C. hypercalcemia.
D. hypocalcemia.
49
Definition Questions
A nurse is caring for a client with a potassium level of 3.2 mEq/L. The nurse should monitor the client for which of the following?
A. Decreased urinary output
B. Irregular heart rate
C. Increased blood pressure
D. Hyperactive bowel sounds
50
Definition Questions
The client has always been a very active senior citizen. She has enjoyed taking art classes and dance classes at the local high school and has recently completed an associate’s degree at the community college. Lately, the client prefers to stay home and rarely socializes. Decreased socialization is a sign of
A. depression.
B. sadness.
C. hedonia.
D. schizophrenia.
51
The Three Most Important Item Writing
Things To Do
Be clear and brief in the stem.
Write at a nursing student level.
Write plausible, yet inarguably wrong
distractors.
52
Be Clear and Brief in the Stem Don’t play word games that leave students
guessing at what you’re really asking about.
The difficulty of the item should not lie in the stem.
Get to the point.
53
Write at a Student Level
Keep in mind that these items are for nursing students, not specialty nurses with years of experience.
Stay at an 8th – 10th grade reading comprehension level.
54
Write Plausible, Yet Inarguably Wrong Distractors
Every wrong answer should fool at least 3% of the students.
Distractors should fool only the students who don’t know the nursing content.
Distractors should be written with…
55
…an utterly ruthless attitude toward students who don’t
know the content.
56
Write Plausible, Yet Inarguably Wrong Distractors
As you write the item, consider the misconceptions your students have about the content and use those misconceptions against them.
Use technical/medical terminology they may have heard but are not certain about.
Use friendly, positive sounding language in the distractors.
Think like a weak student as you write.57
Write Plausible, Yet Inarguably Wrong Distractors
Remember we are talking about a weak nursing student, not a person with no medical knowledge.
Make sure there is a good reason a weak student would pick each distractor. “Because it’s wrong” is not a good enough reason.
No throw away distractors!
Distractors should sound like the truth without getting too close to the truth.
58
Write Plausible, Yet Inarguably Wrong Distractors
When conducting linear anchor block test equating, the items in the anchor block should
A. be out of proportion to the test specifications.B. perform poorly compared to the rest of the test.C. match the statistical properties of the base form.D. have extremely low difficulty values.
59
Write Plausible, Yet Inarguably Wrong Distractors
A toddler is being treated by his pediatrician for sun burn and heat rash. The pediatrician explains to the parent how to prevent these conditions from worsening. Which of the following statements by the parent indicates that she understands the explanation?
A. “I will apply sunscreen to my child 30 minutes prior to sun exposure to prevent the heat rash from spreading.”
60
Write Plausible, Yet Inarguably Wrong Distractors
Which of the following is the most important trait of a successful healthcare instructor?
A. Have a passion for teaching.B. Be a subject matter expert.C. Have interest in improving your teaching skills.D. Use PowerPoint well.
61
Write Plausible, Yet Inarguably Wrong Distractors
The client’s wound is not healing. Which of the following assessments indicate a need for intervention?
A Client gave up smoking two years ago B Diet includes meat and a variety of vegetables C Client is 72 years old D Meds include daily corticosteroids
62
Write Plausible, Yet Inarguably Wrong Distractors
A nurse is caring for a child with an arm cast. Which of the following would indicate a need for immediate intervention?
A. the child’s hands are cool
63
Best Item EverA client diagnosed with bacterial pharyngitis has been prescribed an antibiotic by the primary care provider. Which of the following should be included when providing teaching regarding antibiotic therapy?
A. “It is common for an antibiotic to result in nausea or vomiting.”
B. “Unused antibiotics may be stored at proper temperature for use in the future.”
C. “Take the antibiotic until your sore throat is relieved.”D. “Infecting others is possible until antibiotic therapy
is complete.”
64
Finer Points of Item Writing Strive for symmetry in the 4 options.
Avoid mutually exclusive options.
Avoid absolute terminology in the distractors.
Avoid the same or similar language in the stem and key.
Write options that are independent.
65
Finer Points of Item Writing Avoid creating subsets in the options.
Don’t use “all of the above,” “none of the above,” or anything similar
Avoid undirected stems.
Avoid negatively worded stems.
Avoid options with multiple elements and partially correct answers
66
Symmetry in the 4 Options
Options should be parallel in length and form.
All four options should be independent or in pairs.
67
Symmetry in the 4 Options
A. apple
B. orange
C. banana
D. hammer
68
Symmetry in the 4 Options
A Pulmonary edema
B Cardiac tamponade
C Tracheal collapse
D Pneumothorax
69
Symmetry in the 4 Options
A. apple
B. orange
C. chair
D. hammer
70
Symmetry in the 4 Options
A WBC count increases to 3,000/mm3
B Neutrophils (Bands) increase to 8%
C Rising antibody titer
D WBC count decreases to 9,000/mm3
71
Symmetry in the 4 OptionsA Decreased luteinizing hormone and serum
phosphate
B Elevated testosterone levels and an increase in urinary WBCs
C Decreased hemoglobin and hematocrit
D Elevated PSA and acid phosphatase
72
Avoid Mutually Exclusive Options
A Lower extremity rigidity with upper level flaccidity
B Flaccid paralysis below the level of the injury
C Loss of all sensation and motor activity in upper and lower extremities
D Spastic paralysis below the level of the injury
73
Avoid Absolute Terminology in the Distractors
Why does the sigla frequently overfesk the
trelsum?
A All siglas are mellious.B Siglas are always votial.C The trelsum is usually tarious.D No trelsa are directly feskable.
74
Avoid the Same or Similar Language in Stem and KeyA client is admitted to the hospital for a cholecystectomy. Prior tosurgery, the nurse reviews the lab results and notes that the potassium level is 6.9 mEq/L. Upon assessment of the client, the nurse finds that the client is resting comfortably and is alert and oriented. Which of the following should be the priority nursing action?
A Check the results of a repeat potassium level prior to surgery.B Confirm the client's baseline with him as this level might be
normal for him.C Call the lab and report the error in the level.D Cancel the surgery because of the risk of cardiac arrhythmia.
75
Write Options That Are Independent
First aid for a shock victim should include which of the following?
A. Lowering the body temperatureB. Rapidly cooling the skinC. Elevating the legsD. Orally rehydrating
76
Avoid Creating SubsetsA. Vital signs
B. Blood pressure
C. Heart rate
D. Skin turgor
77
Don’t Use All of the Above, None of the Above, or
Anything Similar
“All of the above” can be answered with partial knowledge.
“None of the above” or “A and B” type items tend to perform poorly.
NCLEX® doesn’t use these item types.
78
Avoid Undirected Stems The candidates should know what is being
asked for without having to read the options. Be specific.
The Trumpeter Swan’s bill
How is the Trumpter Swan’s bill specially adapted for feeding in water?
79
Avoid Undirected Stems
A large Q-angle and/or delayed activation of the vastus medialis oblique A large Q-angle and/or delayed activation of the vastus medialis oblique would likely result in
Stems don’t have to end with a question mark, but it should still be clear what is being asked without reading the options.
80
Avoid Negatively Worded Stems
e.g., “which of the following are not,” “all of the following except”
These tend to perform poorly when otherwise high ability students miss the negative word.
81
Avoid Negatively Worded Stems
Which of the following is NOT a sign and/or symptom of alcohol withdrawal?
A. Nausea/vomitingB. Hand tremorsC. Decreased blood pressure D. Sweating
82
Avoid Negatively Worded Stems
Which of the following is a sign and/or symptom of alcohol withdrawal?
A BradycardiaB Hand tremorsC Decreased blood pressure D Fever
83
Avoid Negatively Worded Stems
There are of course times when it is important for the candidate to know what is wrong as opposed to what is right. This is okay if the stem is phrased positively…
84
Avoid Negatively Worded Stems
“Which of the following are contraindicated?”
“Which of the following statements by the client indicate a need for further teaching?”
“Which of the following are outside of the PN scope of practice?”
“Which of the following are most likely to cause injury?”
85
Avoid Options with Multiple Elements and
Partially Correct OptionsSelect the appropriate condition for each drug in the same correct sequence: metformin, Prevacid, Singulair.
A. diabetes, hypertension, asthmaB. hyperlipidemia, gastroesophageal reflux disease
(GERD), chronic obstructive pulmonary disease (COPD)C. hypertension, asthma, gastroesophageal reflux disease
(GERD)D. diabetes, peptic disease (PUD), asthma
Items such as this mix reading comprehension and common reasoning skills with job ability.
86
Avoid Options with Multiple Elements and
Partially Correct OptionsA nurse is caring for a client with a suspected myocardial infarction. Which of the following assessment findings should the nurse expect?
A. Bradycardia, dyspnea and dizzinessB. Tachycardia, dyspnea and
palpitationsC. Nausea, vomiting, and warm, dry
skinD. Dyspnea, dizziness and hyperactivity
87
Don’t Lose the Forest for the Trees
Spend most of your time and energy worrying about
being clear and brief in the stem. writing at a nursing student level. writing plausible, yet inarguably wrong
distractors.
Don’t get so obsessive over the finer points that you forget that an item is supposed to differentiate between students of differing knowledge/ability levels.
88
NCLEX Terminology and Item Types
89
NCLEX® Terminology/StyleUse Don’t Use
Client Patient
Client reports Client complains of
Prescription/Prescribe Order
Primary care provider or provider Physician, Nurse Practitioner
Active tense: “should do” What would you do?
A nurse, A client Personal pronouns – he, she
A client with diabetesA client diagnosed with asthma
A diabetic clientAn asthma client
generic name (Trade Name) Dilantin
Supplies: indwelling urinary catheter Foley catheter
Assistive Personnel Certified nursing assistant, patient care technician, unlicensed assistive personnel
90
NCLEX® Terminology/Style Write in third person style
Avoid “you” and “your.” Don’t give anyone a name.
Avoid or limit items that focus on medical condition. deal with pathophysiology/etiology. are about definitions/terminology.
Label all values Avoid numbers that are very close to normal
or very far out of range.91
NCLEX® Item Formats Multiple Choice (standard four option) Hot Spot Fill-in-the-Blank Multiple Response Drag-And-Drop Chart Exhibit
92
Hot Spot ItemNCLEX® Task: Perform emergency care
procedures
Stem:
A nurse is caring for an 11-month-old
infant who is hospitalized. The nurse
Finds the infant unresponsive and
Initiates cardiopulmonary resuscitation
(CPR). Identify the location where the nurse
should palpate for a pulse.
93
Fill-in-the-BlankNCLEX ® Task: Perform calculations needed for medication administration.
Stem:
A 2-year-old child who weighs 22 lb is prescribed phenytoin (Dilantin) 5 mg/kg/day in two divided doses. The nurse should calculate an individual dose to be ___ mg. (Round answer to whole number.)
94
Multiple ResponseNCLEX® Task: Evaluate the client/family/significant other’s ability to care for the newborn.
Stem:
A nurse is caring for a client who is a first time mother. Which of the following client statements indicates to the nurse that the client understands care of her neonate? (Select all that apply.)
95
Multiple Response OptionsA. “I will breastfeed my baby on a strict schedule of every 3 hr.”
B. “I will wait to give my baby a tub bath until the umbilical cord has fallen off.”
C. “I will be sure to have my baby sleep on her back.”
D. “I will put a hat on my baby when I take her outside on cold days.”
E. “I will use the bulb syringe first in her nose and then in her mouth.”
Key: B, C, D
96
Drag and DropNCLEX® Task: Provide client/family/significant other with methods to prevent complications associated with activity level/diagnosed illness/disease.
Stem:
A nurse is providing teaching to a school-age child who has just been diagnosed with diabetes mellitus. The nurse provides instructions to the child on how to handle suspected hypoglycemia. (Place the following steps in the appropriate order.)
97
Drag and Drop OptionsA nurse is providing teaching to a school-age child who has just been diagnosed with diabetes mellitus. The nurse provides instructions to the child on how to handle suspected hypoglycemia. Place the following steps in the appropriate order.
A. Eat a snack of 15 g of carbohydrates.
B. Recheck blood glucose level by fingerstick.
C. Stop activity and sit down.
D. Wait 15 min.
E. Check blood glucose level by fingerstick.
1. C 2. E 3. A 4. D 5. B
98
Chart ReviewNCLEX® Task: Apply knowledge of the client’s pathophysiology to laboratory values findings. Stem:A client is admitted with uncontrolled diabetes mellitus. Based on the client information provided, which of the following actions should the nurse take? (Click on the “Exhibit” for additional client information. There are three tabs that contain separate categories of data.)
99
Chart Review OptionsLaboratory Results
Glucose: 347 mg/dL
Dietary Order
Diet as tolerated
A. Contact the client’s primary care provider.B. Administer 6 units of Regular insulin. (correct)C. Provide the client with a high-protein snack. D. Apply oxygen at 2 L/min per nasal cannula.
Medication Administration Record Sliding Scale Insulin Blood Glucose 301-350 give 6 U. Regular Insulin
100
Are you ready to Beat a Cheat (the test wise student that is)?
101
