Principles of Effective Item Writingenglishlanguageartsresource.weebly.com/uploads/1/6/5/9/16598434/... · Useful when there are multiple possible distracters A relatively large number

Principles of Effective Item

Writing

February 2012

University of Lethbridge

Item Development Principles

Well constructed multiple-choice items can test a

range of knowledge and skills involving higher

levels of thinking. They need not be limited to

recall items testing student knowledge of concrete

information. As well, multiple-choice examination

items can provide useful diagnostic information

about individual achievement as all students

experience the same task.

Guidelines for Building MC Items

Use the Program of Studies as your reference. Focus on a

specific outcome prior to writing the item.

Aim to write items that cover a range of difficulty.

Ensure that the item has both curricular and instructional

validity.

Ensure that the item tests something worthwhile.

Multiple-Choice Item Types

There are two types of multiple-choice items:

Discrete Item – a stem is provided and the student

seeks to select the correct response from the

alternatives provided.

Source-Based Item – an item that derives from

the consideration of details in a specific source.

Sources include written texts, graphs, charts and

photographs.

Anatomy of a Multiple-Choice Question

Which of the following celestial bodies emits

light?

A. Star

B. Moon

C. Planet

D. Comet

These are the “alternatives”,

“choices”, or “options”.

This is the “keyed response” or correct choice.

These are the “distracters”,

(alternatives that are incorrect).

This is the stem.

Building Multiple-Choice Questions:

The Stem

A stem may be worded in one of two different ways.

As a question:

What is the area of a rectangle that has a length of 3 cm and a width

of 1 cm?

As a statement:

The area of a rectangle that has a length of 3 cm and a width of 1 cm

is

The stem of the question must be focused. A good stem

provides enough context to enable the student to visualize

a correct response prior to reviewing the alternatives.


Appropriate Diction

Use vocabulary that is appropriate to the reading

level of the students being tested and that is

consistent with the expectations of the

curriculum.

Avoid ambiguous word choices. Be sure to avoid

colloquial expressions, idioms, slang and jargon

that may be an obstacle to the students’

understanding of the question or the alternatives.


Alternatives & Distracters

Distracters should be logical, for example, based on common misconceptions that students may have.

Directing words must be bolded (most directly, mainly, etc)

Alternatives should all be alike in kind (length, complexity, grammatical structure).

Be sure that the keyed answer does not stand out from the alternatives because it is more detailed or more qualified than the distracters.

Avoid creating a question that implies that wrong information is correct.

Make sure that a question contains only one correct answer.



Avoid absolutes such as "all" and "every" because

there has to be only one exception to make the

alternative wrong.

Avoid “all of the above” and “none of the above” as

distracters.



Quality Multiple-Choice Questions

Which of the following

sentences contains an example

of a correctly used apostrophe?

A. I di’dnt do my chores today.

B. I should’nt talk during class.

C. I couldn’t find my homework.

D. I woul’dnt want to go to the zoo.

Which of the following

words contains a

correctly used apostrophe?

A. should’nt

B. woul’dnt

C. couldn’t

D. di’dnt

SO: recognize various uses of apostrophes, and use them appropriately in own

writing (4.2c).

IO: Students will use apostrophes correctly in the their short stories. (applying

taxonomy level)


Alternative Order

Alternatives must be pyramided, which means they are

arranged by length (shortest to longest, or longest to

shortest)

The exception to this rule is when the alternatives contain

direct quotations from a source. In this case, the

alternatives must be placed in the order that the quotations

appear in the source


Not Pyramided

According to democratic theory, active interest groups

and a free press ensure that

A. bureaucratic decisions are given greater credibility

B. minority rights do not become entrenched C. elections reflect the wishes of mainstream political parties D. tyranny of the majority is prevented


Pyramided



C. bureaucratic decisions are given greater credibility

B. minority rights do not become entrenched

D. elections reflect the wishes of mainstream political parties

A. tyranny of the majority is prevented


Reverse Pyramid



B. bureaucratic decisions are given greater credibility

C. minority rights do not become entrenched

A. elections reflect the wishes of mainstream political parties

D. tyranny of the majority is prevented


Quotation Order

The lack of communication between the speaker and the companion

is most clearly suggested in the quotation

A. “My shadow darkens your face” (line 8)

B. “removed / from my words, the touch / of my fingers” (lines 9-11)

C. “What caves do you move through?” (line 12)

D. “all you believe / falls away” (lines 17-18)

What other characteristics of good multiple

choice items does this question demonstrate?


Your Turn

The clocks in the house are

A. in the hall

B. in the living room

C. in the kitchen

D. ticking quietly

What errors do you see? How would

you improve this question?


Pitfalls

These data could most appropriately be modeled using

an

A. exponential function

B. logarithmic function

C. sinusoidal function

D. quadratic function


Pitfalls

Another example:

In lines 46 to 48, the repetition of the word upright mainly

emphasizes Fine Man’s

A. good posture

B. sense of tenacity

C.high moral values

D.persevering attitude


Pitfalls

Which of the following Shakespearean plays is an example of the author’s mastery of the tragic form?

A. Macbeth

B. The Misanthrope

D. The Massacre at Paris C. A Man for All Seasons


Pitfalls

The quintessential woodworker would have which

of the following implements in the domicile in

which he pursues his noble vocation?

A. Knife and fork

B. Inclined plane

C. Chainsaw

D. Veritas maple woodbench with attached vise

The Strengths of Multiple-Choice Items

Strengths:

Can help identify student misconceptions and errors

(helpful for providing diagnostic information for both individual

test-takers and for a group)

Useful when there are multiple possible distracters

A relatively large number of items can be answered in a

typical testing period

Scoring is easy, objective and reliable

Possible to test a range of knowledge and skills including

higher level thinking

Eliminate bluffing and writing skill influences

Guessing becomes a factor

Difficult to write items that are above the knowledge

taxonomy level that are free of ambiguity

Difficult and time consuming to create

Poorly constructed items provide unreliable results and

invalid inferences, creating frustration and confusion

Potential for superficial or recall items that test trivia and

test-wiseness

The Limitations of Multiple-Choice Items

Limitations:

Multiple-Choice Item

Quality Checklist

An item is technically

sound when…

the item is free of cues

to the answer

the stem is focused on a

single, meaningful problem

key words in the stem are

emphasized as needed

distracters are all plausible

alternatives are in an order

that is logical and easily

understood

the item has correct spelling

and grammar

An item is equitable when

the item…

is free from cultural and sexual

bias

is free of irrelevant material

is stated in appropriate and

clear language

is free from cultural references

that would not be familiar to all

students

Correct choice:

• Possesses required

information or skill

• Uses partial information

to get the answer

• Uses cues in the item

and/or test

• Makes a blind, lucky

guess

Incorrect choice:

• Lacks required

information or skill

• Uses partial

information or

misinformation that

favours a distracter

• Makes a blind,

unlucky guess

Common Reasons for Selecting Correct

or Incorrect Multiple-Choice Answers


Test-wiseness

Canadian Knowledge Quiz


Test-wiseness

Test-wiseness can be a huge problem!

– Verbal associations that give away the answer

– Grammatical inconsistencies that eliminate wrong answers

– Specific determiners that make certain answers probable and

others improbable

– Length or location of correct answers

– Material in one item that aids in answering another item

– Implausible alternatives given the context of the question

What Constitutes Quality Source Material?

Quality source material • must relate to the learning outcomes.

• must be novel (new to the students).

• must be at the appropriate length and reading level for the subject and grade level.

• stimulates thought about an issue or topic.

• be authentic and not directly taken from the prescribed resources for the course.

• provide adequate and appropriate context for questions.

• can come from a published source (beware of copyright issues).

• can be created by you.

Be Sure the Source is Required!

Where did the United Nations send its first

peacekeeping force?

A. Hungary

B. the West Bank

C. West Germany

D. the Suez Canal

Be Sure the Source is Required!

The newspaper headline identifies an

effort to resolve an international crisis

situation through the use of

A. collective security

B. mutual deterrence

C. appeasement

D. detente

What Constitutes a Quality Source-

Based Multiple-Choice Item?

Test the knowledge and skills you desire, NOT just the information presented explicitly in the source • The source MUST be required to answer the items

• When choosing or creating your own source, be careful not to include extraneous information

• The item must be at the “Applying” or above taxonomical level

• Follow the same guidelines for discrete multiple-choice item construction

Highlighted Sentences from Jenny’s Short Story:

The Day the Dogs Howled All Night

Sentence #1 I couldn’t believe the sound that came from all the howling dogs.

Sentence #2 I wanted to cover my ear’s but the sound was too loud.

Sentence #3 Ralph, my brown little terrier, started to bark just like my friend

Jessie’s dog.

Sentence #4 The dog didn’t hear me yelling but ran to the gate and escaped.

Use the following information to answer question 1.

Which of the sentences from Jenny’s short story contain an

apostrophe error?

A. Sentence #1

B. Sentence #2

C. Sentence #3

D. Sentence #4

Movie Admission Prices

Year Adult Ticket

2001 $12.25

2002 $12.75

2003 $13.25

2004 $13.75

6. If the pattern shown in the table above continues, then in which

year will the price of one adult ticket have increased to $15.25?

A. 2007

B. 2008

C. 2009

D. 2010

Use the following table to answer question 6.

Strengths and Limitations of

Source-Based MC Items

Strengths An efficient means of

measuring the interpretation of information in various forms (e.g., written, charts, graphs, maps, pictures).

Can measure complex learning outcomes.

Introductory material provides a common basis for responding.

Scoring is easy, objective, and reliable.

Limitations Difficult to construct quality

items using real-world sources.

Written material is highly dependent on reading skills.

Highly subject to cueing.

Ineffective in measuring the ability to originate, organize, and express ideas.

Source-Based Multiple Choice

Quality Checklist An item is technically

sound when…

the item is free of cues to the answer

the item uses source material that is

familiar but new to the student

the source material is an appropriate length

and at an appropriate reading level

the source material is properly cited

the items cannot be answered without the

source material

the items are at the applied taxonomy level

or above

the item has correct spelling and grammar

the items meet the technically sound

checklist for multiple choice items

An item is equitable when

the item…

is free from cultural and sexual

bias



clear language

is free from cultural references

that would not be familiar to all

students

Consider the Benefits of Open-Ended

Constructed Response Items

The following question was given to two hundred applicants for a job…

You are driving along on a stormy night in your little two-seater car. Passing a bus stop you see three people waiting:

A. an elderly woman clearly in need of immediate medical attention

B. an old friend who once saved your life

C. the perfect mate you have dreamed of your entire life

Who would you pick up? Share your choices with a partner.

Given the Following Response…

“I would give the car keys to my friend and let him take

the elderly woman to the hospital. Then I would stay

behind and wait for the bus with the person of my

dreams.”

…What does this response tell us about a benefit of

open-ended responses?

Structure of a Restricted-Constructed

Response Item

During an exam, it is obvious that the person

next to you is copying from the person sitting

in the row in front of you. List 3 reasons why it

is appropriate to report this behaviour.

Answers may be in point form or up to three

sentences in length. (3 marks)

A context is provided

Guiding words limit the

response and define the

subject matter

Scoring information is provided

Directions “bound” the

requirements for the

answer

A Quality Restricted-Constructed

Response Item

Choose the guiding words

carefully so writers create

the responses for which

you hoped.

Compose clear

directions

Point to an answer (few

sentences max)

Provide a meaningful

context

Explain

Predict

Show all calculations

Refer to specific details

Create a diagram


Response Item

Here are the directions and the context for the response item

Read the information below and answer the questions that follow. Marks

for each question are provided.

You have finally saved enough money to purchase your

first cell phone. This is an important purchase, so you

want to make sure you make an informed decision. You

decide to conduct a valid and reliable survey and have

placed your survey data in the following chart.


Response Item

Type of Cell

Phone

Number of Users Satisfaction

Rating (*****)

TouchIt 115 ****

Neptune S3 61 *****

TalkaLot 42 ***

SwishZ 34 ****

Samurai1000 18 **

1. Based on the data collected, formulate a survey question that

participants of this survey would have been asked. (1 mark)

2. Carefully analyzing the data provided, identify TWO valid conclusions that

can be made. Answer using complete sentences or point form. (2 marks)

Restricted-Constructed Response:

Your Turn

We have been studying the importance of the carbon

cycle and how it works. Briefly describe why we need

to know about it, how it works, and its location. Your

answer will be assessed using the following criteria:

Explain why understanding the carbon cycle is important

Identify at least 3 major carbon reservoirs

Describe at least 6 ways that carbon is transferred from one

place to another

3 points

3 points

6 points


Response Item

On a par four hole, Andy takes a practice swing and accidentally nudges his ball off the tee. He replaces it and hits his drive, hooking it into the rough. When he finds the ball, he moves it to a small clump of grass so he can get the club under it. He drops his next shot into the sand trap at the front of the green. He lifts the ball out of its crater and moves it to a level bit of sand. His shot lands two feet in front of the pin and rolls into the cup. Andy records a birdie for the hole.

Identify two violations of the rules of golf Andy has committed on this hole. (2 marks)

If Andy had played by the rules, what score should Andy have recorded on this hole? (1 mark)

Read the situation below and answer the questions that follow in point form. Marks for each question are indicated.

A Restricted-Constructed

Response Item

Answer the questions that follow using complete sentences.

A Restricted-Constructed

Response Item

Describe three ways the artist uses colour in this painting. (3 marks)

Name two additional artistic techniques the artist obviously demonstrates a

loving fondness for in this painting. (2 marks)

Identify the likely time period in which this artist worked. (3 marks)

Describe two characteristics of the period in which this artist worked.

(2 marks)

Bonus: Identify the artist of this painting. (1 mark)


Response Item

Describe three ways the artist uses colour in this painting. (3 marks)

Name two additional artistic techniques the artist demonstrates in this painting.

(2 marks)

Identify the likely time period in which this artist worked. (1 mark)

Describe two characteristics of the art in the time period in which this artist

worked. (2 marks)

Each student who correctly identifies the artist who painted this work will get

to choose a treat from The Jar.

Restricted-Constructed Response

Strengths Guessing is less likely than in

selection-type items.

A range of outcomes can be

measured.

Directions impose boundaries

and limitations such that the

response is limited in scope

Easier to score than

extended items

Limitations Scope of answer is limited

There is a need for

establishing and sharing

scoring criteria ahead of time.

Less freedom of expression

than extended items

Restricted-Constructed Response

Quality Checklist

An item is technically sound

when…

the item contains appropriate guiding words

the item presents a context that is

meaningful and engaging for students

the item points to the answer so

that it is bounded

the item is above the “Remembering” taxonomical level

the item has correct spelling and

grammar

An item is equitable when the

item…

is free from cultural and

sexual bias



clear language

is free from cultural

references that would not be

familiar to all students

How to contact us Rite line: 310-0000 then (780) 427-0010

Director of Diploma Programs - Assessment Sector – [email protected]

English Language Arts 30-2 – [email protected] (Team Leader)

– [email protected] (Examiner)

mailto:[email protected]

Documents

Principles of Effective Item Writingenglishlanguageartsresource.weebly.com/uploads/1/6/5/9/16598434/... · Useful when there are multiple possible distracters A relatively large number