How to Avoid Common Pitfalls in Writing Math Tests Marcia Kastner, Ph.D. Former MCAS Math Assessment Lead Massachusetts Department of Elementary and Secondary

How to Avoid Common Pitfallsin Writing Math Tests

Marcia Kastner, Ph.D.

Former MCAS Math Assessment LeadMassachusetts Department of Elementary and Secondary Education

Copyright © 2011 by Marcia Kastner

Outline

• Why are tests important?

• Definition of terms

• Assumptions

• Examples of pitfalls in math questions

• How to fix the flawed questions

• Your chance to “Test the Test”

2Copyright © 2011 by Marcia Kastner

Why Are Tests Important?

• Used for accountability

• Provide information on what students know – and don’t know!

• For targeting instruction (from deconstructing test results)

• Improve education


Definition of Terms

• Valid test question: measures what it was designed to measure

• Pitfall: flaw in a test question that prevents it from being valid

• Parts of a multiple-choice question:– Options: answer choices– Key: correct option– Distractors: incorrect options


Assumptions

Unless otherwise specified,

• No partial credit

• No calculators allowed


Math Testing Book


TESTING THE TEST

How to Recognize When Math Tests Are FlawedHow to Fix Them

Why We Should Care

(free books available to attendees after the presentation)

Presentation is based on pitfalls described in my book:

MarciaKastner.com

PITFALL:Right Answer for Wrong Reason (RAWR)

Example 1:

What is the median of the numbers below?

6, 2, 3, 1, 3

(Answer: 3)

( Not valid because:mean = 3; mode = 3; middle number in unordered list = 3)

(median = middle number in ordered list: 1, 2, 3, 3, 6)


How to Fix Example 1

Example 1 Revised:


7, 2, 6, 2, 3

(Answer: 3)

Change some of the numbers.

(mean = 4; mode = 2; middle number in unordered list = 6)

(ordered list: 2, 2, 3, 6, 7)


A Second Example of RAWR

Example 2:

What is the area, in square inches, of the square shown below?

4

4(Answer: 16)

(Not valid because:area = 42 = 16 square inches; perimeter = 4x4 = 16 inches)



Change the square’s side lengths.

Example 2 Revised:

What is the area, in square inches, of the square shown below?

5

5(Answer: 25)

(area = 25 square inches; perimeter = 20 inches)10Copyright © 2011 by Marcia Kastner

PITFALLS: Too Many Steps, Too Many Concepts, Option Sticks Out


Example 3: (from New Eng. Common Assessment Program [NECAP], 2007)

Renata is a sales representative for a printer company. She sells two models of printers – Model P and Model Q.

• Last month she sold a total of 120 printers.• The ratio of Model P printers sold to Model Q printers sold was 3:5.

If Renata is paid a $25 commission for every Model P printer sold and a $20 commission for every Model Q printer sold, what was her total commission last month?

A. $1480B. $2475C. $2625 [key: 25(45) + 20(75)]D. $2760

(Not valid because:requires approx. 14 steps)

Pitfalls in Example 3

• Too many steps: approx. 14 steps

• Too many concepts:– Define variables– Model the bullet points as linear equations– Model the word problem as a system of linear equations– Solve the system of linear equations– Compute total commission

• One option sticks out: option A is different

Copyright © 2011 by Marcia Kastner 12


• Break up into multiple parts.

• Give partial credit for correct work shown.

Possible parts:

a) [write first bullet as equation]b) [write second bullet as equation]c) [determine amount of each model]d) [calculate total commission]


PITFALLS: RAWR, Not Testing Common Mistakes

Example 4: (from New York State Testing Program, 2010)

Simplify the expression below.

72 – 9 + 13

A. 37B. 39C. 41D. 43

(Answer: C)

14Copyright 2011 by Marcia Kastner


72 – 9 + 13

RAWR: 13 = 1 = 10 = 11 = 12 = etc.

Distractor logic:

A. 37 [49 – (9 + 3): PEMDAS and cube errors]

B. 39 [49 – (9 + 1): PEMDAS error]C. 41* [49 – 9 + 1: key]D. 43 [49 – 9 + 3: cube error]Common mistakes not tested by distractors:

1. square error: thinking 72 = 7 x 22. perform operations left-to-right



Example 4 Revised:

What is the value of the expression below?

72 – 9 + 23

A. 11 [14 – 9 + 6: exponent errors]B. 21 [(14 – 9 + 2) x 3: PEMDAS & exponent

errors]C. 32 [49 – (9 + 8): PEMDAS error]D. 48 [49 – 9 + 8: key]

(Answer: D)

Change the expression and distractors.

(testing common mistakes)


PITFALL: Irrelevant Information


Example 5: (from Massachusetts Comprehensive Assessment System [MCAS], 2001)

The mean salary of the ten clerks at the Corner Shop was $8.50 per hour. One of the clerks, who had been making $9.50 per hour, was given a raise of $1.00 per hour. What is the new mean salary of the ten clerks?

A. $8.50 B. $8.60 [key: {10(8.50) + 1}/10]C. $8.80 D. $9.00

(Answer: B)

(Not valid because:the $9.50 per hour is irrelevant towhat is being tested)


Copyright © 2011 by Marcia Kastner 18

Example 5 Revised:

The mean salary of the ten clerks at the Corner Shop was $8.50 per hour. One of the clerks was given a raise of $1.00 per hour. What is the new mean salary of the ten clerks?

A. $8.50 B. $8.60C. $8.80 D. $9.00

(Answer: B)

Remove the irrelevant information.

(Irrelevant information is sometimes okay, e.g., area of a triangle.)

PITFALL: Implausible Distractors

Example 6 (same as Example 1 Revised but as a multiple-choice question):


7, 2, 6, 2, 3

A. 3B. 6C. 9D. 12

(Answer: A)(Not valid because:cannot determine how errors made)



Example 6 Revised:


7, 2, 6, 2, 3

A. 2 [mode]B. 3 [key: median]C. 4 [mean]D. 6 [middle number of unordered list]

(Answer: B)

Change the distractors.

(can determine how errors made)20Copyright © 2011 by Marcia Kastner

Your Chance to TEST THE TEST

Find the pitfalls in the following examples.


What Is the Pitfall in Example 7?

Example 7:

What is the value of n2 when n = 2?

(Answer: 4)

(Pitfall: RAWR

Not valid because: When n = 2, n2 = n x 2.)

(So be careful when asking for the area of a circle!Don’t use a radius = 2.)



Change the value of n.

Example 7 Revised:

What is the value of n2 when n = 3?

(Answer: 9)


What Is the Pitfall in Example 8?

Calculators Allowed: All types

Example 8 (same as Example 1 Revised except that a calculator is allowed):


7, 2, 6, 2, 3

(Answer: 3)

(Pitfall: Calculator, not student, solves the question.Not valid because: graphing calculators determine median.)



Calculators Allowed: All types except graphing calculators, OR no calculators allowed

Example 8 Revised (same as Example 1 Revised):


7, 2, 6, 2, 3

(Answer: 3)

Do not allow graphing calculators.


What Are the Pitfalls in Example 9?

Example 9: (from New England Common Assessment Program [NECAP], 2010)

Courtney walks three laps around a ¼-mile track. How many feet does she walk? [1 mi. = 5280 ft.]

A. 440 ft. [1/3 x ¼ x 5280]B. 1320 ft. [¼ x 5280]C. 3960 ft. [key: 3/4 x 5280]D. 7040 ft. [4/3 x 5280]

(Answer: C)



• Confusing & imprecise language: “laps around a ¼-mile track”

(Particularly disadvantages ELL and SPED students)

• Biased context: track

• Too many concepts– Fractions– Unit conversion



• Break up into multiple parts & allow partial credit.

• Add diagram showing oval track.

• Show how distance around is ¼ mile.

• Eliminate or explain word “lap.”

• Bold the word “three” or replace it with “3.”


Summary

• Tests must be valid– Scrutinize test questions for pitfalls

– Revise questions to remove pitfalls

• Diagnose students’ incorrect answers to see where they went wrong

• Focus instruction on areas that need improvement


Questions?

(Be sure to pick up a free copy of my book on your way out.)

Documents

How to Avoid Common Pitfalls in Writing Math Tests Marcia Kastner, Ph.D. Former MCAS Math Assessment Lead Massachusetts Department of Elementary and Secondary