Lesson 3 developing a teacher made test

1

Developing a Teacher Made Test

Dr. Carlo Magno

2

Advance Organizer1 The Test Blueprint

Outline of the Test Development Process

Table of Specifications

2 Designing Selected-Response ItemsBinary-choice items

Instructions in Writing Binary Type of Items

Multiple-choice items

Guidelines in Writing Multiple-choice Items

Matching items

Guidelines in Writing Multiple-choice Items

3 Designing Constructed-Response ItemsShort-answer items

Guidelines in Writing Short Answer Items

Essay items

4 Designing Interpretive ExerciseGuidelines in Writing Intepretive Exercise

Examples of Interpretive Exercise

3

Objectives

• 1. Explain the theories and concepts that rationalize the practice of assessment.

• 2. Make a table of specifications of the test items.

• 3. Design pen-and-paper tests that are aligned to the learning intents.

• 4. Justify the advantages and disadvantages of any pen-and-paper test.

• 5. Evaluate the test items according to the guidelines presented.

4

Outline of Test Development Process• 1. Specify the ultimate goals of the education process• 2. Derive from these the goals of the portion of the

system under study• 3. Specify these goals in terms of expected student

behavior. If relevant, specify the acceptance level of successful learning.

• 4. Determine the relative emphasis or importance of various objectives, their content, and their behaviors.

• 5. Select or develop situations that will elicit the desired behavior in the appropriate context or environment, assuming the student has learned it.

• 6. Assemble a sample of such situations that together represent accurately the emphasis on content and behavior previously determined.

5

Outline of Test Development Process

• 7. Provide for the recording of responses in a form that will facilitate scoring but will not so distort the nature of the behavior elicited that it is no longer a true sample or index of the behavior desired.

• 8. Establish scoring criteria and guides to provide objective and unbiased judgment.

• 9. Try out the instrument in preliminary form.• 10. Revise the sample of situations on the basis of

tryout information.• 11. Analyze reliability, validity, and score distribution in

accordance with the projected use of scores.• 12. Develop test norms and a manual, and reproduce

and distribute the test.

6

Test Length

• The test must be of sufficient length to yield reliable scores

• The longer the test, the more the reliable the results

• The test should be valid if it is reliable• For the grade school, one must consider the

stamina and attention span of the pupils• The test should be long enough to be

adequately reliable and short enough to be administered

7

Test Instruction

• It is the function of the test instructions to furnish the learning experiences needed in order to enable each examinee to understand clearly what he is being asked to do?

• Instructions may be oral, a combination of written and oral instruction is probably desirable, except with very young children.

• Clear concise and specific.

8

Test layout• The arrangement of the test items influences the speed

and accuracy of the examinee• Utilize the space available while retaining readability.• Items of the same type should be grouped together• Arrange test items from easiest to most difficult as a

means of reducing test anxiety.• The test should be ordered first by type then by content• Each item should be completed in the column and page

in which it is started.• If the reference material is needed, it should occur on the

same page as the item• If you are using numbers to identify items it is better to

use letters for the options

9

Scoring the test• Use separate answer sheets• Punched key• Overlay key• Strip keyPlight of the student• The teacher should discuss with the class the

content areas and levels of the cognitive domain to be examined

• The discussion should utilize a vocabulary and a level of complexity appropriate to the development level of the student

• Types of test• Examples of test type

10

Table of Specifications

Content Outline No. of items

1. Table of specifications 102. Test and Item characteristics 203. Test layout 54. Test instructions 55. Reproducing the test 56. Test length 57. Scoring the test 5TOTAL 55

One Grid TOS

11

Table of SpecificationsTwo Grid TOSWeight(Time Frame)

ContentOutline

Knowledge30%

Comprehension40%

Application30%

No. of items by content

area

35% 1. Table of specifications 1 4 4 9

30% 2. Test and Item characteristics 2 3 3 8

10% 3. Test layout 1 1 0 2

5% 4. Test instructions 0 1 0 1

5% 5. Reproducing the test 1 0 0 1

5% 6. Test length 1 0 1 2

10% 7. Scoring the test 2 1 0 3

8 10 8 26

The number of items in a cell is computed using the formula:

items ofnumber totalX skill cognitive of percentage timeTotal

timeGivenXitems

12

Classification of test Items• Selected Response

– Binary Choices– Multiple Choice– Matching Type

• Constructed Response “Supply Test”– Short Form answers - identification– Completion – fill in the blanks, cloz test– Essay

• Performance Type– Paper and pencil type– Identification type– Simulation

13

Item Writing Commandments

• Thou shall not produce opaque directions to students regarding how to respond to your instructions (opaque directions)

• Thou shall not employ ambiguous statements in your assessment item (ambiguous statements)

• Thou shall not unintentionally provide students with clues regarding appropriate response (unintended clues)

• Thou shall not employ complex syntax in your assessment item (complex syntax)

• Thou shall not use vocabulary that is more advanced than required (Difficult vocabulary)

14

SHORT ANSWER ITEMS

• 1. Word the item so that the answer is both brief and definite.

• 2. Do not take statements directly from books to use as a basis for short answer items.

• 3. A direct question is generally more acceptable than an incomplete statement.

• 4. The answer to be expressed in numerical units indicate the type of answer wanted.

• 5. Blanks for answers should be equal in length.• 6. Do not use to many blanks.

15

Writing supply items1. Require short, definite, clear-cut, and explicit answers

FAULTY: Earnest Hemingway wrote______

IMPROVED: The Old Man and the Sea was written by _______.

Who wrote The Old man and the Sea?

2. Avoid multimutilated statements

FAULTY: _____ pointed out in ____ the freedom of thought in America was seriously hampered by ___, ____, & __.

IMPROVED: That freedom of thought in America was seriously hampered by social pressures toward conformity was pointed out in 1830 by ______.

16

Writing supply items

3. If several equal answers equal credit should be given to each one.

4. Specify and announce in advance whether scoring will take spelling into account.

5. In testing for comprehension of terms and knowledge of definition, it is often better to supply the term and require a definition than to provide a definition and require the term.

FAULTY: What is the general measurement term describing the consistency with which items in a test measure the same thing?

IMPROVED: Define “internal consistency reliability.”

17

Writing supply items6. It is generally recommended that in completion items

the blanks come at the end of the statement.FAULTY: A (an) ________ is the index obtained by dividing

a mental age score by chronological age and multiplying by 100.

IMPROVED: The index obtained by dividing a mental age score by chronological age and multiplying by 100 is called a (an) ________

7. Minimize the use of textbook expressions and stereotyped language.

FAULTY: The power to declare war is vested in ______IMPROVED: Which national legislative body has the

authority to declare war?

18

Writing supply items8. Specify the terms in which the response is to be given.

FAULTY: Where does the Security Council of the United Nations hold its meeting?

IMPROVED: In what city of the United States does the Security Council of the United Nations hold its meeting?

FAULTY: If a circle has 4-inch diameter, its area is_____

IMPROVED: A circle has 4-inch diameter. Its area in square inches correct to two decimal places, is _____

9. In general, direct questions are preferable to incomplete declarative sentences.

FAULTY: Gold was discovered in California in the year ___

IMPROVED: In what year was gold discovered in California?

19

Writing supply items

10. Avoid extraneous clues to the correct answer

FAULTY: A fraction whose denominator is greater than its numerator is a _____

IMPROVED: Fractions whose denominator are greater than their numerators are called _____

20

ALTERNATIVE RESPONSE ITEM• 1. Avoid broad general statements if they are to be

judged true or false.• 2. Avoid trivial statements.• 3. Avoid the use of negative statements.• 4. Avoid long complex sentences.• 5. Avoid including two ideas in one statement unless

cause and effect relationship are being measured.• 6. If the opinion is being used, attribute it to some

source unless the ability to identify opinion is being specifically measured.

• 7. True statements and false statements should be equal in length.

• 8. The number of true and false statements should be approximately equal.

21

Writing TRUE-FALSE Items

1. Avoid the use of “specific determiners” FAULTY: No picture-no sound in a television set may

indicate a bad 5U4G.IMPROVED: A bad 5U4G tube in a television set will result

in no picture sound.2. Base true-false items upon statements that are

absolutely true or false, without qualifications or exceptions.

FAULTY: World War II was fought in Europe and the Far East.

IMPROVED: The primary combat locations in terms of military personnel during World War II were Europe and the Far East.

22


3. Avoid negative stated items when possible and eliminate all double negatives.

FAULTY: It is not frequently observed that copper turns green as a result of oxidation.

IMPROVED: Copper will turn green upon oxidizing.

4. Use quantitative and precise rather than qualitative language where possible.

FAULTY: Many people voted for Gloria Arroyo in the 2003 Presidential election.

IMPROVED: Gloria Arroyo received more than 60 percent of the popular votes cast in the Presidential election of 2003.

23


5. Avoid stereotypic and textbook statements.

FAULTY: From time to time efforts have been made to explode the notion that there may be a cause-and-effect relationship between arboreal life and primate anatomy.

IMPROVED: There is a known relationship between primate anatomy and arboreal life.

6. Avoid making the true items consistently longer than the false items.

7. Avoid the use of unfamiliar or esoteric language.

FAULTY: According to some peripatetic politicos, the raison d’etre for capital punishment is retribution.

IMPROVED: According to some politicians, justification for the existence of capital punishment can be traced to the Biblical statement, “An eye for an eye.”

24


8. Avoid complex sentences with many dependent clauses.

FAULTY: Jane Austen, an American novelist born in 1790, was a prolific writer and is best known for her novel Pride and Prejudice, which was published in 1820.

IMPROVED: Jane Austen is best known for her novel Pride and prejudice.

9. It is suggested that the crucial elements of an item be placed at the end of the statement.

FAULTY: Oxygen reduction occurs more readily because carbon monoxide combines with hemoglobin faster than oxygen does.

IMPROVED: Carbon monoxide poisoning occurs because carbon monoxide dissolves delicate lung tissue.

25

Writing Matching Type Test

1. Matching Exercises should be complete on a single page.

2. Use response categories that are related but mutually exclusive.

3. Keep the number of stimuli relatively small (10-15), and let the number of possible responses exceed the number of stimuli by two or three.

4. The direction should clearly specify the basis for matching stimuli and responses.

5. Keep the statements in the response column short and list them in some logical order

26

FAULTY: Match List A with List B. You will be given one point for each correct match.

List A List B

a. cotton gin a. Eli Whitney

b. reaper b. Alexander Graham Bell

c. wheel c. David Brinkley

d. TU54G tube d. Louisa May Alcott

e. steamboat e. None of these• Directions failed to specify the basis for matching• List are enumerated identically• Responses not listed logically• Lacks homogeneity

• Equal number of elements • Use of “None of the above”

27

IMPROVED: Famous inventions are listed in the left-hand column and inventors in the right-hand column below. Place the letter corresponding to the inventor in the space next to the invention for which he is famous. Each match is worth 1 point, and “None of these” may be the correct answer. Inventors may be used more than once.

Inventions Inventors

__ 1. steamboat a. Alexander Graham-Bell

__ 2. cotton skin b. Robert Fulton

__ 3. sewing machine c. Elias Howe

__ 4. reaper d. Cyrus McCormick

e. Eli Whitney

f. None of these

28

Writing Multiple Choice1. It is recommended that the stem be a direct question.2. The stem should pose a clear, define, explicit, and

singular problem.FAULTY: Salvador Dali is

a. a famous Indian.b. important in international law.c. known for his surrealistic art.d. the author of many avant-garde plays.

IMPROVED: With which one of the fine arts is Salvador Dali associated?a. surrealistic paintingb. avant-garde theatrec. polytonal symphonic musicd. impressionistic poetry

29

Writing Multiple Choice

3. Include in the stem any words that might otherwise be repeated in each response.

FAULTY: Milk can be pasteurized at home bya. heating it to a temperature of 130o

b. Heating it to a temperature of 145o

c. Heating it to a temperature of 160o

d. Heating it to a temperature of 175o

IMPROVED: The minimum temperature that can be used to pasteurize milk at home is:

a. 130o

b. 145o

c. 160o

d. 175o

30

Writing Multiple Choice4. Items should be stated simply and understandably, excluding all

nonfunctional words from stem and alternatives.FAULTY: Although the experimental research, particularly that by

Hansmocker must be considered equivocal and assumptions viewed as too restrictive, most testing experts would recommend as the easiest method of significantly improving paper-and-pencil achievement test reliability toa. increase the size of the group being tested.b. increase the differential weighting of items.c. increase the objective of scoring.d. increase the number of items.e. increase the amount of testing time.

IMPROVED: Assume a 10-item, 10-minute paper-and-pencil multiple choice achievement test has a reliability of .40. The easiest way of increasing the reliability to .80 would be to increaseda. group sizeb. scoring objectivityc. differential item scoring weightsd. the number of itemse. testing time

31


5. Avoid interrelated items6. Avoid negatively stated itemsFAULTY: None of the following cities is a state capital

excepta. Bangorb. Los Angelesc. Denverd. New Haven

IMPROVED: Which of the following cities is a state capital?a. Bangorb. Los Angelesc. Denverd. New Haven

32

Writing Multiple Choice7. Avoid making the correct alternative systematically different from

other options8. If possible the alternatives should be presented in some logical,

numerical, or systematic order.9. Response alternatives should be mutually exclusive.FAULTY: Who wrote Harry Potter and the Goblet of Fire?

a. J. K. Rowlingb. Manny Paquiaoc. Lea Salongad. Mark Twain

IMPROVED: Who wrote Penrod?a. J. K. Rowlingb. J. R. R. Tolkienc. V. Hugo d. L. Carrol

33


10. Make all responses plausible and attractive to the less knowledgeable and skillful student.

FAULTY: Which of the following statements makes clear the meaning of the word “electron”?a. An electronic toolb. Neutral particlesc. Negative particlesd. A voting machinee. The nuclei of atoms

IMPROVED: Which of the following phrases is a description of an “electron”?a. Neutral particleb. Negative particlec. Neutralized protond. Radiated particlee. Atom nucleus

34


11. The response alternative “None of the above” should be used with caution, if at all.

FAULTY: What is the area of a right triangle whose sides adjacent to the right angle are 4 inches long respectively?

a. 7

b. 12

c. 25

d. None of the above

IMPROVED: What is the area of a right triangle whose sides adjacent to the right angle are 4 inches and 3 inches respectively?

a. 6 sq. inches

b. 7 sq. inches

c. 12 sq. inches

d. 25 sq. inches

e. None of the above

35

Writing Multiple Choice12. Make options grammatically parallel to each other and consistent

with the stem.FAULTY: As compared with the American factory worker in the early

part of the 19th century, the American factory worker at the close of the centurya. was working long hoursb. received greater social security benefitsc. was to receive lower money wagesd. was less likely to belong to a labor union.e. became less likely to have personal contact with employers

IMPROVED: As compared with the American factory worker in the early part of the century, the American factory worker at the close of the centurya. worked longer hours.b. had more social security.c. received lower money wages.d. was less likely to belong to a labor unione. had less personal contact with his employer

36


13. Avoid such irrelevant cues as “common elements” and “pat verbal associations.”

FAULTY: The “standard error of estimate’ refer to

a. the objectivity of scoring.

b. the percentage of reduced error variance.

c. an absolute amount of possible error.

d. the amount of error in estimating criterion scores.

IMPROVED: The “standard error of estimate” is most directly related to which of the following test characteristic?

a. Objectivity

b. Reliability

c. Validity

d. Usability

e. Specificity

37


14. In testing for understanding of a term or concept, it is generally preferable to present the term in the stem and alternative definitions in the options.

FAULTY: What name is given to the group of complex organic compounds that occur in small quantities in natural foods that are essential to normal nutrition?

a. Calorie

b. Minerals

c. Nutrients

d. Vitamins

IMPROVED: Which of the following statements is the best description of a vitamin?

15. Use objective items – items’ whose correct answers are agreed by experts

38

Factual Knowledge• The Monroe Doctrine was announced about 10 years after the

a. Revolutionary War

b. War of 1812

c. Civil War

d. Spanish-American War

Conceptual Knowledge

2. Which of the following statements of the relationship between market price and normal price is true?

a. Over a short period of time, market price varies directly with changes in normal price.

b. Over a long period of time, market price tends to equal normal price.

c. Market price is usually lower than normal price.

d. Over a long period of time, market price determines normal price.

39

Translation from symbolic form to another form, or vice versa

3. Which of the graphs below best represent the supply situation where a monopolist maintains a uniform price regardless of the amounts which people buy?

A B C D

S

Pric

e

Quantity

S

Pric

e

Quantity

S

SP

rice

Quantity

S S

Pric

e

Quantity

S

40

Application

In the following items (4-8) you are to judge the effects of a particular policy on the distribution of income. In each case assume that there are no other changes in policy that would counteract the effect of the policy described in the item. Mark the item:

A. If the policy described would tend to reduce the existing degree of inequality in the distribution of income,

B. If the policy described would tend to increase the existing degree of inequality in the distribution of income, or

C. If the policy described would have no effect, or an indeterminate effect, on the distribution of income.

__ 4. Increasingly progressive income taxes.

__ 5. Confiscation of rent on unimproved

__ 6. Introduction of a national sales tax

__ 7. Increasing the personal exemptions from income taxes

__ 8. Distributing a subsidy to sharecroppers on southern farms

41

Analysis

9. An assumption basic to Lindsay’s preference for voluntary associations rather than government order… is a belief

a. that government is not organized to make the best use of experts

b. that freedom of speech, freedom of meeting, freedom of association, and possible only under a system of voluntary associations.

c. in the value of experiment and initiative as a means of attaining an ever improving society

d. in the benefits of competition

42

Judgments in terms of external criteriaFor items 14-16, assume that in doing research for a paper about the

English language you find a statement by Otto Jespersen that contradicts one point of view in a language you have always accepted. Indicate which of the statements would be significant in determining the value of Jespersen’s statement. For the purpose of these items, you may assume that these statements are accurate. Mark each item using the following key.

A. Significant positively – that is, might lead you to trust his statement and to revise your own opinion.

B. Significant negatively – that is, night lead you to distrust his statement

C. Has no significance__ 14. Mr. Jesperson was professor of English at Copenhagen

University __ 15. The statement in question was taken from the very first article

that Jespersen published__ 16. Mr. Jespersen’s books are frequently referred to in other works

that you consult.

43

Essay Questions• 1. Ask questions or set tasks that will require the

examinee to demonstrate a command of essential knowledge.

• 2. Ask questions that are determinate, in the sense that experts could agree that one answer is better than another.

• 3. Define the examinee’s task as completely and specifically as possible without interfering with measurements of the achievement intended.

• 4. In general, give preference to more specific questions that can be more answered briefly.

• 5. Avoid giving the examinee a choice among optional questions unless special circumstances make such option necessary.

• 6. Test the questions by writing an ideal answer

44

Types of Essays:• General – extensiveness of responses• Restrictive Response – reliable scoringLearning outcomes measured by Essay:• Explain cause-effect relationship• Describe applications of principles• Present relevant arguments• Formulate tangible hypothesis• Formulate valid conclusions• State necessary assumptions• Describe the limitations of data• Explain methods and procedures• Produce, organize, and express ideas• Integrate learnings in different areas• Create original forms• Evaluate the worth of ideas

45

Understanding:A. Comparison of two phenomena on a single designated basis:

Compare the writers of the English Renaissance to those of the nineteenth century with respect to their ability tot describe nature

B. Comparison of two phenomena in generalCompare the French and Russian Revolutions

C. Explanation of the use or exact meaning of a phrase or statementThe book of John begins “In the beginning was the word…” From what philosophical system does this statement derive?

D. Summary of a text or some portion of itState the central theme of the Communist Manifesto

E. Statement of an artist’s purpose in the selection or organization of materialWhy did Hemingway describe in detail the episode in which Gordon, lying wounded, engage the oncoming enemy?What was Beethoven’s purpose in deviating from the orthodox form of a symphony in Symphony No. 6?

46

Application:A. Causes or effects

Why may too frequent reliance on penicillin for the treatment of minor ailments eventually result in its diminished effectiveness against major invasion of body tissues by infectious bacteria?

B. AnalysisWhy was Hamlet torn by conflicting desires?

C. Statement of relationshipIt is said that intelligence correlates with school achievement at about .65. Explain this relationship

D. Illustrations or examples of principlesName three examples of uses of the lever in typical American homes

E. Application of rules or principlesWould you weigh more or less on the moon? On the sun? Explain.

F. Reorganization of factsSome writers have said that the American Revolution was not merely a political revolution against England but also a social revolution, within the colonies, of the poor against the wealthy. Using the same evidence what other conclusion is possible?

47

Judgment:

A. Decision for or against

Should members of the Communist Party be allowed to teach in American colleges? Why or why not?

B. Discussion

Discuss the likelihood that four-year private liberal arts colleges will gradually be replaced by junior colleges and state universities.

C. Criticism of the adequacy, correctness, or relevance of a statement

The discovery of penicillin has often been called an accident. Comment on the adequacy of this explanation.

D. Formulation of new questions

What should one find out in order to explain why some students of high intelligence fail in school?

48

Designing Interpretive Exercise• Guidelines in Writing Interpretive Exercise• 1. Select an introductory that is in harmony with the

objectives of the course.– Amount of emphasis of various interpretive skills is a factor.– Do not overload test takers with interpretive items in a particular

area.– Selection of introductory should be guided by general emphasis

to be given to the measurement of complex achievement.

• 2. Select introductory material that is appropriate to the curricular experience and reading ability of the examinees.

49

Guidelines in Writing Interpretive Exercise

• 3. Select introductory material that is new to pupils.• 4. Select introductory material that is brief but

meaningful.• 5. Revise introductory material for clarity, conciseness,

and greater interpretive value. • 6. Construct test items that require analysis and

interpretation of introductory material.• 7. Make the number of items roughly proportional to the

length of the introductory material.• 8. Observe all suggestions for constructing objective

test items.

50

• Ability to Recognize the Relevance of Information

51

• Ability to Recognize Warranted and Unwarranted Generalizations

52

• Ability to Recognize Inferences

53

• Ability to Interpret Experimental Findings

54

• Ability to Apply Principles

55

• Ability to Recognize Assumptions

56

Reading comprehension• Bem (1975) has argued that androgynous people

are “better off” than their sex-typed counterparts because they are not constrained by rigid sex-role concepts and are freer to respond to a wider variety of situations. Seeking to test this hypothesis, Bem exposed masculine, feminine, and androgynous men and women to situations that called for independence (a masculine attribute) or nurturance (a feminine attribute). The test for masculine independence assessed the subject’s willingness to resist social pressure by refusing to agree with peers who gave bogus judgments when rating cartoons for funniness (for example, several peers might say that a very funny cartoon was hilarious). Nurturance or feminine expressiveness, was measured by observing the behavior of the subject when left alone for ten minutes with a 5-month old baby. The result confirmed Bem’s hypothesis. Both the masculine sex-typed and the androgynous subjects were more independent (less conforming) on the ‘independence” test than feminine sex-typed individuals. Furthermore, both the feminine and the androgynous subjects were more “nurturant” than the masculine sex-typed individuals when interacting with the baby. Thus, the androgynous subjects were quite flexible, they performed as masculine subjects did on the “feminine” task.

35. What is the independent variable in the study? a. Situations calling for independence and

nurturanceb. Situation to make the sex type reactc. Situations to make the androgynous be

flexibled. Situations like sex type, androgynous

and sex role concepts 36. What are the levels of the IV? e. masculine attribute and feminine

attributef. rating cartoons and taking care of a

babyg. independence and nurturanceh. flexibility and rigidity

57

Interpreting DiagramsInstruction. Study the following illustrations and answer the following

questions.

Figure 1

Pretest Posttest

101. Which group received the treatment? a. group A b. group Bb. c. none of the above 102. Why did group B remain stable across the experiment? a. there is an EVb. there is no treatmentc. there is the occurence of ceiling effect 103. What is the problem at the start of the experiment? c. the groups are nonequivalentd. the groups are competing with each otherc. the treatment took place immediately

Documents

Lesson 3 developing a teacher made test