Upload
nema-grace-medillo
View
174
Download
20
Embed Size (px)
Citation preview
ADMINISTERING, ANALYZING, AND IMPROVING THE TEST OR
ASSESSMENT
NEMA GRACE B. MEDILLO
GOAL IN THIS CHAPTER:
To provide suggestion on how to avoid common pitfalls in test assembly, administration, and scoring.
Start
Assembling the Test
Analysing the Test Debriefing
Scoring the Test
Process of Evaluating Classroom Achievement
Administering the Test
Nema Grace B. Medillo
ASSEMBLING THE TEST
Packaging the Test Reproducing the Test
Packaging the Test
Group together all items of similar formatArrange test items from easy to hardSpace the items for easy readingKeep the items and options on the same pagePosition illustration near descriptions and above the itemCheck your answerDetermine how the students record their answerProvide space for Name and DateCheck test directionsProofread the test
Reproducing the TestKnow the Photocopying Machine
Specify Copying Instructions
File Original Test
ADMINISTERING THE TEST
Maintain a Positive AttitudeMaximize Achievement MotivationEqualize AdvantagesAvoid SurprisesClarify the RulesRotate Distribution
Remind Students to Check Their Copies
Monitor Students
Minimize Distractions
Give Time Warnings
Collect Test Uniformly
SCORING THE TEST
Prepare the Answer Key
Check the Answer Key
Score Blindly
Check Machine – Scored Answer Sheets
Check Scoring
Record Scores
ANALYZING THE TEST
Quantitative Item Analysis
Qualitative Item Analysis
Item Analysis Modifications for the Criterion – Referenced Test
Item Analysis Terminology
Item Analysis TerminologyQuantitative Item Analysis• A numerical method for analyzing test items employing
students response alternatives or options.
Qualitative Item Analysis• A non – numerical method for analyzing test items not
employing students responses, but considering test objectives, content validity, and technical item quality
Key• Correct option in a multiple – choice item
Item Analysis TerminologyDistractor• Incorrect option in a multiple – choice item
Difficulty Index• Proportion of students who answered the item correctly.
Discrimination Index D• Measure of the extent to which a test item discriminates or
differentiates between students who do well on the overall test and those who do not do well on the overall test.
Discrimination Index (D)
• Those who did well on the overall test chose the correct answer for a particular item more often than those who did poorly on the overall test.
Positive Discrimination Index
• Those who did poorly on the overall test chose the correct answer for a particular item more often than those who did well on the overall test.
Negative Discrimination Index
• Those who did well and those who did poorly on the overall test chose the correct answer for a particular item with equal frequency
Zero Discriminanation Index
Quantitative Item Analysis
Case Sample and Guide Questions
Difficulty Level
Discrimination Index
Miskeying Guessing Ambiguity
Consider the case belowSuppose your students chose the options to a four – alternative multiple – choice item.
Let C as the correct answer.
A B C* D3 0 18 9
How does this information help us?Is the item too difficult/easy for the students?Are the distractors of the items effective?
Item X
Guide questions in quantitative item analysis
1.What is the difficulty level?2.What is the discrimination index?3.Should this item be eliminated?4.Should any distractor(s) be modified?
Difficulty LevelTo compute the difficulty level of an item;
A B C* D3 0 18 9
What is the difficulty level of the item?Do you consider the item difficult or easy? Why?
𝒑=𝑵𝒖𝒎𝒃𝒆𝒓 𝒔𝒆𝒍𝒆𝒄𝒕𝒊𝒏𝒈𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒂𝒏𝒔𝒘𝒆𝒓
𝑻𝒐𝒕𝒂𝒍 𝑵𝒖𝒎𝒃𝒆𝒓 𝒕𝒂𝒌𝒊𝒏𝒈𝒕𝒉𝒆𝒕𝒆𝒔𝒕
Item X
𝒑=𝑵𝒖𝒎𝒃𝒆𝒓 𝒔𝒆𝒍𝒆𝒄𝒕𝒊𝒏𝒈𝒄𝒐𝒓𝒓𝒆𝒄𝒕 𝒂𝒏𝒔𝒘𝒆𝒓
𝑻𝒐𝒕𝒂𝒍 𝑵𝒖𝒎𝒃𝒆𝒓 𝒕𝒂𝒌𝒊𝒏𝒈𝒕𝒉𝒆𝒕𝒆𝒔𝒕
𝒑=𝟎 .𝟔𝟎
Solving the difficulty index
𝒑=𝟏𝟖𝟑𝟎
Since the difficulty level of the item is 0. 60 (60%), the item is moderately difficult.
A B C* D3 0 18 9
Item X
Note: If P level > 0.75, the item is considered relatively easy.If P level < 0. 25, the item is considered relatively difficult.
Discrimination IndexSteps in determining Discrimination Index
1. Arrange the papers from highest to lowest score.2. Separate the papers into an upper group and
lower 3. For each item, count the number in the upper
group and the number in the lower group that chose each alternatives
4. Record your information for each item Example for item X ( Class Size = 30)
Options A B C* DUpper 1 0 11 3Lower 2 0 7 6
5. Compute D, by plugging the appropriate numbers in the formula
What is the discrimination index of item X?Is the discrimination index positive or negative?Which of the groups frequently get the item correctly?
(if group sizes are unequal, choose the higher number)
Plugging the data
Since the discrimination index of item X is 0. 267, which is positive. More students who did well on the overall test answered the item correctly than students who did poorly on the overall test.
Implication Difficulty Level (p) = 0. 60
Discrimination Index (D) = 0.267
Should this item be eliminated?
The item is considered a moderately difficult item that has positive (desirable) discrimination ability.
NO
Should any distractor(s) be modified?
Option B is ought to be modified or replaced.
A B C* D3 0 18 9
Item X YES
(No one chose it)
Experts say;
• Build tests that have most items between p levels of 0.20 and 0. 80, with an average p level about 0.50.
• Consider any item with a positive D value.
Let’s look at the responses for another itemItem Y ( Class size = 28)
Options A* B C DUpper 4 1 5 4Lower 1 7 3 3
Item Z ( Class size = 30)
Options A B* C DUpper 3 4 3 5Lower 0 10 2 3
1. What is the difficulty level?2. What is the discrimination index?3. Should this item be eliminated?4. Should any distractor(s) be modified?
Item Y Difficulty Level = 0.18
Discrimination Index = 0. 214Should this item be eliminated?
Should any distractor(s) be eliminated?
No, since it is positively discriminating. However, it is difficult item; only 18% of the class got it right.
Yes; C and D attracted more students who did well on the test overall.
Remember:Fewer students who do well on the test should choose each distractor than students who do poorly. More students who do well on the test should choose the correct answer than students who do poorly
Difficulty Level = 0.467Discrimination Index = - 0.40
Should this item be eliminated?
Should any distractor(s) be eliminated?
Yes! The item is moderately difficult but it discriminate negatively.
Since we already decided to eliminate the item, this is moot question.
Item Z
Remember:Testing is to discriminate between those students who know their stuff and those who do not.
MISKEYINGMost students who did well on the
test will likely select an option that is a distractor, rather than the option that is keyed.
Consider the miskeyed itemWho was the first astronaut
to set foot on the moon?
a. John Glennb. Scott Carpenterc. Neil Armstrong
*d. Alan Sheppard
Responses A B C D*Upper 1 1 9 2
Most students in the upper half of the class fail to select the keyed option.
Remember, just as you are bound to make scoring errors, you are bound to miskey an item occasionally.
GUESSINGLikely to occur when the item measures content that is,
• Not covered in class or the text• So difficult that even the upper – half students
have no idea what the correct answer is• So trivial that students are unable to choose from
among the options provided.
The choice distribution suggest that guessing occurred:
A B C* DUpper Half 4 3 3 3
Each alternative is about equally attractive to students in the upper half.
AmbiguityAmong the upper group, one of the distractors is
chosen with about the same frequency as the correct answer.
The distribution suggests that an item is ambiguous:
A B C D*Upper Half 7 0 1 7
QUALITATIVE ITEM ANALYSIS
• Matching items and objectives • Editing poorly written items• Improving content of the validity of the test• Analyzing grammatical cues, specific determiners,
double negatives, multiple defensible answers, and items that fail to match instructional objectives
ITEM ANALYSIS MODIFICATIONS FOR THE CRITERION – REFERENCED TEST
Using Pre- and Posttest as Upper and
Lower Group
Comparing the Percentage
Answering Each Item Correctly on Both Pre- and Posttest
Determining the Percentage of
Items Answered in the Expected
Directions
Limitations of Modifications
Using Pre- and Posttests as Upper and Lower Groups
Pretest prior instructions
Most of the test items answered incorrectly
Results for Lower Group
P level = 0. 30 or lower
Post test after instructions
Most items were answered correctly
Results for Upper Group
P = 0.70 or higher
Analyzing sample DataExample 1:
Number of students choosing option (n = 25)Option At pretest (L) At posttest(U)A 9 1B 7 1C 3 2D* (Key) 6 21
Steps1. Compute p Levels for both tests.
PretestPosttest
= 0.24 =0.84
It was an improvement from 24% to 84%.
2. Determine the discrimination index (D) for the key
The key has a positive discrimination index.
3. Determine whether each option separately discriminates negatively.
Option A: Option B: Option C:
Summary1. There was sizeable increase in p value from
pretest to posttest.2. The D index for the key was positive.3. The distractors are all discriminated
negatively.If a criterion – reference test item manifests these features, it has passed our “test” and a probably
is a good item with little or no need for modification.
Contrast with, the test item failed. Rather than modify, it is probably more efficient to
replace it with another.
Comparing the Percentage Answering Each Item Correctly on Both Pre- and
PosttestIf your test is sensitive to your objectives.
What to do?Percentage Passing Posttest – Percentage of Passing Pretest
The more positive the difference, the more the item tapping the content you are teaching.
Analyzing Sample DataConsider the following percentages for five test items:
Item Percentage passing pretest
Percentage passing posttest
Difference
1 16 79 +63%
2 10 82 +72%
3* 75 75 0%
4 27 91 +64%
5* 67 53 -14%
Item 3 may be eliminated. (Students already know the content it represents.) Item 5 may be revised. (Instruction was not related to the item or confused some students.)
Determining the Percentage of Items Answered in the Expected
Directions for the Entire TestUses the number of items each
learner failed on the test prior to instruction but passed on the test after instruction
Steps1. Find the number of items each student failed on the
pretest but passed on the posttest.
Result of MaryItem Pretest Posttest1* Incorrect Correct2 Correct Correct3* Incorrect Correct4 Correct Incorrect5 Incorrect Incorrect
Do same things to other students.
2. Add the counts and divide by the number of students.
ExampleMary 18 Carlos 15Sharon 22 Amanda 20Charles 13
3. Divide by number of test items.
4. Multiply by 100.
0.70 x 100 = 70%
The greater the overall positive percentage of change, the more your test is likely to match your instruction and to be a content – valid test
Limitations of Modification
Difficult
Unit of instruction is brief
From norm – referenced test to criterion – referenced test
Time devote to instruction (pre – post)
DEBRIEFINGDiscuss Problem ItemsListen to Students ReactionsAvoid – on – the Spot DecisionsBe Equitable with ChangesAsk Students to Double – CheckAsk Students to Identify Problems
PROCESS OF EVALUATING CLASSROOM
ACHIEVEMENTS
THANK YOU…
THANK YOU…
THANK YOU…
THANK YOU…THANK YOU…
THANK YOU…
Shutting down