Upload
ordell
View
35
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Measure Up! Benchmark Assessment Quality Assurance Process. RCAN September 10, 2010. Measure Up!. Objective To monitor and improve Benchmark Assessments in order to attain the most accurate possible measurement of student achievement with respect to California Content Standards Tests. - PowerPoint PPT Presentation
Citation preview
1
Measure Up!Benchmark Assessment
Quality Assurance Process
RCANSeptember 10, 2010
2
Measure Up! ObjectiveTo monitor and improve Benchmark Assessments in order to attain the most accurate possible measurement of student achievement with respect to California Content Standards Tests.
Measure Up! Components• Content—Structure & Course Guides• Predictability
– Correlation of Benchmark Exams to CST scores– Association of Benchmark Exams with CST Performance Levels
• Item Analysis– Difficulty– Discrimination– Representative of CST items
Evolving
4
PSUSD Benchmark Structure
CST
~153 ID
60 to 75 Items
Blueprint Aligned
Benchmark#1
~45 ID
~20-35 Items
1st 45 ID Paced
Standards
Partial Match to
CST
Benchmark#2
~90 ID
~20-35 Items
2nd 45 ID Paced
Standards
Partial Match to
CST
Benchmark#3
~135 ID
~20-35 Items
3rd 45 ID Paced
Standards
Partial Match to
CST
5
2008-2009
Algebra I(8th Grade)Aggregation of 3 Benchmarks
2008-209
Algebra I(8th Grade)
CST
Predictability
6
Prof = 38
38
Algebra I BM 08-09 (8th Grade)
Alge
bra
I CST
200
9 (8
th G
rade
)
7
N = 23
N = 119 N = 125
N = 51
N = 5
8
Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 8 Algebra I
using STAR Performance Band Cut Points
1 5 38
189
3
32 56
11
14
50
54
3
5
32
18
1
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
N = 9
N = 70
N = 123
N = 100
N = 21
9
2008-2009
ELA 10th GradeAggregation of 3 Benchmarks
2008-209
ELA 10th Grade
CST
Predictability
10ELA 10th Gd BM 08-09
ELA
10th
Gd
CST
2009
Prof = 53
53
11
N = 26
N = 151
N = 326
N = 249
N = 36
12
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 10 English Language Arts
using STAR Performance Band Cut Points
18 28 21
5143
7
50
162
30
7
107
85
31
76
35
41-Far Below
Basic2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erf
orm
an
ce
Lev
els
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
6 Basic6 Below Basic
7 Proficient
4 Far Below Basic
2 Basic6 Below Basic
2 Advanced
N = 26
N = 138
N = 364
N = 202
N = 58
13
2008-2009
US History11th GradeAggregation of 3 Benchmarks
2008-209
US History11th Grade
CST
Predictability
14
r = .46
US His 11th Gd BM 08-09
US
His
11th
Gd
CST
2009
Prof = 40
63
15
N = 68
N = 116
N = 142
N = 72
N = 7
16
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 11 US History
using STAR Performance Band Cut Points
48
19 2410
25
24 18
12
25
2644
23
15
30
28
5
5
13
2
3
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
2 Below Basic2 Far Below Basic
2 Basic
N = 100
N = 84
N = 121
N = 86
N = 14
17
2008-2009
Science8th GradeAggregation of 3 Benchmarks
2008-209
Science8th Grade
CST
Predictability
18
r = .77
Science 8th Gd BM 08-09
Scie
nce
8th G
d CS
T 20
09
Prof = 39
44
19
N = 84
N = 307
N = 494
N = 275
N = 24
20
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 8 Science
using STAR Performance Band Cut Points
6430 13
67
89
3515
31
110
77
20
6
59
154
111
10
40
154
91
1
5
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
1 Basic1 Below Basic
N = 168
N = 293N = 319
N = 301
N = 103
21
2008-2009
Math6th GradeAggregation of 3 Benchmarks
2008-209
Math6th Grade
CST
Predictability
22
r = .84
Math 6th Gd BM 08-09
Mat
h 6th
Gd
CST
2009
Prof = 43
48
23
N = 164
N = 393
N = 460
N = 224
N = 31
24
Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 6 Math
using STAR Performance Band Cut Points
41 57
42
202
89
116
246
43
9129
142
83
416 4 4
3
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic
N = 98
N = 384
N = 472
N = 272
N = 45
2 Basic
8 Proficient5 Advanced
25
2008-2009
ELA4th GradeAggregation of 3 Benchmarks
2008-209
ELA4th Grade
CST
Predictability
26
r = .80
Prof = 44
102
27
N = 88
N = 176
N = 334
N = 388
N = 93
28
Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 4 English Language Arts
using STAR Performance Band Cut Points
25 27 8
1842
54
13
39
167
62
8
106
167
31
23
103
169
1
7
7
1-Far BelowBasic
2-Below Basic 3-Basic 4-Proficient 5-Advanced
Benchmark Exams Performance Levels
CS
T P
erfo
rman
ce L
evel
s
5-Advanced
4-Proficient
3-Basic
2-Below Basic
1-Far Below Basic1 Proficient
1 Far Below Basic
N = 51
N = 115
N = 358N = 346
N = 209
29
Item Level Analysis
30
Item Difficulty
• The p value for any item– percentage of correct answers– usually in decimal form
• Ideally p value range is .30 to .80 for most items• For example
– p value of .28 = 28% of the test takers got the item right– p value of .75 = 75% of the test takers got the item right– P value of .95 = 95% of the test takers got the item right
31
Item Difficulty MonitoringItem Number N Students Standard p Value
1 853 NS1.2 .37
2 853 NS1.2 .58
3 853 NS1.2 .77
4 853 MG2.1 .25
5 853 MG2.1 .35
6 853 MG2.1 .40
7 853 PS3.3 .43
8 853 PS3.3 .59
9 853 PS3.3 .75
10 853 PS3.3 .95
32
Item Discrimination
• Is item “discriminating” appropriately between higher & lower scoring students
• Discrimination Index (DI) = difference between how upper half and lower half of students score on an item
• DI ranges between -1 and +1• We want items to discriminate positively
33
Item Discrimination MonitoringItem
NumberN
Students Standard p Value DI Max DI
1 853 NS1.2 .37 .45 .742 853 NS1.2 .58 .53 .843 853 NS1.2 .77 .30 .464 853 MG2.1 .25 -.12 .505 853 MG2.1 .35 .10 .706 853 MG2.1 .40 .62 .807 853 PS3.3 .43 .40 .868 853 PS3.3 .59 .58 .829 853 PS3.3 .75 .41 .50
10 853 PS3.3 .95 .09 .10
34
2008-2009
7th Grade Math3rd Benchmark
2009-2010
7th Grade Math3rd Benchmark
Representative of CST Items and our continual Revision Process
35
7NS1.6
2008-2009 Item
p = .33DI = .34
36
RTQ for 7NS1.6
CST = 1RTQ = 1
37
7NS1.6
2009-2010 Item
p = .42DI = .31
38
Additionally…
• Item #3 replaced (as #3) with item modeled after RTQ # 16 (7NS1.7*)—CST = 5, RTQ = 7
• Item #15 replaced (as #13) with item modeled after RTQ #47 (7AF1.5)
• Item #s 23 & 24 replaced (as #s 21 & 22) with items modeled after RTQ #s 89 & 88 (7MG3.4*)
39
Measure Up! Next Steps
• Benchmark Exam Structure• Institutionalize System• CAHSEE• Writing Prompt• Discrimination• Distractor Shaping
40
Questions?