Upload
sera
View
34
Download
0
Embed Size (px)
DESCRIPTION
Valid, Reliable & Efficient:. A Psychometric Evaluation of “Flash” Word Recognition and “NSSI” Passage Reading Measures. Kathleen J. Brown Matthew K. Fields University of Utah R. Darrell Morris Appalachian State University. Impetus for Current Study. - PowerPoint PPT Presentation
Citation preview
Valid, Reliable & Efficient:
A Psychometric Evaluation of
“Flash” Word Recognition and
“NSSI” Passage Reading Measures
Kathleen J. BrownMatthew K. FieldsUniversity of Utah
R. Darrell Morris Appalachian State University
Impetus for Current Study
Need for valid, reliable, efficient instruments to determine instructional reading level
Flaws with current instruments DRA – no rate & time consuming DIBELS – screen only IRIs – psychometric evaluation often weak or
missing
Impetus for Current Study
Growing use of “Flash” & selected graded passages a.k.a. “NSSI” (Virginia/ASU effect)
Initial psychometric evaluations positive (Frye, 2004; Frye & Trathen, 2004; Frye
Trathen, Olson, & Schlagal, 2002; Palmer, Trathen, Olson, & Schlagal, 2002)
Theoretical Framework
Instrument Reliability Validity
Flash
(WRI)
Alternate Form
(manual vs. computer)
Content (rep of grade level?)
Concurrent (GORT)
NSSI
(IRI)
Alternate Form
(A passages vs. B passages)
Content (rep of grade level?)
Concurrent (GORT)
(Anastasi, 1988; APA, 1985)
Methods
4 schools 2 = Title 1 1 = public, 1 = parochial 2 = non-Title 1 both = public & mixed SES
192 students in G2-G5 in March, 2006
Rank ordered DIBELS or QRI, then sampled 12 students per grade: 4 high, 4 average, 4 poor to achieve a representative distribution for testing
Methods
135 minutes of assessment in 3 sessions
Presentation order counterbalanced
Flash item selection counterbalanced
9 on data team; 4 hours protocol training
Manual flash interrater differences = n.s.
Alternate Form Reliability
measure of temporal stability for scores
measure of consistency of response for scores
Alternate Form Reliability
To what extent are NSSI A passage scores equivalent to NSSI B passage scores?
To what extent are computer “Flash” scores equivalent to manual “Flash” scores?
Results: Alternate Form Reliability
NSSI Form B
Flash Manual
Flash Comp
NSSI A .878** .761** .803**
NSSI B Add
n.s. on t-tests
.807** .859**
Flash Manual
.847**
**p < .01
Content Validity
provides evidence that items on test represent a specific domain
provides evidence that the format and response properties of the test represent the domain
Content Validity
To what extent do the NSSI passages reflect/measure expected grade level benchmarks?
Maybe look at separate means for accuracy, rate, & comp & report those to show
To what extent does the Flash measure reading instructional level?
NSSI Reading Level Criteria
Acc. (%)
Rate (wpm)
Comp (%)
Mid G1 95 30 60 End G1 95 40 60 Mid G2 95 65 60
End G2 95 or
97 90 60 or
75 End G3 95 90 60 End G4 95 110 60 End G5 95 120 60 End G6 95 130 60 End G7 95 150 60 End G8 95 150 60
Performance Levelfor NSSI by Grade
NSSI Form A
NSSI Form B
Grade 2 M (SD)
2.14 (1.42)
2.28
(1.24)
Grade 3 M
(SD) 3.52
(1.09) 3.49
(1.11)
Grade 4 M (SD)
4.04
(1.43) 3.87
(1.34)
Grade 5 M (SD)
5.06
(1.85) 4.90
(1.57)
Performance Levelfor Flash by Format & Criterion
Manual (70%)
Comp (70%)
Manual (85%)
Comp (85%)
Grade 2 M (SD)
3.26
(1.24) 3.21
(1.31) 2.58
(1.09) 2.48
(1.20)
Grade 3 M (SD)
4.82
(1.82) 4.59
(1.58) 3.50
(1.31) 3.80
(1.41)
Grade 4 M (SD)
5.81
(1.63) 5.91
(1.88) 4.28
(1.63) 4.57
(1.79)
Grade 5 M (SD)
6.99
(1.49) 6.97
(1.93) 5.46
(1.76) 5.66
(1.94)
Concurrent Validity
To what extent are Flash scores and NSSI scores consistent with scores achieved on a “flagship” standardized reading measure (i.e., the GORT)?
Results: Concurrent Validity
NSSI Form B GORT
Flash Manual
Flash Comp
NSSI A .878** .823** .761** .803**
NSSI B .842** .807** .859**
GORT .808** .826**
Flash Manual
.847**
**p < .01
Average Performance Levelfor NSSI & GORT by Grade Level
NSSI Form A
NSSI Form B GORT
Grade 2 M (SD)
2.14
(1.42) 2.28
(1.24) 3.92
(1.72)
Grade 3 M (SD)
3.52
(1.09) 3.49
(1.11) 5.63
(1.55)
Grade 4 M (SD)
4.04
(1.43) 3.87
(1.34) 6.34
(1.53)
Grade 5 M (SD)
5.06
(1.85) 4.90
(1.57) 6.78
(1.22)
Conclusions: For G2-G5
NSSI A and NSSI B seem to have high validity for identifying students’ instructional reading levels
NSSI A and NSSI B can be considered equivalent forms
Conclusions: For G2-G5
Manual Flash and Computer Flash seem to have high validity for identifying students’ instructional reading levels when the criterion is set at 85%
Manual Flash and Computer Flash seem to be equivalent forms
Conclusions: For G2-G5
The GORT does not seem to have high validity for identifying students’ instructional levels—at any grade level.
The GORT over-predicts instructional level—by approx. 2 years.
Note: most GORT comp questions are passage independent (Keenan & Betjemann, 2006)