Designing Accessible Reading Assessments National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of DARA Project

Designing Accessible Reading Assessments

National Accessible Reading Assessment Projects

General Advisory Committee December 7, 2007

Overview of DARA Project Research

Linda Cook


Background

• Research Questions– Do students with reading-based learning disabilities

receive differential performance gains from an audio (read aloud) accommodation/modification when compared to students without disabilities?

• Differential boost– Do reading comprehension tests and test items taken

with and without a read aloud accommodation/modification measure similar constructs?

• Factor analysis• Differential item functioning analysis

– Distractor analysis– Cognitive labs


Background

• DARA Tests and Samples– Two different tests and several different

samples used for the studies– Gates-MacGinitie Reading Test (GMRT)

• Fourth and eighth grade students (randomized design)

– State standards-based assessment of English-language arts (ELA)

• Fourth and eighth grade students (archival data)


Summary of DARA AnalysesTest/Sample Analyses

GMRT Grade 4

GMRT Grade 8

ELA Grade 4

ELA Grade 8

Differential Boost

X X

Factor Analysis

X X X X

Differential Item Functioning

X X X X

Distractor Analysis

X X

Cognitive Labs X X


Overview of Presentation

• Gates MacGinitie Reading Test (GMRT)– Differential boost– DIF and factor analysis studies

• English-language arts assessment (ELA)– DIF and factor analysis studies

• Next steps


Differential Boost Description • Framework for studying impact of changes in testing

conditions (accommodations or modifications to the standard testing procedures)

• A change in testing condition may be viewed as an accommodation if students with disabilities receive a significant increase in scores compared to students without disabilities


Differential Boost Study• Two forms of GMRT (reading comprehension

section only)– Form S– Form T

• Two grades– Grade 4– Grade 8

• All forms and grade levels contained 48 passage based questions

• All questions were multiple-choice


Differential Boost Samples

• All NJ public school 4th and 8th grade students were invited to participate

• Samples– Fourth grade

• 1170 students– 522 (RLD)– 648 (NLD)

– Eighth grade• 855 students

– 394 (RLD)– 461 (NLD)


Differential Boost Data Collection Design

Group

Session 1 Session 2

Form Testing Condition Form Testing Condition

1 S Standard T Audio

2 S Audio T Standard

3 T Standard S Audio

4 T Audio S Standard


Grade 4 Differential Boost Analysis

Differential Boost Study: Grade 4 Summary Statistics for GMRT Non-LD RLD Condition N Mean SD N Mean SD Audio 648 502 33 522 477 30 Standard 648 497 38 522 457 32 Boost 648 5 24 522 20 29


Grade 8 Differential Boost Analysis

Differential Boost Grade 8 Summary Statistics for GMRT Non-LD RLD Condition N Mean SD N Mean SD Audio 461 555 31 394 521 27 Standard 461 553 30 394 511 28 Boost 461 2 21 394 10 23


Differential Boost ANOVA

• Results of RM-ANOVA showed differential boost– Students with disabilities had significantly larger boost

than students without disabilities– Significant interaction between score gain (boost) and

disability status was observed

• Answer to first question: Students with reading based learning disabilities do exhibit differential performance gains when they take a reading test with a read aloud test change


Purpose of Differential Item Functioning and Factor Analyses of GMRT

• Determine if the test is measuring the same underlying construct (s) for the following comparisons:– Students without disabilities who take the test with

and without a read aloud test change– Students with disabilities who take the test with and

without a read aloud test change


Differential Item Functioning and Factor Analysis of GMRT Data

• Comparisons of item responses for four groups

– Reading based learning disability (RLD), no accommodation (Standard)

– Reading based learning disability (RLD), audio modification (Audio)

– No disability (NLD), no accommodation (Standard)

– No disability (NLD), audio modification (Audio)


Differential Item Functioning of GMRT Data

• Analyzed Form S (4th and 8th grade levels) and Form T (4th and 8th grade levels)

• Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change

• Does the GMRT measure the same underlying construct (s) for the study groups?


Summary Statistics for Grade 4 Form S DIF and Factor Analysis Samples

Comparison Reference Group N Mean SD Focal Group N Mean SD

1 RLD Standard 258 19.18 9.05 RLD Audio 269 24.36 8.81

2 NLD Standard 326 30.08 9.68 NLD Audio 328 32.44 8.81


Differential Item Functioning (DIF)

• Test takers matched on proficiency level– Used total test score as matching criterion

• Reference Groups– Students with and without disabilities who took test

under standard conditions

• Focal Groups– Students with and without disabilities who took test

with read aloud test change


Differential Item Functioning (DIF) Analyses

• Used Mantel-Haenszel procedure with total score as criterion

• Mantel-Haenszel categorization– A—negligible DIF– B—slight to moderate DIF– C—moderate to large DIF

• Direction of DIF Flags– Negative favors reference group– Positive favors focal group


Comparison 1: RLD Standard (reference group) vs. RLD Audio (focal group)

Grade 4 Form S RLD Standard vs. RLD Audio

0

0.25

0.5

0.75

1

-4 -3 -2 -1 0 1 2 3 4

MH D-DIF

Refe

ren

ce P

+

B


Comparison 2: NLD Standard (reference group) vs. NLD Audio (focal group)

Grade 4 Form S NLD Standard vs. NLD Audio

0

0.25

0.5

0.75

1

-4 -3 -2 -1 0 1 2 3 4

MH D-DIF

Refe

ren

ce P

+

B


Factor Analysis of GMRT Data

• Analyzed Form S (4th and 8th grade levels) and Form T (4th and 8th grade levels)

• Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change

• Does the GMRT measure the same underlying construct (s) for the study groups?


Factor Analysis of GMRT Data

• Single group exploratory factor analysis of four groups– RLD who took the test under standard conditions– RLD who took the test with an audio test change– NLD who took the test under standard conditions– NLD who took the test with an audio test change

• Confirmatory factor analysis– Single-group to establish single factor– Multi-group to establish base line model – Multi-group comparisons to establish measurement invariance

• Comparison 1—RLD standard vs. RLD read aloud• Comparison 2—NLD standard vs. NLD read aloud


Summary of Multi-group CFA for RLD Standard vs. RLD Audio

Summary of CFA Testing Equality of Factor Loadings, Variances, and Residuals for RLD Standard vs. RLD Audio

Model DF Normal Theory Chisq.

Normal Theory Chisq./DF

RMSEA GFI CFI Change in CFI

Multi-Group CFA (3a)

40 58.325 1.458 .042 .973 0.990 --

Multi-Group CFA (3b)

47 66.916 1.424 .040 .970 0.989 .001

Multi-Group CFA (3c)

48 67.044 1.397 .039 .970 0.989 .000

Multi-Group CFA (3d)

56 72.604 1.297 .034 .967 0.991 -.002


Summary of GMRT Analyses

• Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without a read aloud test change

• Results of the differential boost study showed that students with reading based learning disabilities receive a differential boost on a reading test taken with a read aloud test change

• Results of DIF and factor analysis studies indicate that test measures the same underlying construct for the RLD group, or for the NLD group when taken with and without a read aloud test change


Analyses of English-language Arts Assessment

• Purpose– To examine if the test measures the same

construct (s) for• Students without disabilities• Students with learning disabilities who took the

test with and without an accommodation/modification


Analyses of ELA Assessment• Analyses

– Factor analyses– Differential item functioning analyses and distractor analyses

• Groups of Interest– Students with learning disabilities who took the test with and

without a change in testing conditions• Test

– Grade 4 and grade 8 English-language Arts (ELA) assessment

• Focus– Determine if the test measures the same constructs for

• Examinees without disabilities• Examinees with learning disabilities who took the test with and

without a change in testing conditions


Number of Items for Grade 4 English-Language Arts Assessment

Test Content No. of Items

Reading Word Analysis, Fluency, and Systematic Vocabulary Development

18

Reading Comprehension 15

Literary Response and Analysis

9

Total—Reading 42

Writing Writing Strategies 15

Writing Applications (Genres and Their Characteristics)

1*

Written and Oral English Language Conventions

18

Total—Writing 34

*Essay item (all others are multiple-choice). The essay item was not used in the study


English-Language Arts Grade 4 Summary Statistics

Grade 4 Total Group and Sample Summary Statistics for English-Language Arts Assessment

Grade 4 Total Groups

Grade 4 Samples (N=500)

Group N Mean SD Mean SD Students Without Disabilities

298,622 48 14 47 14

LD, Without a Change in Testing Conditions

9,045 29 12 29 12

LD, 504/IEP Accommodations

4,724 27 10 27 10

LD, Read-aloud Change in Testing Conditions

1,367 29 11 29 11


Factor Analyses of ELA Assessment

Purpose: to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study

• Exploratory analyses (separately in each group)– how many factors

• Confirmatory (multi-group)– Establish base-line model– Confirm number of factors needed to describe data

across all groups


Differential Item Functioning (DIF) Analyses

• The purpose of the DIF study was to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study

• Used Mantel-Haenszel procedure with total score as criterion

• Mantel-Haenszel categorization– A—negligible DIF– B—slight to moderate DIF– C—moderate to large DIF

• Direction of DIF Flags– Negative favors reference group– Positive favors focal group


Comparisons Made in the DIF Study

Comparison Number Reference Group Focal Group

1 Without disabilities LD no change in testing

condition

2 “ LD IEP/504 accommodations

3 “ LD read-aloud change in testing

condition


Interpreting the Results of the DIF Study

• Fourth grade results– 1 C DIF item, 10 B DIF items– 5 B DIF items were reading items that

favored students with disabilities who took test with read-aloud change in testing conditions

– 5 B DIF items (3 reading and 2 writing) favored students without disabilities

– Results consistent with factor analyses results


Interpreting the Results of the DIF Study

• Eighth grade results– 1 C DIF item, 7 B DIF items– Five B DIF items (4 reading and 1 writing)

favored students who took test with read-aloud change in testing conditions

– Again, results consistent with factor analyses results


Summary of Research Results

• GMRT– Differential boost study

• Students with reading disabilities show differential performance gains when taking a test with read aloud

– DIF and Factor Analysis• Internal structure (constructs) measured by test taken by

students with and without disabilities taking test with and without read aloud very similar

• ELA– DIF and Factor Analysis

• Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without test changes very similar


Research Answers

• Do students with reading-based learning disabilities receive differential performance gains from read aloud?– yes

• Do reading tests and test items taken with and without a read aloud accommodation/modification measure similar constructs?– Measure the same single factor for all groups – Show little evidence of differential item performance


Next Steps

• Complete analysis of data from cognitive labs

• Design and carry out simulations for multi-stage reading assessment

• Design and carry out field trial of prototype assessment


Analyzing Data from Cognitive Labs

• Cognitive labs are a means of measuring mental processes through a think aloud protocol

• Administered GMRT to 34 fourth grade students and 15 eighth grade students

• Gained insight into test taking strategies as well as issues related to test format and content


Reading Comprehension

Routing Test

Reading FluencyExtended Reading

ComprehensionTest

Decoding and Extended ComprehensionTest with Audio

Extended ComprehensionTest with Audio


NARAP Field Trial

• Large data collection effort

• Design and develop accessible reading assessment prototypes based on NARAP Principles and Guidelines

• Will field trial DARA, PARA, and TARA prototypes


Questions?

Documents

Designing Accessible Reading Assessments National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of DARA Project