A Statistical Linkage Between

NAEP and ECLS-K Grade Eight Reading Assessments

Enis Dogan

Burhan Ogut

Young Yee Kim

Sharyn Rosenberg

NAEP Education Statistics Services InstituteAmerican Institutes for Research

A Statistical Linkage Between

NAEP and ECLS-K Grade Eight Reading Assessments

Enis Dogan

Burhan Ogut

Young Yee Kim

Sharyn Rosenberg

NAEP Education Statistics Services InstituteAmerican Institutes for Research

Purpose

In Spring of 2007, about 1300 students took the NAEP and ECLS-K grade eight reading assessments

The purpose of this study is to establish a statistical link between these two assessments using this common sample

We also test the validity of the link by comparing the model projected results to reported NAEP results

BACKGROUND

Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K)

Provides longitudinal data on students’ academic achievement in reading, mathematics, and science

Began in the fall of 1998 with a national sample of 21,000 kindergartners

Each student is tested by a series of two-stage, 30-minute adaptive tests, administered face to face, on seven occasions

Item parameters and individual scores are estimated using IRT

NAEP 8th Grade Reading 2007

Provides cross-sectional data on students’ academic achievement in reading

Nationally representative sample of more than 350,000 students participated in the 2007 reading assessment

Each student takes just a portion of the test (BIB Design), consisting of two 25-minute sections or one 50-minute section

Item parameters are estimated using IRT, but results are reported at group level. NAEP does not report individual scores

Differences between NAEP and ECLS-K

The two assessments have similar frameworks, but do not share any common items

NAEP uses a nationally representative sample ECLS-K is representative of children enrolled in first grade

in the U.S. in 1999-2000

ESTIMATING THE PROJECTION EQUATION

Linking through a common sample

In Spring of 2007, about 1300 public school students took NAEP and ECLS-K grade 8 reading assessments. o About ¼ were part of the operational NAEP sample o The rest took the NAEP assessment for the purposes of

the common sample linking study

Method and challenges

NAEP scores in the form of plausible values were not available for about ¾ of the linking sample since they were not part of the operational NAEP assessment

Using item level data from the entire common sample, MML regression was used (with the AM software) to estimate a projection equation predicting NAEP score distributions from ECLS-K scores

The projection equation was

where y is the NAEP score and x is the ECLS-K score

i i iy a bx e

The projection equation

The projection equation was estimated using two different weights:o the original ECLS-K sampling weights, and o poststratification weights

Parameter estimates predicting NAEP scale scores from ECLS-K scale scores

1 Using original weights: RMSE = 18.84, F(1,168) = 326.47

2 Using poststratification weights: RMSE = 18.60, F(1,168) = 370.64

Parameter Original estimate1 Poststratified estimate2

Intercept 79.32 (10.46) 75.86 (9.92)

Slope 1.10 (0.06) 1.12 (0.06)

Variance of the projection

For the projection equation, the a, b parameter estimates are denoted

and their covariance as

The projected NAEP average reading score is

The variance of the projection at mean is

i i iy a bx e

2

2

ˆ ˆˆˆ( , )ˆ ˆa ab

ab b

Var a b

ˆˆ ˆy a bx

2 2 2 2ˆˆ ˆ ˆ ˆVar 2 Vara ab by x x b x

TESTING THE VALIDITY OF PROJECTION EQUATION

Comparing reported and projected NAEP mean

National average scores and 95% confidence intervals for projected and reported 2007 NAEP reading scores: grade 8, public

Scores by gender and race/ethnicity

Left bar: Confidence Interval for the mean based on the projection using the original weights

Right bar: Confidence Interval for the reported NAEP mean

N

AE

P S

cale

AN APPLICATION OF THE LINK: READING PERFORMANCE AT EARLIER GRADES AND PROFICIENCY IN NAEP

What is the relationship between reading performance at first, third and fifth grades and Proficiency in eighth-grade NAEP reading assessment?

Early childhood reading performance and proficiency in NAEP

Show an overall understanding of the text, including inferential as well as literal information

Extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences—including other reading experiences

Identify some of the devices authors use in composing text

Proficiency in eighth-grade NAEP reading assessment

Level 1 : Letter recognition: identifying upper- and lower-case letters by name

Level 2 : Beginning sounds

Level 3 : Ending sounds

Level 4 : Sight words

Level 5 : Comprehension of words in context

Level 6 : Literal inference

Level 7 : Extrapolation

Level 8 : Evaluation

Level 9 : Evaluating nonfiction

Level 10: Evaluating complex syntax: evaluating complex syntax and understanding high-level nuanced vocabulary in biographical text.

ECLS-K Reading performance levels

Percentage of students at Levels 5 through 9: Grade 5

Reading performance at grade 5 and Proficiency in 8th grade NAEP

Percentage of students at Levels 5 through 9: Grade 5

Reading performance at grade 5 and Proficiency in 8th grade NAEP

33 %

48 %

72 %Evaluation: demonstrating understanding of author’s craft …

13 %

Percentage of students at Levels 4 through 8: Grade 3

Reading performance at grade 3 and Proficiency in 8th grade NAEP

Percentage of students at Levels 4 through 8: Grade 3

Reading performance at grade 3 and Proficiency in 8th grade NAEP

11 %25 %

46 % 62 %

Extrapolation: identifying clues used to make inferences, …

Percentage of students at Levels 3 through 7: Grade 1

Reading performance at grade 1 and Proficiency in 8th grade NAEP

Reading performance at grade 1 and Proficiency in 8th grade NAEP

13 %

34 %

49 %

73 %

85 %

Percentage of students at Levels 3 through 7: Grade 1

Comprehension of words in context: reading words in context

Summary of findings

Using a common sample, a linking equation was estimated

Using the equation, projected NAEP scores were computed for ECLS-K 8th graders

Mean projected NAEP scores for the nation and the gender and racial/ethnic groups were close to reported NAEP results

Mean projected NAEP scores were used to examine the relationship between proficiency in grade eight NAEP reading and earlier reading performance

Limitations

Composition of the ECLS-K sample and the

population it represents

Projection results for Hispanic students

Future analyses

Further validation of the link

Modeling growth in reading on the NAEP scale

Discussion