Developing a Severity-of-Disability Scale and Modeling Early Reading and

Math Performance in a Longitudinal Study of Preschoolers with Disabilities

Elaine Carlson Tamara Daley

Frank Jenkins

Westat

The Pre-Elementary Education Longitudinal Study is funded by the U.S. Department of Education.

PEELS design 6-year national longitudinal study of preschoolers with

disabilities, currently in its fourth year. Designed to describe

the children, the services they receive, their transitions from early intervention to preschool

and preschool to elementary school, and their early academic and functional skills.

Sample includes 3,104 children who were ages 3-5 and were receiving special education services in 2003-04.

Produces statistical estimates that generalize to the national population of children with disabilities ages 3-5.

PEELS design, cont’d

Data collection a one-on-one assessment of early academic skills, a parent telephone interview, mail questionnaires to children’s teachers and school-,

district-, and state-level administrators, and indirect assessments of students’ social skills,

problem behaviors, academic skills, and motor skills

Sample design

Nationally representative sample of 223 LEAs (208 in Wave 1, plus 15 added in Wave 2)

LEAs stratified by

enrollment size geographic region wealth

LEA participation Main sample

709 released for recruiting 245 agreed in 2001 189 participated in 2003-04 (Wave 1)

Nonresponse sample 32 of 464 non-recruited LEAs selected 19 participated in 2003-04 (Wave 1)

Supplemental sample to add 1 previously nonparticipating SEA 15 of 24 participated in 2004-05 (Wave 2)

Total Sample = 223 LEAs

Family participation

Wave 1 main and nonresponse samples

(ages 3-4 in 2003-04)

52,871 on sampling frame

5,330 selected

4,072 eligibility determined

2,906 agreed (81% of those eligible)

Family participation

Wave 2 supplemental sample

(ages 4-6 in 2004-05)

7,727 on sampling frame

542 selected

433 eligibility determined

198 agreed (63% of those eligible)

Total sample = 3,104 families

PEELS schedule

Unweighted response rates, by wave

Wave

1 2 3

SEA questionnaire 100% - -

LEA questionnaire 84% - -

Principal/program director questionnaire* 73% 77% 56%

Teacher questionnaire 79% 84% 84%

Parent interview 96% 93% 88%

Child assessment 96% 94% 93%

English direct assessment 84% 87% 88%

Alternate assessment 11% 7% 5%

-Not applicable

* QED data were used to supplement these data, bringing percentage of children with some school data in waves 1, 2, & 3 to 94%, 95%, & 94%, respectively

Demographic characteristics

Gender

Male 70.7

Female 29.3

Race/ethnicity

American Indian/Alaska Native 3.3

Asian 3.2

Black or African American 14.2

White 84.6

Pacific Islander 0.8

Hispanic, any race 21.0

NOTE Demographic and household characteristics are for the unweighted sample.

SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), “Parent Interview (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

Household characteristics

Mother’s education

<H.S diploma 18.9

H.S. diploma or GED 31.0

Some college 29.4

4-year degree or higher 21.0

Household Income

$20,000 or less 26.3

$20,000-$40,000 28.3

>$40,000 45.3

SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), “Parent Interview (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

Primary disabilities

Speech or language impairment 49.8

Developmental delay 26.4

Autism 6.7

Mental retardation 4.0

Learning disability 2.5

Other health impairment 2.4

Orthopedic impairment 1.6

Emotional disturbance 1.1

Low incidence disability 5.5

NOTE Low incidence includes visual impairment, hearing impairment, deaf-blindness, traumatic brain injury, multiple disabilities.

SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), ‘Early Childhood and Kindergarten Teacher Questionnaires (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

Developing a Severity-of-Disability Scale

Severity

Under IDEA, children classified according to one of 13 disability categories

Children in each category have a range of abilities

Heterogeneity within categories often overlooked

Categorical labels especially problematic for preschoolers

Approaches to measurement

Pediatric health conditions (PEDI, WeeFim)

Self-care, mobility, and social cognition

Require a certain threshold of impairment

Single disability conditions in children

Hearing, vision, language, depression

ICIDH (WHO, 1980)

Published as a research tool

Not widely adopted for use

Approaches to measurement ABILITIES Index (Simeonsson & Bailey, 1991): Severity in 9 domains

Audition Behavior and social skills Intellectual function Limbs Intentional communication Tonicity Integrity of physical health Eyes Structural status

Approaches to measurement (cont’d) Since its original publication, 6 new domains added

(Simeonsson, 2006): Regulation of attention, Regulation of activity level, and Regulation feeling/emotions, Academic skills, Motivation, and Impulse control

ABILITIES index

In Special Education Expenditures Project (SEEP) (Chambers et al., 2004) :

Federal disability category able to explain only 10% of the variation in total expenditures

In contrast, the ABILITIES Index alone was able to account for 40% of the variation in total educational expenditures for special education students

Including the federal disability categories increased the variance accounted for by 2%

Question of current study

To what extent do functional markers of severity of childhood disability predict measurement of child outcomes?

Overall method

Identify items from parent interview similar to ABILITIES domains

Create composite domains where necessary Examine distribution of severity within the 15 domains Examine regressions of the 15 domains on multiple

outcomes Create short version of index using 6 domains Compare 6 domain and 15 domain versions of index to

outcomes to select better version Compare 6 domain version of index with other indicators

of severity

Outcome measures Cognitive

PPVT-III Woodcock-Johnson III: Letter-Word Identification Woodcock-Johnson III: Applied Problems

Social/Behavioral PKBS Social Competence PKBS Problem Behaviors

Alternate Assessment ABAS Conceptual Domain ABAS Practical Domain ABAS Social Domain

Wave 1 data

Creating the severity measure

Used parent report from Wave 1 CATI

Created 4-point scale for each domain (normal/typical, mild, moderate, severe)

11 domains already with 4 point scales:

4 levels of severity 4 domains with 3 point scales:

1 1, 2 2, 3 4 to preserve range

Creating the severity measure

Some domains matched to a single question Use of arms (within limbs domain): “How well

does {child} use {his/her} arms and hands for things like throwing, lifting, or carrying?”

Motivation: “Some children try to finish things, even if it takes a long time. How much does this sound like {child}… ”

For domains with multiple questions, worked with ABILITIES Index author to best reflect original intention of domain

Example: Combining Items to Create ‘Communication with Others’

When {CHILD} talks to people {he/she} doesn’t know well, is {he/she}1: Very easy to understand2: Fairly easy to understand3: Somewhat hard to understand4: Very hard to understand5: DOES NOT OR WILL NOT TALK AT ALL

Compared with other children about the same age, how well does {CHILD} make {his/her} needs known to you and others? Communication can be any form, for example crying, pointing or talking. Would you say {he/she}1: Communicates just as well as other children2: Has a little trouble communicating3: Has a lot of trouble communicating4: Does not communicate at all?

1: Communicates just as well as other children and very easy to understand2: Some difficulties communicating or being understood3: Moderate difficulties communicating or being understood4: Does not communicate at all or very hard to understand

ResultsPopulation weighted percentages (n = 2,986)

Audition, vision, use of arms, legs, regulation of emotions 80-95% in normal/typical category

Inappropriate or unusual behavior, overall health, social skills, use of hands, understanding 50-60% in normal/typical, 20-30% mild

ResultsPopulation weighted percentages (n = 2,986)

Motivation, regulation of attention, regulation of activity level, 20-30% in normal/typical, 33-42% mild, 16-

30% severe Communicating with others

29% in normal/typical, 9% mild, 45% moderate, 17% severe

Intellectual function 10% in normal/typical, 43% mild, 32%

moderate, 14% severe

Regressions on outcome variables All predictors entered simultaneously; examined beta

weights and bivariate correlations between predictors and each outcome

Significant predictors of at least three outcomes:cognition, communicating with others, understanding, overall health, and regulation of activity level

Significant predictor of two outcomes:regulation of attention

The remaining severity domains generated a mixture of significant and nonsignificant associations

No significant loading on any outcome variables: use of hands and arms and use of legs

Creation of indices

Index A, 15 items: Sum of all domains

Index B, 6 items: cognition, communicating with others, understanding, overall health, regulation of activity, regulation of attention

No significant differences in correlations between Index A and outcomes and Index B and outcomes

Index A and B correlations with outcomes

Index A15 var*

Index B6 var*

PPVT -.32 -.36

Letter-Word Identification -.22 -.26

Applied Problems -.40 -.45

PKBS Social Skills Composite -.47 -.43

PKBS Problem Behavior Composite .35 -.35

ABAS Conceptual Domain -.53 -.46

ABAS Practical Domain -.53 -.43

ABAS Social Domain -.40 -.35*The associated p-values are less than .0001 for all coefficients.

Final severity measure items

Cognition Communicating with others Understanding Overall health Regulation of activity Regulation of attention

Distribution of final severity measure

Validation: correlations between Index B and other indicators Wave 1 Parent report:

Age at which children began receiving special education or therapy servicer (2,802) = -.22, p < .0001

Wave 1 Teacher report: Amount of modification needed to curriculum materialsr (248) = .42, p < .0001

Wave 1 Teacher report: Number of services the child receives in school r (2,014)= .37, p < .0001

Validation: comparison of mean scores on severity measure

From teacher/parent declassification measure, Wave 1: Children remaining in special educationM = 13.2Children no longer receiving special education M = 10.7 , p < .0001

From assessment measures, Wave 1:Children taking the alternate assessment M = 16.3Children completing the direct assessmentM = 12.4, p < .0001

Summary

To what extent do functional markers of severity of childhood disability predict measurement of child outcomes?

Of the 15 domains examined, most were significant predictors of at least two outcomes

An index of only six variables was as effective as the longer version

Severity was significantly correlated with intervention variables

Severity differentiated children in two groups

Modeling Early Reading and Math Performance

Hierarchical analysis

PEELS has children in naturally-occurring hierarchies:

time points within children

children within districts

districts

What we want to know

What factors relate to children’s cognitive growth over 3 years?

What we have

observations of PEELS children over 3 years.

Yearly information about child’s SES, health, severity, and services received

Yearly measures of 3 academic outcomes

Adapted Peabody Picture Vocabulary Test (PPVT)

Woodcock-Johnson III: Letter-Word Identification

Woodcock-Johnson III: Applied Problems

Problem: How to make sense out of longitudinal data

Hierarchical data is clustered: i.e., repeated measures are not independent observations. Standard regression assumes independent observations.

Ordinary repeated measures analyses do not allow for missing time points or clustering.

Repeated cross-sectional analyses ignore the growth of individual children. Mean growth is not the same as growth of individuals.

Solution: Hierarchical linear modeling

Data is modeled at 3 levels of hierarchy at the same time.

Most of the clustering in the sample is accounted for, leading to correct statistical tests.

Focus is on individual growth profiles. Modeling seeks to explain differences in growth between children.

Hierarchical structure of data

District 1 District n Districts

S1 Si S1 SiStudentsW/in District

T1 T2 T3 T1 T2 T3

. . . . . . . .

3 time pointsper student

. . . .

T1 T2 T3T1 T2 T3

Level 3

Level 2

Level 1

HLM model

Level 1: Repeated observations within child

is the outcome for individual i measured at wave t .

is the age of the child at time at time t.

is the growth intercept: average achievement for the individual

is the growth curve slope: How much the outcome changes over years.

is a random error term.

jitY

, where0 1jit jit jitji ji

Y π π Time e

jitTime

0 jiπ

1 jiπ

jite

HLM modelLevel 2: Children nested within districts.

0 ji 00 j 0 ji

10 11 1p1ji 1ji pji 1ji

π = β + r

π = β +β X +...+ β X r

where,

0 jiπ is an intercept for child ji,

1 jiπ is the individual growth curve for child ji

1 jiX to pjiX are child factors that predict growth. 0 jir is the deviation from average achievement for each child.

1jir is the deviation from average growth for each child.

HLM modelLevel 3: Districts

Where,

is the achievement status of district j , is the grand mean of achievement, and is the district effect on achievement.

00 000 00 ,j jβ γ u

00 j

000ju00

Hypothetical growth curve HLMIndividual slopes estimated for high & low severity children

<- Low severity

<- High severity

District 1: Low Achievement & GrowthHigh vs. Low Severity Children

100

200

300

400

500

1 2 3

Wave of Data

Ac

hie

ve

me

nt

Individual slopes estimated for high & low severity children

<- Low severity

<- High severity

District 2: High Achievement & GrowthHigh vs. Low Severity Children

100

200

300

400

500

1 2 3

Wave of Data

Ach

ieve

men

t

Four Sets of Predictors

Predictors are added in four sets of similar variables. These sets include

SES: e.g., mother’s education, SES scale

Severity: e.g., severity scale, age services started

Health: e.g., child’s general health, health scale

Services: e.g., % time in a regular classroom, parent involvement scale

Predictors of PPVT growth

Accounted for 22 % of PPVT growth

Note: Controlled for age and cohort

Predictor Factor Effect Prob

Intercept 229.55 .00

Slope 54.24 .00

# Years in special education Severity -1.37 .01

Severity of disability scale Severity -.78 .00

Ease of transition Service -1.85 .01

% Time in a regular classroom

Service .03 .03

Predictors of Letter-Word growth

Accounted for 57 % of Letter-Word growth

Note: Controlled for age and cohort

Predictor Factor Effect Prob

Intercept 318.18 .00

Slope 37.76 .00

Years in high poverty school SES 2.10 .00

Mother had some education after HS SES 2.92 .00

# Years in special education Severity -1.62 .00

Age services started Severity .09 .02

Severity of disability scale Severity -.69 .00

Ease of transition Service -2.61 .00

% Time in regular classroom Service .08 .00

Parent involvement scale Service 1.49 .03

Predictors of Applied Problems growth

Predictor Factor Effect Prob

Intercept 393.26 .00

Slope 24.00 .00

Household Income SES -1.26 .03

Child is Hispanic SES 1.53 .03

Mother had some education after HS SES 1.42 .02

Parent SES scale SES 1.38 .01

Problems with health Health 2.50 .00

Health scale Health 1.18 .05

Child’s general health Health 2.49 .00

Predictors of Applied Problems growth (continued)

Accounted for 12 % of Applied Problems growth

Note: Controlled for age and cohort

Predictor Factor Effect Prob

# Years in special education Severity -1.02 .00

Age services started Severity .09 .00

Severity of disability scale Severity -.24 .02

% Time in regular classroom Service .03 .00

Parent involvement scale Service .75 .06

Summary

Scores increased from 1½ to 2 standard deviations over 3 years

SES, severity, health, & service predictors accounted for 12% to 57% of growth

Percent of growth accounted for

PPVT 22%

Letter-Word 57%

Applied Problems 12%

Summary (cont.)

Service-related predictors of growth

% Time in a regular classroom

Positively related to growth for all outcomes

Parent involvement

Positively related for 2 of 3 outcomes

(Letter-Word & Applied Problems)

Ease of transition

Positively related for 2 of 3 outcomes

(PPVT & Letter-Word)

Summary (cont.)

Other predictors of growth # Years child was in special education

Predicted lower growth for all outcomes

Parent’s rating of severity

Predicted lower growth for all outcomes

Mother’s education

Positively related to growth for 2 of 3 outcomes

(Letter-Word & Applied Problems)

WEBSITE: WWW.PEELS.ORG