INFORMING EARLY CHILDHOOD POLICY: AN ANALYSIS OF THE SENSITIVITY OF A

SCHOOL READINESS RISK INDEX TO CHANGES IN INDICATOR SELECTION

KRISTA S. SCHUMACHER, PHD CANDIDATEAPRIL 3, 2015

2

PURPOSE Examine sensitivity of Oklahoma School Readiness Risk Index (OK SRRI) to changes to the

indicator set

OK SRRI Originally published 2013 by Oklahoma Department of Human Services Revised in 2014 as part of Oklahoma School Readiness Reach-by-Risk Report Ranks counties according to risk for children starting school unprepared using 10 indicators Rankings compared to availability of early education and child care Intended use:

Inform decisions regarding early childhood policies and resource allocation

3

OK SRRI INDICATORS 18 indicators considered

race/ethnicity, family and economic characteristics, health Final set of 10 selected using principal components analysis (PCA) and OLS

multiple regression Indicators organized into domains via PCA results Index scores: mean z-scores for individual indicators, equal weighting

Hispanic Background Family Structure / Economic Distress

Children in Child Welfare

Hispanic ethnicity Poverty (<100% FPL) Abuse/neglect

English-language learners (ELL) Single parent Foster care

Migrant Young maternal age (< age 20)

Low maternal education (no high school diploma)

American Indian race

Excluded indicatorsBehind on vaccinations

Tobacco use during pregnancyLate or no prenatal care

Low birth weight

Black race

Homeless

IEP

Kindergarten retention

4

ALTERNATE INDEX Alternate OK SRRI created based on original set of 18 indicators

Goals for alternate index: 1. Indicators can be addressed through interventions and/or public policy2. Theoretically based: Transactional/ecological theory of child development and cumulative risk model3. Full representation of multidimensional construct of school readiness risk4. Follow formative measurement model

Final indicators selected by analyzing indicator descriptive statistics, correlationsReady Families Ready Communities Ready Services - Health

Single parent Poverty (<100% FPL) Behind on vaccinations

Low maternal education ELL Tobacco use during pregnancy

Young maternal age Late or no prenatal care

Abuse/neglect Low birth weight

Foster care

Excluded indicators

Hispanic ethnicity

American Indian race

Black race

Migrant

Homeless

IEP

Kindergarten retention

5

MEASUREMENT MODELS: REFLECTIVE & FORMATIVE

Reflective Scale development to measure latent variable, e.g.., extroversion, happiness

Latent variable exists independently of its indicators Indicators must be highly correlated for unidimensional scale Error can be extracted

Formative Index development to measure composite variable, e.g., school readiness risk

Composite variable depends on the indicators Indicators form a multidimensional construct Highly correlated indicators = construct underrepresentation Error cannot be extracted (considered disturbance term)

Extroversion

Social

Talkative

Enthusiastic

Risk

Poverty

ELL

Teen mother

6

INDICATOR CORRELATIONS AND OUTLIERS # outliers

American Indian 1Migrant 6Hispanic 7ELL 10Poverty 5Single parent 0Young maternal age 0Low maternal education 3Abuse/neglect 3Foster care 1Vaccinations 1Birth weight 1Prenatal care 2Tobacco use 2

7

METHODS1. Indicator and domain dominance

• Pearson’s r correlations of indicator and domain z-scores with overall index scores• Multiple regression commonality analysis

2. Relative effect of indicators and domains on outcome rankings• Excluded one indicator or domain at a time from index score calculations• Examined rankings (percentiles) on reduced scenarios compared to full index

• ≥ 15 percentiles = moderate shift; ≥ 20 percentiles = significant shift

• Compared rankings across full original and alternate indexes

3. Associations of index rankings with a proxy outcome of school readiness risk• Spearman rank-order correlations of full and reduced indexes to percent of kindergartners

scoring below proficient on pre-literacy assessment

8

FINDINGS: DOMINANCE ANALYSES Correlations

One negative association between indicators and scores on alternate index (vaccinations) One negative association between domains on both indexes Families domain dominated alternate index

Commonality analysis Little unique contributions of indictors to either index

Vaccinations and tobacco use indicators small confounding effect (negative shared effects) Domains

More unique effects than indicators Largest unique and total effects

Families domain (alternate index); Family Structure/Economic Distress domain (original index)

Four indicators had largest correlations and explained variance in both indexes Low maternal education, young maternal age, poverty, single parent

Changes in indicator impact from original to alternate indexes Abuse/neglect, foster care and teen mothers increased Low maternal education decreased marginally ELL decreased considerably

9

FINDINGS: EFFECTS ON RANKINGSRange of median rank shifts on each reduced

scenario

# counties shifting ≥ 15 percentiles on each

reduced scenario

# counties shifting ≥ 15 percentiles across all

reduced scenariosIndicators

Original 1.3 to 3.9 1 to 3 12

Alternate 2.6 to 3.9 3 to 7 26

Domains

Original 5.2 to 7.8 11 to 24 37

Alternate 5.2 to 18.2 10 to 41 53

10

FINDINGS: ORIGINAL & ALTERNATE INDEX RANK COMPARISONS

rs = .88 Median rank shift = 8 percentiles 26 counties shifted by ≥ 15 percentiles, 16 by ≥ 20 percentiles

Even split of number increasing and decreasing Trend of shifts from original to alternate indexes

Increases: Relatively moderate to low rankings on original index Several counties with high scores on at least one health indicator

Decreases: Relatively high rankings on original index Several counties with high scores on racial/ethnic indicators

Changes to risk groupings occurred for 22 counties

11

FINDINGS: ASSOCIATIONS WITH LITERACY SKILLS

Original and alternate indexes moderately correlated with literacy skills rs = .20 and .21, respectively

Associations with indicator-reduced scenarios Original: rs = .17 (single parent) to .24 (Hispanic) Alternate rs = .15 (single parent) to .24 (vaccinations)

Associations with domain-reduced scenarios Original: rs = .08 (Family Structure/Economic Distress) to rs = .26 (Hispanic Background) Alternate: rs = .01 (Ready Families) to rs = .24 (Ready Services – Health)

12

CONCLUSIONS OK SRRI sensitive to changes to indicator set The same indicators can behave differently when used with varying sets of indicators

E.g., abuse/neglect, foster care, ELL Indicators with greatest impact on both indexes

Low maternal education, young maternal age, poverty and single parent Use of highly correlated indicators results in double counting

More critical with indicators of greater variability and numerous outliers Reliance on statistical methods of indicator selection underrepresented the construct

More problematic with population-level data Ecological fallacy Differential behavior of correlations

Domains must be organized conceptually and unambiguously to fulfill intended purpose Identify areas of concern and promote appropriate responses

13

IMPLICATIONS

Choice of indicators matters Less sensitivity of original index to changes to indicator set a result of:

High correlations Fewer dimensions of school readiness risk represented Is sensitivity to changes to indicator set necessarily a bad thing?

Construct of school readiness risk highly dependent on indicators used Different sets of indicators can significantly change construct definition Need consensus regarding construct definition, construct domains and issues of relevance

Selecting indicators should be participatory process Indexes inherently political in nature

Excluding part of construct from index excludes it from consideration regarding policy and resources Indexes have power to direct resources toward or away from counties with particular characteristics

“The act of deciding what to count is value oriented and subjective in nature” (Simpson, 2006, p. 5).