Demystifying the Regression Coefficient: Rethinking a Complex
Tool for Use in Policy ResearchJeffrey S. Napierala
Prof. Glenn D. DeaneDepartment of Sociology, SUNY Albany
Prof. Donald J. HernandezDepartment of Sociology, Hunter College & CUNY Graduate Center
Research was supported by a Grant from the Annie E. Casey FoundationPlease do not cite or distribute this research without consent from the
Authors
OutlineThe origins of this methodAn example using children's reading
performanceA “hybrid” approach
Standard ErrorsResultsConclusions
OriginsThe audience for policy research often has a
diverse background in statistics.To accommodate those without any proficiency,
the presentation must be simple and transparent.
To satisfy those with advanced proficiency, current methods must be used.
We want to have our cake and eat it too…
OriginsAt one end of the spectrum: Sample
means/proportions are simple and easy to understand …but may not clearly translate into effective policies…
Towards the other end: A standard regression approach has obvious advantages, but results can be difficult to explain to general audiences.
Origins
Hernandez, Donald J. Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation. Baltimore: Annie E. Casey Foundation
An Example…Policy Question: How much would income
transfers increase the percentage of (3rd Grade) students reading at the proficient level?
Data: National Longitudinal Survey of Youth (NLSY)4,060 children in 2369 families followed across about
30 yearsDependent variable: Peabody Individual
Achievement (PIAT) Reading Comprehension Test. Continuous variable ranging from 1-99.
Methods: Weighted, linear GEE models with a correction for clustering within families.
An Example: Model SpecificationTwo models to highlight approach:
1. A “Base” model with just income and income squared predicting PIAT Reading Comprehension.
2. A “Full” model with controls for the 1) non-linear effect of mother’s education, 2) health insurance coverage, 3) whether a child attended head start or preschool, 4) the “quality” of their neighborhood, 5) race, 6) sex, and 7) year of interview.
A “hybrid” ApproachLet’s utilize the power of a regression, but
keep the presentation as simple and flexible as possible.
1. Create a statistical model of the outcome.2. “Simulate” new outcomes using the relevant
parameters of the model.3. Meaningfully summarize the distributions
before and after the “simulation” using simple statistics/tabulations.
4. Compare the summary statistics/tabulations.
A “hybrid” Approach: Standard ErrorsUsing the common formula for the standard error
of a proportion (with and without a Design Effect multiplier of 1.388).
A Monte-Carlo approach to incorporate error from the sample regression and sample proportion.First, the effect(s) of covariates are added into the
original score (the raw DV) by sampling from a normal distribution with a mean and s.d. from the regression.
Rates (of reading proficiency) are computed then additional sampling error is introduced.
After 5000 iterations, the S.E. of the distribution is computed from all the rates.
Also, to compare rates from the same group of children (before and after “simulations”) a “paired proportions” t-test is used (Altman 1997).
Source: Altman, Douglas G. 1997. Practical Statistics for Medical Research. London: Chapman & Hall.
A “hybrid” Approach: Standard Errors
Results…
$0 $2 $4 $6 $8 $10 $1210
20
30
40
50
60
70
80Predicted and Actual Values of PIAT Reading Score by
Income
Sample Mean
"Base" Model
"Full" Model
Family Income in 10's of thousands
PIA
T R
ead
ing
Score
Results…
2.5
12.5
22.5
32.5
42.5
52.5
62.5
72.5
82.5
92.5
0
5000
10000
15000
20000
25000
30000
35000
40000Distribution of Low Income Children by PIAT Reading
Score
Sample Mean
"Full" Model
PIAT Scores
Nu
mb
er
of
Ch
ild
ren
(in
1
,00
0's
)
Results…
5 15 25 35 45 55 65 75 85 95
-30,000
-25,000
-20,000
-15,000
-10,000
-5,000
0
5,000
10,000
15,000
20,000
Change in Distribution of Low Income Children by PIAT Reading Score after “Full” Model Simulation
PIAT Score
Ch
an
ge i
n N
um
ber
of
Ch
ild
ren
(i
n 1
,00
0's
)
Results…
1st Quintile 2nd Quintile 3rd Quintile 4th Quintile 5th Quintile
Sample Means 16.39 23.3 34.1 39.71 45.54
"Base" Model Simula-tion
NaN 23.42 29.96 36.57 42.66
"Full" Model Simulation NaN 20.51 22.73 25.83 27.65
13
18
23
28
33
38
43
48
Effect of Income on Reading Proficiency using Sample Means and Simulated Outcomes
Perc
en
t R
ead
ing
Pro
ficie
ntl
y
Results…
1st Quintile 2nd Quintile 3rd Quintile 4th Quintile 5th QuintileSample Percent 1.14 1.25 1.29 1.35 1.38"Simulated" 1.15 1.27 1.31 1.40 1.40Sample Percent w/ DEFT 1.58 1.73 1.80 1.88 1.92"Simulated" w/ DEFT 1.58 1.75 1.82 1.89 1.91
Standard Errors for Percent Reading Proficiently in 3rd Grade for Children in Poverty - "Full" Model
Results…Q2 - Q1 Q3-Q2 Q4-Q3 Q5-Q4
Difference 4.21 2.22 3.1 1.82S.E. 0.62 0.45 0.54 0.41
S.E. w/ DEFT 0.86 0.63 0.74 0.57
Difference Between Rates in "Full" Model Simulation
ConclusionsThe “hybrid” approach has a few notable
advantages over other methods…The independent effect of income on reading
proficiency is much (much) less than might be expected from looking at bivariate or univariate results.
We expect that about 4% (2.5-5.9; 95% C.I.) more kids in poverty would read proficiently if their families were given additional income to move them out of povery.
Thank You!Email: [email protected]
Recommended
