Embed Size (px)
DESCRIPTION
Estimated coefficients from an OLS model of birth weight in grams Model A: Without interactions Model B: With interactions βt- statistic β Main effects terms Race (ref. = non-Hisp. white) Non-Hispanic Black (NHB)–172.6**–9.86–168.1**–5.66 Mexican American (MA)–23.1–1.02–104.2**–2.16 Mother’s ed. (ref. = >HS) Less than high school (
Citation preview
Approaches to testing statistical significance of interactions
Jane E. Miller, PhD
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Overview• Review: Testing statistical significance of individual
model terms• The TEST statement• All-interaction-term dummies model specification • Introduction to post-hoc probing of interaction
effects– Simple slopes calculations for compound coefficients– Changing the reference category
• Presenting the results of statistical tests of interaction pattern
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Estimated coefficients from an OLS model of birth weight in gramsModel A: Without
interactionsModel B: With
interactionsβ t-
statisticβ t-
statisticMain effects termsRace (ref. = non-Hisp. white)
Non-Hispanic Black (NHB) –172.6** –9.86 –168.1** –5.66Mexican American (MA) –23.1 –1.02 –104.2** –2.16
Mother’s ed. (ref. = >HS) Less than high school (<HS) –55.5** –2.88 –54.2** –2.35
High school graduate (=HS) –53.9** –3.64 –62.0** –3.77Interactions: race & education
NHB_<HS –38.5 –0.88MA_<HS 99.4 1.72NHB_=HS 18.4 0.47MA_=HS 93.7 1.49
F-statistic 94.08 65.59Degrees of freedom (df) 9 13
Statistical significance of βs on individual interaction terms
• Statistical significance of coefficients on each of the interaction terms is assessed as for any other independent variable in a multivariate regression model
• In the example from the previous slide, none of the βs on the interaction terms between race/ethnicity and mother’s education achieve statistical significance as assessed by their t-statistics – E.g., βNHB_<HS = –38.5, with a t-statistic of –0.88
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
What don’t inferential statistics for individual terms in a model tell us?
• Based on the separate test statistics for each of the individual main effect and interaction s alone cannot assess statistical significance of differences in predicted birth weight – For example, for non-Hispanic blacks born to mothers with < HS
compared to non-Hispanic whites born to mothers with > HS (the reference category)
– across racial/ethnic groups within the < HS group– across education levels among non-Hispanic blacks
• Remember: each of these comparisons involves comparing values calculated from more than one
<HS + NHB + <HS_NHB
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Substantive question behind the interaction model: “Does race modify the association between education
and birth weight?”
• The bar for each race/education combination involves the sum of the intercept and one to three other coefficients
• t-tests for individual βs won’t tell us about statistical significance of differences in those sums
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Testing differences between s
• To formally test statistical significance of differences between coefficients, e.g., H0: βj = βk, calculate the test statistic:– Divide the difference between the estimated coefficients
(j − i) by the standard error of the difference– Compare the value of the test statistic against the critical
value with one degree of freedom
• See podcast on testing statistical significance of differences across coefficients
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Standard error of the difference• The standard error of the difference is calculated:
√[var(j) + 2 × cov(j, k) + var(k) ]– var(j) and var(k) are the variances of j and k,
respectively– cov(j, k) is the covariance between j and k
• The complete variance-covariance matrix for regression coefficients can be requested as part of the output
• The variance of each coefficient can be calculated from its standard error (s.e.): var(j) = [s.e.(j)]2
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Square root
TEST statement
• The test statistics for each β tests whether it is statistically significantly different from the reference category
• To test other contrasts among categories, request the test statistic for equality of coefficients for pairs of βs– E.g., to test whether predicted birth weight is statistically
significantly different for infants born to mothers with < HS than for those born to mothers with = HS education
• Specify “TEST <HS = =HS” in your SAS syntax• Compare the value of that test statistic against the critical value
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Evaluating whether interaction specification could be simplified
• For a specification that involves several main effects and interaction terms, might be able to simplify the specification if some terms can be omitted
• E.g., for a three-category variable, might it be possible to– Combine one of the modeled categories with the
reference category?– Combine the two modeled categories with one another?
• Use standard approaches for comparing fit of models with different specifications based on model GOF
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Respecify the model with simplified classifications of categorical variable
• Create a dummy variable for ≤HS – Combines former categories of < HS and = HS– Specification will compare those two education levels
together against the reference category (> HS)
• Compare the overall fit of models– With two separate dummies for <HS and =HS– With one dummy for ≤HS
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Caveat about combining categories• Only combine categories for which it makes
substantive sense to do so. E.g.,– <HS and >HS aren’t adjacent ordinal categories, so you
would NOT combine them with one another to compare against =HS
– For some research questions, you could combine non-Hispanic blacks with Mexican-Americans because both are considered racial/ethnic minority groups in the United States
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Simplifying the interaction terms• If there is no statistically significant difference in
model GOF for– the model with collapsed education main effects – the model with detailed education classification
• Could estimate a model with simplified interaction terms:o ≤HS_NHB instead of <HS_NHB and =HS_NHBo ≤HS_MA instead of <HS_MA and =HS_MA
• Compare the fit of that model against the fit of the model with detailed education by race interactions.
• Choose parsimonious model based on model GOFThe Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Model specification with individual dummies for all interaction combinations
• An alternative for categorical-by-categorical interactions is to– Create a dummy variable for each interaction
combination except the reference category– Estimate a model with all of those dummies but
NOT main effects terms for the variables involved in the interaction
– Then request a formal inferential test of differences among pairs of interaction terms
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Example: Race by education interaction
• The all-interaction-term specification for a model of birth weight can be written:y = β0 + β1<HS_NHB + β2<HS_NHW + β3<HS_MA + β4=HS_NHB +
β5HS_NHW + β6=HS_MA + β7>HS_NHB + β8>HS_MA + βiXi,
• No main effects terms needed because these eight dummies distinguish among all nine possible combinations of race and education
Race
Mother’s educational attainment
< HS = HS > HSNon-Hispanic black <HS_NHB =HS_NHB >HS_NHB
Mexican-American <HS_MA =HS_NHW >HS_MA
Non-Hispanic white <HS_NHW =HS_MA Reference category
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Example: Testing differences across groups other than the reference category
• The test statistics for β<HS_NHB and β<HS_MA each test whether predicted birth weight is statistically significantly different for that group compared to > HS & NHW (the reference category)
• To contrast other combinations of race and education, request the test statistic for equality of βs– E.g., to test whether predicted birth weight is statistically
significantly different for non-Hispanic black with < HS than for Mexican American infants of mothers with < HS
• In SAS: “TEST <HS_NHB = <HS_MA”• Compare the value of that test statistic against the critical value
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Presenting results of statistical tests for interactions
• Much of the work for testing statistical significance of interactions should be done behind the scenes
• Create a table of is, standard errors, and model goodness-of-fit statistics
• Supplement it with a table or chart reporting the predicted values of the dependent variable for pertinent values of the IVs in the interaction, calculated from the is– Use symbols to denote which values are statistically
significantly different from one another
Chart to present results of statistical significance testing of interactions
* denotes statistically significantly different at p < 0.05 from non-Hispanic white > HS† denotes statistically significantly different at p < 0.05 from non-Hispanic black < HS£ denotes statistically significantly different at p < 0.05 from non-Hispanic black = HS¥ denotes statistically significantly different at p < 0.05 from Mexican-American = HS
*†
*†£
*†
*¥
*¥
* † ¥
*£
†£¥
Summary• In models using main effects and interaction terms,
calculating the overall shape of an interaction requires summing several βs– Tests of the βs don’t address statistical significance of differences in
the overall interaction pattern
• Approaches to addressing statistical significance of interactions include– Comparing goodness-of-fit of models with and without interactions– Using an all-interaction-dummy specification and test of difference
across coefficients– Using simple slopes techniques for post-hoc comparisons– Changing the reference category to test different contrasts
Separate podcastsThe Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested resources• Miller, J. E. 2013. The Chicago Guide to Writing about
Multivariate Analysis, 2nd Edition. University of Chicago Press, chapters 11, 15, and 16.
• Cohen, Jacob, Patricia Cohen, Stephen G. West, and Leona S. Aiken. 2003. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, 3rd Edition. Florence, KY: Routledge, chapters 8 and 9.
• Figueiras, Adolfo, Jose Maria Domenech-Massons, and Carmen Cadarso. 1998. Regression Models: Calculating the Confidence Interval of Effects in the Presence of Interactions. Statistics in Medicine 17: 2099–2105.
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested online resources• Podcasts on
– Testing statistical significance of differences between coefficients
– Testing whether a multivariate specification can be simplified
– Calculating the overall shape of an interaction from regression coefficients
– Conducting post-hoc tests of compound coefficients using simple slopes
– Using alternative reference categories to test statistical significance of an interaction
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
Suggested exercises: Simplifying an interaction specification
• Estimate an OLS model with an interaction between a three-category independent variable (IV1) and a two-category independent variable (IV2)
• Test whether the βs on the main effects terms for the two included categories of IV1 are statistically significantly different – From the reference category– From each other
• Consider those results in conjunction with theoretical criteria to decide whether it makes sense to combine categories
Exercise: Simplifying an interaction, cont.
• Re-specify the model combining the two IV1 categories
• Compare GOF with your initial model• Calculate revised interaction terms with the
simplified IV1 specification• Estimate a model with simplified interaction terms• Compare GOF with the more detailed interaction
specification
Suggested exercises: All-interaction-dummies specification
• For an interaction between a three-category independent variable (IV1) and a two-category independent variable (IV2), create dummy variables for each possible combination of IV1 and IV2 except the reference category
• Specify a model with those terms – Remember NOT to include main effects terms
• Use the TEST statement to test statistical significance of differences in βs – Across categories of IV1 within strata defined by IV2– Across categories of IV2 within strata defined by IV1
Contact information
Jane E. Miller, [email protected]
Online materials available athttp://press.uchicago.edu/books/miller/multivariate/index.html
The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition.
