Sensitivity Analysis: Quantifying the Discourse About Causal Inference Kenneth A. Frank Help from Yun-jia Lo and Mike Seltzer AERA workshop April 4, 2014

Slide 1

What would it take to Change your Inference? Quantifying the Discourse about Causal Inferences in the Social Sciences Kenneth A. Frank Help from Yun-jia Lo and Mike SeltzerAERA workshop April 4, 2014 (AERA on-line video cost is $95)Motivation Statistical inferences are often challenged because of uncontrolled bias. There may be bias due to uncontrolled confounding variables non-random selection into a sample. We will answer the question about what it would take to change an inference by formalizing the sources of bias and quantifying the discourse about causal inferences in terms of those sources. For example, we will transform challenges such as But the inference of a treatment effect might not be valid because of pre-existing differences between the treatment groups to questions such as How much bias must there have been due to uncontrolled pre-existing differences to make the inference invalid?

ApproachesIn part I we will use Rubins causal model to interpret how much bias there must be to invalidate an inference in terms of replacing observed cases with counterfactual cases or cases from an unsampled population. In part II, we will quantify the robustness of causal inferences in terms of correlations associated with unobserved variables or in unsampled populations. Calculations for bivariate and multivariate analysis will be presented in the spreadsheetfor calculating indices [KonFound-it!] with some links to SPSS, SAS, and Stata. FormatThe format will be a mixture of presentation, individual exploration, and group work. Participants may include graduate students and professors, although all must be comfortable with basic regression and multiple regression. Participants should bring their own laptop, or be willing to work with another student who has a laptop. Participants may choose to bring to the course an example of an inference from a published study or their own work, as well as data analyses they are currently conducting.

Materials to download: spreadsheetfor calculating indices [KonFound-it!] powerpointwith examples and calculations

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum overviewReplacement Cases Framework (40 minutes to reflection)Thresholds for inference and % bias to invalidate and inferenceThe counterfactual paradigmApplication to concerns about non-random assignment to treatmentsApplication to concerns about non-random sample Reflection (10 minutes)Examples of replacement frameworkInternal validity example: Effect of kindergarten retention on achievement (40 minutes to break)External validity example: effect of Open Court curriculum on achievement Review and ReflectionExtensionsExtensions of the frameworkExercise and break (20 minutes: Yun-jia Lo, Mike Seltzer support)Correlational Framework (25 minutes to exercise) How regression worksImpact of a Confounding variableInternal validity: Impact necessary to invalidate an inferenceExample: Effect of kindergarten retention on achievementExercise (25 minutes: Mike Seltzer supports on-line)External validity (30 minutes)combining estimates from different populationsexample: effect of Open Court curriculum on achievementConclusion (10 minutes)

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

https://www.msu.edu/~kenfrank/research.htm#causalMaterials for courseReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quick SurveyCan you make a causal inference from an observational study?Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Answer: Quantifying the DiscourseCan you make a causal inference from an observational study?

Of course you can. You just might be wrong. Its causal inference, not determinism.

But what would it take for the inference to be wrong?Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Interpretation of Amount of Bias Necessary to Invalidate the Inference: Sample RepresentativenessTo invalidate the inference:54% of the estimate must be due to sampling bias to invalidate Borman et al.s inferenceYou would have to replace 54% of Bormans cases (about 30 classes) with cases in which Open Court had no effect to invalidate the inferenceAre 54% of Borman et al.s cases irrelevant for non-volunteer schools?We have quantified the discourse about the concern of external validity

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum What Would It Take to Change an Inference? Using Rubins Causal Model to Interpret the Robustness of Causal Inferences

Abstract We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubins causal model (RCM) to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse. Keywords: causal inference; Rubins causal model; sensitivity analysis; observational studies Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013.What would it take to Change an Inference?:Using Rubins Causal Model to Interpret the Robustness of Causal Inferences.Education, Evaluation and Policy Analysis.Vol35:437-460.http://epa.sagepub.com/content/early/recent

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Maybe put unpublished version on website

7

How Regression Works: Partial Correlation

Partial Correlation: correlation between s and y, where s and y have been controlled for the confounding variable

Spss syntaxCORRELATIONS /VARIABLES=y s confound /PRINT=TWOTAIL NOSIGSas syntaxtproc corr data=one;var y s confound;run;

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum CORRELATIONS /VARIABLES=y s confound /PRINT=TWOTAIL NOSIG /MISSING=PAIRWISE.proc corr data=one;var y s confound;run;

92Quantifying the Discourse: FormalizingBias Necessary to Invalidate an Inference =a population effect, =the estimated effect, and# =the threshold for making an inference

An inference is invalid if: (1)

An inference is invalid if the estimate is greater than the threshold while the population value is less than the threshold.

Defining bias as -, (1) implies an estimate is invalid if and only if:

Expressed as a proportion of the estimate, inference invalid if:



# {}% bias necessary to invalidate the inference

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Robustness and Replicability

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Original study effect size versus replication effect size (correlation coefficients).

Open Science Collaboration Science 2015;349:aac4716Published by AAAShttps://osf.io/ezcuj/wiki/home/dataOriginal study effect size versus replication effect size (correlation coefficients). Diagonal line represents replication effect size equal to original effect size. Dotted line represents replication effect size of 0. Points below the dotted line were effects in the opposite direction of the original. Density plots are separated by significant (blue) and nonsignificant (red) effects.Interpretation of % Bias to Invalidate an Inference% Bias is intuitiveRelates to how we think about statistical significanceBetter than highly significant or barely significantBut need a framework for interpretingReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Framework for Interpreting % Bias to Invalidate an Inference: Rubins Causal Model and the CounterfactualI have a headacheI take an aspirin (treatment)My headache goes away (outcome)

Q) Is it because I took the aspirin?Well never know it is counterfactual for the individualThis is the Fundamental Problem of Causal InferenceReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 15Holland 1986 JASA.Example of Internal Validity from Observational Study : The Effect of Kindergarten Retention on Reading and Math Achievement(Hong and Raudenbush 2005)1. What is the average effect of kindergarten retention policy? (Example used here)

Should we expect to see a change in childrens average learning outcomes if a school changes its retention policy?

Propensity based questions (not explored here)2. What is the average impact of a schools retention policy on children who would be promoted if the policy were adopted?

Use principal stratification.

Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Childrens Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205224

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 26Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Childrens Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205224See page 208.47Application to Randomized Experiment: Effect of Open Court Curriculum on Reading AchievementOpen Court scripted curriculum versus business as usual917 elementary students in 49 classroomsComparisons within grade and schoolOutcome Measure: Terra Nova comprehensive reading score

Borman, G. D., Dowling, N. M., and Schneck, C. (2008). A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389-407.

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 47Extensions of the FrameworkOrdered thresholds for decision-makingAlternative hypotheses and scenariosRelationship to confidence intervalsRelated techniquesReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum II: Correlation FrameworkConcerns about internal validityAt what levels must an omitted variable be correlated with a predictor and outcome to change an inference about the predictor?Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194

Concerns about external validityWhat must be the correlation between predictor and outcome in the unsampled population to invalidate an inference about an effect in a population that includes the unsampled population?*Frank,K. A.and Min, K. 2007. Indices of Robustness for Sample Representation.Sociological Methodology.Vol37, 349-392. *cofirst authors.Combined application (with propensity scores)Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, MarisaCannata, Linda Chard, Ann Krause, RavenMcCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers.Education, Evaluation, and Policy Analysis.Vol30(1): 3-30.Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Using the Counterfactual to Interpret % Bias to Invalidate the InferenceHow many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (# =4):1- # /

=1-4/6=.33 =(1/3)666 6.00

The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4

000

6434510119Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 20Which Cases to ReplaceThink of it as an expectation: if you randomly replaced 1 case, and repeated 1,000 times, on average the new estimate would be 4Assumes constant treatment effectconditioning on covariates and interactions already in the modelAssumes cases carry equal weightReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum


To invalidate the inference, replace 33% of cases with counterfactual data with zero effectReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Fundamental Problem of Inference and Approximating the Unsampled Population with Observed Data (External Validity)910 Yt|Z=p1134 Yc|Z=p566 6 664How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: # =4:1- # /

=1-4/6=.33 =(1/3)

6 Yt|Z=p6 Yc|Z=p0Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 23


To invalidate the inference, replace 33% of cases with cases from unsampled population data with zero effectReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Review & ReflectionReview of FrameworkPragmatism thresholdsHow much does an estimate exceed the threshold % bias to invalidate the inferenceInterpretation: Rubins causal modelinternal validity: % bias to invalidate number of cases that must be replaced with counterfactual cases (for which there is no effect)external validity: % bias to invalidate number of cases that must be replaced with unobserved population (for which there is no effect)ReflectWhich part is most confusing to you?Is there more than one interpretation?Discuss with a partner or two

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum DataEarly Childhood Longitudinal Study Kindergarten cohort (ECLSK)US National Center for Education Statistics (NCES). Nationally representativeKindergarten and 1st gradeobserved Fall 1998, Spring 1998, Spring 1999 Student background and educational experiencesMath and reading achievement (dependent variable)experience in classParenting information and styleTeacher assessment of studentSchool conditionsAnalytic sample (1,080 schools that do retain some children)471 kindergarten retainees 10,255 promoted studentsReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Effect of Retention on Reading Scores(Hong and Raudenbush)

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Possible Confounding Variables(note they controlled for these)GenderTwo Parent HouseholdPovertyMothers level of Education (especially relevant for reading achievement)Extensive pretestsmeasured in the Spring of 1999 (at the beginning of the second year of school) standardized measures of reading ability, math ability, and general knowledge; indirect assessments of literature, math and general knowledge that include aspects of a childs process as well as product;teachers rating of the childs skills in language, math, and science

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference:Obtain spreadsheetFrom https://www.msu.edu/~kenfrank/research.htm#causalChoose spreadsheet for calculating indices

Access spreadsheetspreadsheetfor calculating indices [KonFound-it!]Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Kon-Found-it: Basics


Obtain t critical, estimated effect and standard errorEstimated effect( ) = -9.01Standard error=.68

n=7168+471=7639;df > 500,t critical=-1.96From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Childrens Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205224

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum covariates

Page 215

Df=207+14+2=223Add 2 to include the intercept and retentionReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Kon-Found-it: Basics% Bias to InvalidateEstimated effect( ) = -9.01

Standard error=.68n=7168+471=7639Covariates=223Specific calculationsReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Calculating the % Bias to Invalidate the Inference:Inside the Calculations# =the threshold for making an inference = se x tcritical, df>230 =.68 x -1.96=-1.33[user can specify alternative threshold]

% Bias necessary to invalidate inference = 1-#/ =1-1.33/-9.01=85%

85% of the estimate must be due to bias to invalidate the inference.

}Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Using the Counterfactual to Interpret % Bias to Invalidate the InferenceHow many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (# =4):1- # /

=1-4/6=.33 =(1/3)666 6.00

The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4

000

6434510119Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 36

Original cases that were not replacedReplacement counterfactual cases with zero effectOriginal distributionRetainedPromotedExample Replacement of Cases with Counterfactual Data to Invalidate Inference of an Effect of Kindergarten Retention Counterfactual:No effect

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Interpretation1) Consider test scores of a set of children who were retained that are considerably lower (9 points) than others who were candidates for retention but who were in fact promoted. No doubt some of the difference is due to advantages the comparable others had before being promoted. But now to believe that retention did not have an effect one must believe that 85% of those comparable others would have enjoyed most (7.2) of their advantages whether or not they had been retained.

This is even after controlling for differences on pretests, mothers education, etc.

2) The replacement cases would come from the counterfactual condition for the observed outcomes. That is, 85% of the observed potential outcomes must be unexchangeable with the unobserved counterfactual outcomes such that it is necessary to replace those 85% with the counterfactual outcomes to make an inference in this sample. Note that this replacement must occur even after observed cases have been conditioned on background characteristics, school membership, and pretests used to define comparable groups.

3) Could 85% of the children have manifest an effect because of unadjusted differences (even after controlling for prior achievement, motivation and background) rather than retention itself?

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Evaluation of % Bias Necessary to Invalidate InferenceCompare bias necessary to invalidate inference with bias accounted for by background characteristics1% of estimated effect accounted for by background characteristics (including mothers education), once controlling for pretestsMore than 85 times more unmeasured bias necessary to invalidate the inferenceCompare with % bias necessary to invalidate inference in other studiesUse correlation metricAdjusts for differences in scaleReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum % Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies

t taken from HLM: =-9.01/.68=-13.25n is the sample size q is the number of parameters estimated

Where t is critical value for df>200

% bias to invalidate inference=1-.023/.152=85%Accounts for changes in regression coefficient and standard errorBecause t(r)=t()Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Kon-Found-it: Basics% Bias to InvalidateEstimated effect( ) = -9.01

Standard error=.68n=7168+471=7639Covariates=223Specific calculationsReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

% Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies

% bias to invalidate inference=1-.023/.152=85%Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Compare with Bias other Observational StudiesReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 44

% Bias to Invalidate Inference for observational studieson-line EEPA July 24-Nov 15 2012Kindergarten retention effectReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 1 : % Bias necessary to Invalidate an Inference for Internal ValidityTake an example from an observational study in your own data or an articleCalculate the % bias necessary to invalidate the inferenceInterpret the % bias in terms of sample replacementWhat are the possible sources of bias?Would they all work in the same direction?What happens if you change thesample sizestandard errordegrees of freedomDebate your inference with a partnerReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Borman, G. D., Dowling, N. M., and Schneck, C. 2008. A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389-407.

48

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Value of RandomizationFew differences between groupsBut done at classroom levelTeachers might talk to each otherSchool level is expensive (Slavin, 2008)Slavin, R. E. (2008). Perspectives on evidence-based research in education-what works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.


n=27+22=49Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Differences between Open Court and Business as UsualDifference across grades: about 10 units7.95 using statistical modelstatistically significant unlikely (probability < 5%) to have occurred by chance alone if there were really no differences in the populationBut is the Inference about Open Court valid in other contexts?Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtaining # parameters estimated, t critical, estimated effect and standard error

Estimated effect( ) = 7.95Standard error=1.833 parameters estimated,Df=n of classrooms-# of parameters estimated=49-3=46.t critical = t.05, df=46=2.014

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quantifying the Discourse for Borman et al:What would it take to change the inference? =a population effect, =the estimated effect = 7.95, and # =the threshold for making an inference = se x tcritical, df=46 =1.83 x 2.014=3.69

% Bias necessary to invalidate inference = 1- #/ =1-3.69/7.95=54%

54% of the estimate must be due to bias to invalidate the inference


Calculating the % Bias to Invalidate the Inference:Inside the Calculationst critical= 2.014standard error =1.83 =the estimated effect = 7.95 sample size=49


Calculating the % Bias to Invalidate the Inference:Inside the Calculations# =the threshold for making an inference = se x tcritical, df=46 =1.83 x 2.014=3.686

% Bias necessary to invalidate inference = 1-d#/d =1-3.686/7.95=54%

54% of the estimate must be due to bias to invalidate the inference.Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Fundamental Problem of Inference and Approximating the Unsampled Population with Observed Data (External Validity)910 Yt|Z=p1134 Yc|Z=p566 6 664How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: # =4:1- # /

=1-4/6=.33 =(1/3)

6 Yt|Z=p6 Yc|Z=p0Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum 58

Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court CurriculumOpen CourtBusiness as UsualOriginal volunteer cases that were not replacedReplacement cases from non-volunteer schools with no treatment effectOriginal distribution for all volunteer casesBusiness as UsualUnsampled Population:No effect

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum The Fundamental Problem of External ValidityBefore a randomized experiment:People believe they do not know what generally works People choose treatments based on idiosyncratic conditions -- what they believe will work for them (Heckman, Urzua and Vytlacil, 2006)After a randomized experiment:People believe they know what generally worksPeople are more inclined to choose a treatment shown to generally work in a study because they believe it works The population is fundamentally changed by the experimenter (Ben-David; Kuhn)The fundamental problem of external validitythe more influential a study the more different the pre and post populations, the less the results apply to the post experimental populationAll the more so if it is due to the design (Burtless, 1995)

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Heckman, J., S. Urzua and E. Vytlacil. (2006). Understanding Instrumental Variables in Models with Essential Heterogeneity, Review of Economics and Statistics, 88(3): 389-432.Burtless, G. 1995. The case for randomized field trials in economic and policy research. Journal of Economic Perspective, 9(2), 63-84.

61Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology.Journal of Quantitative Criminology,26(4), 489-500. page 495


Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology.Journal of Quantitative Criminology,26(4), 489-500. page 496Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Comparisons across Randomized Experiments (correlation metric)


Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 2 : % Bias Necessary to Invalidate an Inference for External ValidityTake an example of a randomized experiment in your own data or an articleCalculate the % bias necessary to invalidate the inferenceInterpret the % bias in terms of sample replacementWhat are the possible sources of bias?Would they all work in the same direction?What happens if you change thesample sizestandard errordegrees of freedom

Debate your inference with a new partnerReplacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Ordered Thresholds Relative to Transaction CostsChanging beliefs, without a corresponding change in action.Changing action for an individual (or family)Increasing investments in an existing program.Initial investment in a pilot program where none exists.Dismantling an existing program and replacing it with a new program.

Definition of threshold: the point at which evidence from a study would make one indifferent to policy choices Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum How District Leaders Really Make Decisions, and About Whatfrom Design-Based Implementation Research: A Strategic Approach to Scaling and Sustaining Educational Innovation: William R. PenuelUniversity of ColoradoPhilip BellUniversity of WashingtonThis graph shows the most common use of research evidence among a sample of 30 or so administrators from a large urban district. My colleagues Cynthia Coburn, Caitlin Farrell, Annie Allen, and I are studying how district administrators here and in two other districts use research to inform district decision making. The pattern, I can say, for the two districts not shown, is the same.

What the chart shows is that the most frequent uses of research appear on the top of the graph, the least common uses on the bottom. Note we ask about data and research use, and the most common uses of evidence are to improve professional development, existing programs, select standards to focus on, and find concepts to improve ongoing improvement efforts. Very low down is choose between programs, the canonical use imagined in the scale-up model.72Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology.Journal of Quantitative Criminology,26(4), 489-500.

Alternative Hypotheses: Non Zero Null HypothesisNon-zero null hypotheses (for kindergarten retention)H0:> 6. se x tcritical, df=7639=.68 x (1.645)= 1.12 (one tailed test).


Alternative Hypotheses: Non Zero Null Hypothesis# = 61.12=7.121 #/ =1 (7.12/9)=.21. 21% of estimated effect would have to be due to bias to invalidate inference for H0:> 6.

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Failure to Reject Null when null is False (type II error)Assume the data are comprised of two subsamples, one with an effect at the threshold (r#) and the other (r ) with 0 effect, and define as the proportion of the sample that has 0 effect. rxy =(1-)r# +r xy .

*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Sample Replacement: Estimate Does not Exceed the ThresholdSet rxy=0 and solve for : =1- rxy /r#If of the cases have 0 effect then if you replace them (with cases at the threshold) then the overall estimate will be at the threshold.


Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs."Journal of the American statistical Association94, no. 448 (1999): 1053-1062.Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Alternative Thresholds for InferenceUse # = 4

To Invalidate the inference, 56% of the estimate would have to be due to bias if the threshold for inference is -4.

Replacement Cases FrameworkConclusionCorrelational FrameworkoverviewThresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the frameworkCorrelational Framework Impact of a Confounding variableInternal validityExample: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Non-zero Effect in the Replacement (e.g., non-volunteer) Population

Inference invalid if 1-p

Documents

Sensitivity Analysis: Quantifying the Discourse About Causal Inference Kenneth A. Frank Help from Yun-jia Lo and Mike Seltzer AERA workshop April 4, 2014