CESS Workshop

Oxford University

June 2013

Don Green

Professor of Political Science

Columbia University

Outline

•Examples of ongoing field experiments•Why experiment? Why experiment in the field?•Key ideas in Field Experiments: Design, Analysis, and Interpretation

• Randomization inference• Noncompliance• Attrition• Spillover

Brief sketches of some current experimental projects

• Voter turnout and persuasion in the US and abroad (mass media, mail, phones, shoe leather, events…) with a recent focus on the influence of social norms

• Downstream experiments: habit, education and political participation

• Prejudice reduction• Criminal sentencing and deterrence• Civic education and political attitudes• Media and pro-social behavior

Why experiment?

The role of scientific procedures/design in making the case for unbiased inference

Unobserved confounders and lack of clear stopping rules in observational research

The value of transparent & intuitive science that involves ex ante design/planning and, in principle, limits the analyst’s discretion

Experiments provide benchmarks for evaluating other forms of research

Why experiment in field settings?

Four considerations regarding generalizability Subjects, treatments, contexts, outcome

measures

Narrowing the gap between an experimental design and the target estimand

Trade off: must attend to treatment fidelity

Systematic experimental inquiry can lead to the discovery and development of useful and theoretically illuminating interventions

What is an experiment?

Perfectly controlled experiments in the physical sciences versus randomized experiments in the behavioral sciences

A study that assigns subjects to treatment with a known probability between 0 and 1

Different types of random assignment: simple, complete, blocked, clustered

Confusion between random sampling and random assignment

Experiment versus Alternative Research Designs: Example of Assessing the Effects of Election Monitors on Vote Fraud (inspired by the work of Susan Hyde)

Randomized experiment: Researcher randomly assigns election monitoring teams to polling locations

Natural/quasi experiment: Researcher compares polling locations visited by monitoring teams to polling locations not visited because some monitoring team leaders were sick and unable to travel

Observational Study: Researcher compares polling locations visited by monitoring teams to polling locations not visited

Are experiments feasible?

Large-scale, government-funded “social experiments” and evaluations

Research collaborations with political campaigns, organizations, etc. who allocate resources

Researcher-driven/designed interventions

Seizing opportunities presented by naturally occurring randomized assignment

Are experiments necessary?

Not when the counterfactual outcome is obvious (e.g., the effects of parachutes on the well-being of skydivers) By the way, how do we know it’s obvious?

Not when there is little or no heterogeneity among the units of observation (e.g., consumer product testing)

Not when the apparent effect is so large that there is no reason to attribute it to unobserved heterogeneity (e.g., the butterfly ballot effect in the 2000 election)

…for most behavioral science applications, experiments are indispensable

Themes of FEDAI

Importance of… Defining the estimand

Appreciating core assumptions under which a given experimental design will recover the estimand

Conducting data analysis in a manner that follows logically from the randomization procedure

Following procedures that limit the role of discretion in data analysis

Presenting results in a detailed and transparent manner

Potential outcomes

Potential outcomes: Yi(d) for d = {0,1}

Unit-level treatment effect: Yi(1) – Yi(0)

Average treatment effect: E[Yi(1) – Yi(0)]

Indicator for treatment received: di

Observed outcome: Yi =diYi(1) + (1-di)Yi(0)

Example: a (hypothetical) schedule of potential outcomes

Core assumptions

Random assignment of subjects to treatments implies that receiving the treatment is statistically

independent of subjects’ potential outcomes Non-interference: a subject’s potential outcomes reflect

only whether they receive the treatment themselves A subject’s potential outcomes are unaffected by how the

treatments happened to be allocated Excludability: a subject’s potential outcomes respond

only to the defined treatment, not other extraneous factors that may be correlated with treatment

Importance of defining the treatment precisely and maintaining symmetry between treatment and control groups (e.g., through blinding)

Randomization inference

State a set of assumptions under which all potential outcomes become empirically observable (e.g., the sharp null hypothesis of no effect for any unit)

Define a test statistic: estimated it in the sample at hand and simulate its sampling distribution over all (or effectively all) possible random assignments

Conduct hypothesis testing in a manner that follows from the randomization procedure (e.g., blocking, clustering, restricted randomizations) using the ri() package in R.

Analogous procedures for confidence intervals

Using the ri() package in R (Aronow & Samii 2012)

Uses observed data and a maintained hypothesis to impute a full schedule of potential outcomes

Detects complex designs and makes appropriate adjustments to estimators of the ATE (or CACE)

Avoids common data analysis errors related to blocking or clustering

Simulates the randomization distribution and calculates p-values and confidence intervals

Provides a unified framework for a wide array of tests and sidesteps distributional assumptions

Example: RI versus t-tests as applied to a small contributions experiment (n=20)

One-tailed p-values for the estimated ATE of 70: randomization inference p=0.032 t(equal variances) p=0.082 t(unequal variances) p=0.091

Example of a common error in the analysis of a block-randomized design

A GOTV phone-calling experiment conducted in two blocks: competitive and uncompetitive. We effectively have two experiments, one in each block.

Voted in Nov., 2002? * Called by Phone Bank? * Competitive House Race? Crosstabulation

657709 8540 666249

57.0% 56.9% 57.0%

496900 6460 503360

43.0% 43.1% 43.0%

1154609 15000 1169609

100.0% 100.0% 100.0%

166649 7888 174537

52.5% 52.6% 52.5%

150533 7112 157645

47.5% 47.4% 47.5%

317182 15000 332182

100.0% 100.0% 100.0%

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Abstained

Voted

Voted in Nov.,2002?

Total

Abstained

Voted

Voted in Nov.,2002?

Total

CompetitiveHouse Race?No

Yes

Not called(control)

called(treatment)

Called by Phone Bank?

Total

Doh! Notice what happens if you neglect to control for the blocks!

Statistically significant – and misleading – results…

Voted in Nov., 2002? * Called by Phone Bank? Crosstabulation

824358 16428 840786

56.0% 54.8% 56.0%

647433 13572 661005

44.0% 45.2% 44.0%

1471791 30000 1501791

100.0% 100.0% 100.0%

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Count

% within Calledby Phone Bank?

Abstained

Voted

Voted in Nov.,2002?

Total

Not called(control)

called(treatment)

Called by Phone Bank?

Total

Noncompliance: avoiding common errors

People you fail to treat are NOT part of the control group!

Do not throw out subjects who fail to comply with their assigned treatment

Base your estimation strategy on the ORIGINAL treatment and control groups, which were randomly assigned and therefore have comparable potential outcomes

A misleading comparison: comparing groups according to the treatment they received rather than the treatment they were assigned

What if we had compared those contacted and those not contacted in the phone bank study? Voted in Nov., 2002? * Reached by Phone Bank? * Competitive House Race? Crosstabulation

663404 2845 666249

57.0% 51.6% 57.0%

500687 2673 503360

43.0% 48.4% 43.0%

1164091 5518 1169609

100.0% 100.0% 100.0%

171819 2718 174537

52.6% 46.9% 52.5%

154567 3078 157645

47.4% 53.1% 47.5%

326386 5796 332182

100.0% 100.0% 100.0%

Count

% within Reachedby Phone Bank?

Count

% within Reachedby Phone Bank?

Count

% within Reachedby Phone Bank?

Count

% within Reachedby Phone Bank?

Count

% within Reachedby Phone Bank?

Count

% within Reachedby Phone Bank?

Abstained

Voted

Voted in Nov.,2002?

Total

Abstained

Voted

Voted in Nov.,2002?

Total

CompetitiveHouse Race?No

Yes

No Yes

Reached by PhoneBank?

Total

Addressing (one-sided) noncompliance statistically

Define “Compliers” and estimate the average treatment effect within this subgroup

Model the expected treatment and control group means as weighted averages of latent groups, “Compliers” and “Never-takers”

Assume excludability: assignment to treatment only affects outcomes insofar as it affects receipt of the treatment (the plausibility of this assumption varies by application)

Simplified model notation

Suppose that the subject pool consists of two kinds of people: Compliers and Never-takers

Let Pc = the probability that Compliers vote

Let Pn = the probability that Never-takers vote

Let = the proportion of Compliers people in the subject pool

Let T = the average treatment effect among Compliers

Expected voting rates in control and treatment groupsProbability of voting in the control group (V0) is a weighted average of Complier and Never-taker voting rates:

V0 = Pc + (1- ) Pn

Probability of voting in the treatment group (V1) is also a weighted average of Complier and Never-taker voting rates:

V1 = (Pc+ T) + (1- ) Pn

Derive an Estimator for the Treatment-on-treated Effect (T)

V1 - V0 = (Pc+T)+(1- )Pn – { Pc + (1- ) Pn }

= T

T is the “intent to treat” effect

To estimate T, insert sample values into the formula:

T* = (V*1 – V*0)/where is the proportion of contacted people

(Compliers) observed in the treatment group, and V*1 and V*0 are the observed voting rates in the assigned treatment and control groups, respectively

Example: Door-to-door canvassing (only) in New Haven, 1998: =.3208

Example: V*1=.4626, V*0=.4220

Estimate Actual Treatment Effect

( V*1– V*0 ) / = ( 46.26 – 42.20) / .3208 = 12.7

In other words: actual contact with canvassers increased turnout by 12.7 percentage-points

This estimator is equivalent to instrumental variables regression, where assignment to treatment is the instrument for actual contact.

Notice that we NEVER compare the voting rates of those who were contacted to those who were not contacted!…Why not?

Design question: Compare the assigned treatment group to an untreated control group or a placebo group?

Placebo must be (1) ineffective and (2) administered in the same way as the treatment, such that assignment to placebo/treatment is random among those who are contacted Assignment to placebo should be blinded and

made at the last possible moment before treatment

Placebo design can generate more precise estimates when contact rates are low

Nickerson’s (2005) Canvassing Experiment: GOTV, Placebo, and Control Groups

Contact rate was low (GOTV: 18.9% and placebo: 18.2%)

Turnout results for GOTV and control: GOTV treatment (N=2,572): 33.9% Control (N=2,572): 31.2% Treatment on treated: b=.144, SE=.069.

Turnout results for contacted GOTV and contacted placebo: GOTV treatment (N=486): 39.1% Placebo (N=470): 29.8% Treatment on treated: b=.093, SE=.031

Bottom line: placebo control led to more precise estimates

Attrition

Can present a grave threat to any experiment because missing outcomes effectively un-randomize the assignment of subjects

If you confront attrition, consider whether it threatens the symmetry between assigned experimental groups

Consider design-based solutions such as an intensive effort to gather outcomes from a random sample of the missing

Spillovers

Complication: equal-probability random assignment of units does not imply equal-probability assignment of exposure to spillovers

Unweighted difference-in-means or unweighted regression can give severely biased estimates

Hypotheses about spillovers

Contagion: The effect of being vaccinated on one’s probability of contracting a disease depends on whether others have been vaccinated. Displacement: Police interventions designed to suppress crime in one location may displace criminal activity to nearby locations.

Communication: Interventions that convey information about commercial products, entertainment, or political causes may spread from individuals who receive the treatment to others who are nominally untreated. Social comparison: An intervention that offers housing assistance to a treatment group may change the way in which those in the control group evaluate their own housing conditions.

Persistence and memory: Within-subjects experiments, in which outcomes for a given unit are tracked over time, may involve “carryover” or “anticipation.”

Example: Assessing the effects of lawn signs on a congressional candidate’s vote margin

Complication: the precinct in which a lawn sign is planted may not be the precinct in which those who see the lawn sign cast their votes

Exposure model: define a potential outcome for (1) precincts that receive signs, (2) precincts that are adjacent to precincts with signs, and (3) precincts that are neither treated nor adjacent to treated precincts

Further complication: precincts have different probabilities of assignment to the three conditions

Non-interference: Summing up

Unless you specifically aim to study spillover, displacement, or contagion, design your study to minimize interference between subjects. Segregate your subjects temporally or spatially so that the assignment or treatment of one subject has no effect on another subject’s potential outcomes.

If you seek to estimate spillover effects, remember that you may need to use inverse probability weights to obtain consistent estimates

Expect to perform a series of experiments.

In social science, one experiment is rarely sufficient to isolate a causal parameter.

Experimental findings should be viewed as provisional, pending replication, and even several replications and extensions only begin to suggest the full range of conditions under which a treatment effect may vary across subjects and contexts.

Every experiment raises new questions, and every experimental design can be improved in some way.