Survey Experiments

Defined

• Uses a survey question as its measurement device

• Manipulates the content, order, format, or other characteristics of the survey as a treatment

Methodological Issues

• Missing Data

• Matching

• Both can be an issue in experiments other than surveys

Missing Data

• Some observations missing data on the DV or IVs

• If missing at random, not a problem to drop from the analysis

• But usually not missing at random• Deleting non-random missing causes bias

Missing Data II

• Data can also be missing intentionally:• Some cases not “treated”• Possible to “guess” what would have

happened to a subject had they been in another treatment group– Allows within-subject comparison of two

treatments, the one they received and the one they could have received

Solution: Imputation

• Suppose Yi = a + b1Xi1 + b2Xi2 +ei

• But Yi missing for some observations

• Xi1 and Xi2 not missing

• Regress Y on Xi1 and Xi2 for all non-missing observations

• Use b1 and b2 to calculate predicted Ypi

Better Yet: Multiple Imputation

• Ypi is a predicted value with uncertainty

• Multiple imputation predicts multiple values for Yp

i drawn from a distribution of predicted values

• 5 or so predicted Ypi sufficient for inference,

no need for many• Gary King’s Amelia program available free on-

line

Matching

• Experiments can be pre-matched to avoid large random sample

• Match subjects on important characteristics such as– Sex– Race– Age– Education levels– Other traits?

Matching

• Often necessary in field experiment when randomization more difficult to control

• propensity score is the probability of an observation being assigned to a particular treatment in a study given a set of known variables.

• Propensity scores reduce selection bias by equating groups based on these variables

A Theory of Nonseparable Preferences in Survey Responses

Question

• Why do people change their answers to survey questions if the order of questions changes?

• Does changing survey responses indicate that people do not have well-formed opinions

Theory

• Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue

• Separable Preferences: What a person wants on every issue is independent of what they get on other issues

Measuring Nonseparable Preferences

Method

• Randomize the order of pairs of survey questions– For some issues, aggregate responses different

across question order

• Each subject answers questions in order– Issue 1 then Issue 2– Issue 2 then Issue 1

Method

• Impute what subject would have answered had they heard questions in different order

• For each question we then haveYi (if first) – Yi (if second)

• One of these will be imputed for each person since they cannot answer a question both first and second in the order

• First study to analyze individual differences in question orders, not simply aggregate differences

Conclusions

• Nonseparable preferences explain question order effects

• Political information level does not • Response instability not due to uninformed

respondents

Are Survey Experiments Externally Valid?

JASON BARABAS and JENNIFER JERITAmerican Political Science Review

2010

Question

• Many survey experiments expose subjects to different information to show effect of on responses

• In a survey experiment, subjects are a “captive audience” that must pay attention

• Do the same information effects appear in the real world

• Compare survey experiments with natural experiments

Method

• Survey experiments give people to political information about immigration and medical care

• Pre-post survey also in field during change in medical insurance and immigration– Ask respondents which media sources they use

• Is the effect of information in the survey experiment as large as in the natural experiment?