Experimental Methods in the Social Sciencessites.duke.edu/niou/files/2013/07/beijing-experiments-2013-intro.pdf · Experimental Bias Expectancy E ects: the experimenter in uences

Experimental Methods in the Social Sciences

Introduction to Experiments

August 5, 2013

Experiments Increasingly Important in theSocial Sciences

· Field experiments in political science as early as

1920s

· Growing use of experiments in development

economics, transportation, education, criminology,

· Global experiments in economic development,

alternative energy sources, education

2

Increase in Articles Using Experiments in

American Political Science Review

3

Why More Experiments Now?

· Better technology

· Maturation of social sciences

· Increased understanding of the complexity and

interdependence of research in social sciences

· Social sciences catching up to physical sciences

and medicine

4

Key Elements of Experiments

· Standardization

· Randomization

· Treatment versus Control Groups

· Between-Subjects versus Within-Subjects Design

· Internal versus External Validity

· Experimental Bias

5

Standardization

· The Dependent Variable (DV) – the thing to be

explained – is measured the same way in the same

context across subjects

· Independent Variables (IVs) – the explanatory

variables – also controlled and standardized

· Compared to field studies, experiments eliminate

or control unmeasured influences on the DV and

IVs

6

Treatment versus Control

· Treatment is the Independent Variable that the

experimenter manipulates.

· The Treatment Group receives the treatment.

· The Control Group that does not receive the

treatment

· An experiment may contain multiple treatments

and a control group.

7

Blinded Experiment

· In a blind experiment, subjects do not know

whether they received the treatment or control.

· A placebo is a false treatment, meant to make the

control group believe they received the treatment.

· In a double-blind experiment, the experiment

administrators do not know who received the

treatment or control

8

Randomization

· The key feature of an experiment is randomization

· In laboratory and survey experiments, subjects

assigned randomly to treatment and control

groups

· If subjects volunteer for either treatment or

control, we cannot be sure that different

outcomes are due to selection bias or to the

treatment

9

Randomization

· A simple randomizing device such as a coin flip

can assign subjects to treatment or control.

· An n-factor design includes n simultaneous

treatments.

· Each subject randomized across the n treatments.

10

Factorial Design

· Factorial design indicates the number of different

treatments and conditions per treatment

· Each subject receives a combination of treatments (or

controls)

· For example, a researcher may want to test whether a

combination of:

vitamins (vitamin or no vitamin)

exercise (exercise or no exercise

drugs (Drug A, Drug B, no drug)

affect a subject’s health

11

· A 2x2x3 factorial design indicates three separate

treatments.

· The first and second have two conditions (such as a

treatment and control)

· The third has three conditions (such as a control and two

treatments)

· The 2x2x3 design breaks subjects into 12 different groups

for analysis.

12

Between-Subjects versus Within Subjects

· Between Subjects breaks subjects into treatment

and control groups to measure differences

between the two groups after treatment.

· Within Subjects design measures subjects before

treatment and after treatment. Each subject is its

own control.

· Within Subjects also a Pre-post design.

13

Randomized Controlled Trial (RCT)

· The most powerful experimental technique in

scientific research

· RCT’s are usually necessary in a clinical trial for

medical treatments

· Randomized Trial similar to RCT but does not

contain a control group

14

RCT’s Classified by Study Design

1. Parallel Group: Each subject randomly assigned

to a group. All subjects in a group receive the

same treatment (or control)

2. Crossover: Each subject receives or does not

receive the treatment in a random sequence

15

3. Cluster: Pre-existing groups of subjects are

selected to receive the treatment (or control).

Groups may include schools, classes, villages, city

blocks, provinces, countries.

4. Factorial: Each subject randomly assigned to a

group that receives combinations of treatments

(or controls).

16

Standard Reporting Flowchart forParallel RCT

17

18

Example: Dartmouth SPORT Study(Weinstein, et al.)

· Spine Patients Outcome Research Trial (SPORT)

· RCT to assess whether surgery or conservative

care (rest, physical therapy, drugs) provided

better outcomes for herniated disc, spondyliosis,

and other ailments of the spine

· Published in the Journal of the American Medical

Association, 2007, 2008

19

2,720 Patients were screened

501 enrolled in the randomized cohort

245 were assigned to surgery 256 were assigned tonon-surgical treatment

729 Patients were ineligible

426 Were not surgical candidates 19 Had fracture, infection, or deformity 129 Had inadequate non-operative treatment 20 Had cancer 135 Had other reasons

1,991 Patients were eligible

747 Patients declined to participate

743 enrolled in the observational cohort

521 chose surgery222 chose non-surgical

treatment

203 Were available at 6 wk 40 Missed the follow-up visit 2 Withdrew 0 Died

75 (31%) Had undergone surgery

198 Were available at 3 mo 45 Missed the follow-up visit 2 Withdrew 0 Died




202 Were available at 1 yr 29 Missed the follow-up visit 14 Withdrew 0 Died










































149 Were available at 4 yr 47 Missed the follow-up visit 33 Withdrew 1 Died 15 Had visits pending


150 Were available at 4 yr 46 Missed the follow-up visit 43 Withdrew 2 Died 15 Had visits pending


342 Were available at 4 yr100 Missed the follow-up visit 76 Withdrew 3 Died




Figure 1. Exclusion, enrollment, randomization, and follow-up of trial participants. The values for surgery, withdrawal, and death are cumulative over4 years. For example, a total of 1 patient in the group assigned to surgery died during the 4-year follow-up period. [Data set 04/10/2008].

2791Surgical vs. Nonoperative Treatment for Lumbar Disc Herniation • Weinstein et al

SPORT Study Has Many of the SameProblems Social Scientists Encounter

· Many subjects did not comply. Some assigned to

surgery opted for conservative care. Some

assigned to conservative care opted for surgery

· Many subjects could not re-contacted

· Potential placebo effect from surgery

· Some outcomes measured using surveys: People

were asked how they felt

21

Internal versus External Validity

· Internally validity asks, “Did the experimental

treatment make a difference in this specific

experiment.”

· External validity asks, “To what populations,

settings, treatment variables, and measurement

variables can this effect be generalized?”

22

Internal Validity Threatened by...

· History

· Intra-experiment events (intrasession history)

· Selection bias in subjects

· Maturation of subjects

· Performance effects (conditioning)

· Regression toward the mean

· Mortality or attrition of subjects

23

External Validity Threatened by...

· Uncontrolled interactions, omitted variables,

spurious correlations

· Testing may make subjects more sensitive than

rest of population to variables under investigation

· Unrepresentative samples (college students in

particular)

24

External Validity Threatened by...

· Hawthorne effects: subjects change their behavior

when observed

· Professionalized subjects (Panel conditioning:

experiment may change subjects)

· Mortality or attrition of subjects

25

Experimental Bias

· Expectancy Effects: the experimenter influences

subjects

· Demand Effects: Subjects anticipate the purpose

of the experiment

· Experimenter Bias. in recruiting subjects,

selecting experiment time and place, coding and

analyzing data.

26

Ethical Issues

· No physical, financial, emotional harm

· Subjects must give informed consent

· Subjects should not reveal information that is

illegal or threatening to them

· Deceiving subjects acceptable as long as they are

debriefed

27

Common Types of Experiments

· Laboratory experiments

· Field experiments

· Survey experiments

· Natural experiments

· Quasi-experiments: usually non-random

28

Other Thoughts

· “You can solve only one hard problem at a time.”

· Think about causation and mechanisms

· Start with a theory. Experiments test theories.

· Proof is theoretical. Confirmation is empirical.

· Read across academic disciplines

· To paraphrase Marvin Minsky: for any problem (research

question), the best solution (experiment) is the one that

uses the least time, energy, resources

29

Documents

Experimental Methods in the Social Sciencessites.duke.edu/niou/files/2013/07/beijing-experiments-2013-intro.pdf · Experimental Bias Expectancy E ects: the experimenter in uences