Experiments and Quasi-Experiments

1

Experiments and Quasi-Experiments

Introduction

•Experiment: using a controlled situation to observe a result

•Involves taking and observing action

•Great for hypothesis-testing

•Theory-full

2

The Classical Experiment

•Involves three major pairs of components:

•Independent and dependent variables

•Pre-Testing and Post-Testing

•Experimental and Control groups

•Randomization

3

Variables, X and Y

• X = Independent Variable (IV), cause, influencer

• Y = Dependent Variable (DV), effect, outcome

4

Control and Experimental Groups

• Experimental group – exposed to whatever treatment, policy, initiative we are testing

• Control group – very similar to experimental group, except that they are NOT exposed

5

Selecting Subjects

• Decide on target population 1st– the group to which the results of your experiment will apply

• Cardinal rule – ensure that C and E groups are as similar as possible

• Randomization helps towards this

6

Hawthorne Effect• Pointed to the necessity of control groups

• IV: improved working conditions (better lighting)

• DV: improvement in employee satisfaction and productivity

• Workers were responding more to the attention than to the improved working conditions

7

Placebo

• We often don’t want people to know if they are receiving treatment or not

• We expose our control group to a “dummy” IV just so we are treating everyone the same

• Medical research: participants don’t know what they are taking

• Ensures that changes in DV actually result from IV and are not psychologically based

8

Pre-Testing and Post-Testing

• First, subjects measured on DV prior to association with the IV (pre-tested)

• Next, subjects are exposed to the IV

• Third, subjects are remeasured in terms of the DV (post-tested)

• Difference?--must be the intervention!

9

Double-Blind Experiment

• Subjects and experimenters do not know who is in the control and experimental groups

10

Experiments and Causal Inference• Experimental design ensures:

• Cause precedes effect via taking posttest

• Empirical correlation exists via comparing pretest to posttest

• No spurious 3rd variable influencing correlation via posttest comparison between experimental and control groups, and via randomization

11

Internal Validity Threats (12)• Conclusions drawn from experimental results

may not reflect what went on in experiment

1. History – external events may occur during the course of the experiment

2. Maturation – people grow

3. Testing – the process of testing and retesting

12

More Internal Validity Threats4. Instrumentation – Changes in the

measurement process

5. Statistical regression – Extreme scores regress to the mean

6. Selection bias – the way in which subjects are chosen

7. Experimental mortality – subjects may drop out prior to completion of experiment

8. Causal time order – ambiguity about order of stimulus and DV – which caused which?

13

Last, Internal Validity Threats9. Diffusion/imitation of treatment – when E and C

groups communicate, E group may pass on elements to C

10. Compensatory treatment – C group is deprived of something considered to be of value

11. Compensatory Rivalry – C group deprived of the stimulus may try to compensate by working harder

12. Demoralization – feelings of deprivation result in C group giving up

14

Construct Validity Threats

• Concerned with generalizing from experiment to actual causal processes in the real world

• Link construct and measures to theory

• Clearly indicate what constructs are represented by what measures

• Decide how much treatment is required to produce change in DV

15

External Validity Threats

• Significant for experiments conducted under carefully controlled conditions rather than more natural conditions

• But, this reduces internal validity threats!

• A conundrum!

• Suggestion – explanatory studies -> internal validity; applied studies -> external validity

16

Statistical Conclusion Validity Threats (Low Power)

• Problem is likely when using small samples

• With more cases, it is easier to see more differences

17

Quasi-Experimental Designs

• When?—randomization not possible

• Quasi = “to a certain degree” or, in short, “like”

18