1

Experimental and Quasi-Experimental Designs

•Experimentation is an approach to research best suited for explanation and evaluation

•An experiment is “a process of observation, to be carried out in a situation expressly brought about for that purpose”

•Experiments involve:

•Taking action

•Observing the consequences of that action

•Especially suited for hypothesis testing

2

•Variables, time order, measures, and groups are the central features of the classical experiment

•Involves three major pairs of components:

•Independent and dependent variables

•Pretesting and posttesting

•Experimental and control groups

3

• The Independent Variable takes the form of a

dichotomous stimulus that is either present or

absent

• It varies (i.e., is independent) in our experimental

process

• “The Cause”

4

• The outcome, the effect we expect to see

• Depends on the Independent Variable

• Might be physical conditions, social behavior,

attitudes, feelings, or beliefs

• “The Effect”

5

• Subjects are initially measured in terms of the Dependent Variable prior to association with the Independent Variable (pretested)

• Then, they are exposed to the Independent Variable

• Then, they are re-measured in terms of the Dependent Variable (posttested)

• Differences noted between the measurements on the Dependent Variable are attributed to influence of the Independent Variable

6

• Experimental group – Exposed to whatever treatment, policy, initiative we are testing

• Control group – Very similar to experimental group, except that they are NOT exposed

• If we see a difference, we want to make sure it is due to the Independent Variable, and not to a difference between the two groups

7

• Pointed to the necessity of control groups

• Independent Variable: improved working conditions (better lighting)

• Dependent Variable: improvement in employee satisfaction and productivity

• Workers were responding more to the attention than to the improved working conditions

8

• We often don’t want people to know if they are receiving treatment or not

• We expose our control group to a “dummy” Independent Variable just so we are treating everyone the same

• Medical research: Participants don’t know what they are taking

• Ensures that changes in Dependent Variable actually result from Independent Variable and are not psychologically based

9

• Experimenters may be more likely to “observe” improvements among those who received drug

• In a Double-Blind experiment, neither the subjects nor the experimenters know which is the experimental group and which is the control group

• Broward County Florida and Portland, Oregon domestic violence policing units study: “keeping safe” strategies

10

• First, must decide on target population – the group to which the results of your experiment will apply

• Second, must decide how to select particular members from that group for your experiment

• Cardinal rule – ensure that Experimental and Control groups are as similar as possible

• Randomization purposes towards this

11

• “Randomization”

• Central feature of the classical experiment

• Produces experimental and control groups that are statistically equivalent

• Farrington and associates:

• “Randomization insures that the average unit in the treatment group is approximately equivalent to the average unit in another group before the treatment is applied”

• “All Other Things are Equal”

12

• Experiments potentially control for many threats to the validity of causal inference

• Experimental design ensures:

• Cause precedes effect via taking posttest

• Empirical correlation exists via comparing pretest to posttest

• No spurious 3rd variable influencing correlation via posttest comparison between experimental and control groups, and via randomization

13

• Conclusions drawn from experimental results may not reflect what went on in experiment

• History: External events may occur during the course of the experiment

1. Maturation: People constantly are growing

2. Testing: The process of testing and retesting

14

4. Instrumentation: Changes in the measurement process

5. Statistical regression: Extreme scores regress to the mean

6. Selection biases: The way in which subjects are chosen (use random assignment)

7. Experimental mortality: Subjects may drop out prior to completion of experiment

15

8. Causal time order: Ambiguity about order of stimulus and Dependent Variable – which caused which?

9. Diffusion/Imitation of treatments: Experimental group may pass on elements to Control group when communicating

10. Compensatory treatment: Cgroup is deprived of something considered to be of value

16

11. Compensatory Rivalry: Control group deprived of the stimulus may try to compensate by working harder

12. Demoralization: Feelings of deprivation among control group result in subjects giving up

17

• Potential threats to internal validity are only some of the complications faced by experimenters; they also have the problem of generalizing from experimental findings to the real world

• Two dimensions of generalizability:

• Construct Validity

• External Validity

18

• Concerned with generalizing from experiment to actual causal processes in the real world

• Link construct and measures to theory

• Clearly indicate what constructs are represented by what measures

• Decide how much treatment is required to produce change in Dependent Variable

19

• Significant for experiments conducted under carefully controlled conditions rather than more natural conditions

• Reduces internal validity threats

• John Eck (2002): "diabolical dilemma."

• Suggestion:

• explanatory studies internal validity

• applied studies external validity

20

• Becomes an issue when findings are based on small samples

• More cases allows you to reliably detect small differences; less cases result in detection of only large differences

• Finding cause-and-effect relationships through experiments depends on two related factors:

• Number of Subjects

• Magnitude of posttest differences between the experimental and control groups

21

• Four basic building blocks present in experimental designs:

1.The number of experimental & control groups

2.The number & variation of experimental stimuli

3.The number of pretest & posttest measurements

4.The procedures used to select subjects and assign them to groups

• Variations on the classical experiment can be produced by manipulating the building blocks of experiments

22

• When randomization isn’t possible for legal or ethical reasons

• Renders them subject to Internal Validity threats

• Quasi = “to a certain degree”

• Two categories:

• nonequivalent-groups designs

• time series designs

23

• When we cannot randomize, we cannot assume equivalency; hence the name

• We take steps to make groups as comparable as possible

• Match subjects in Experimental and Control groups using important variables likely related to Dependent Variable under study

• Aggregate matching – comparable average characteristics

24

• Cohort – Group of subjects who enter or leave an institution at the same time

• Ex: A class of police officers who graduate from a training academy at the same time, All persons who were sentenced to probation in May

• Necessary to ensure that two cohorts being examined against one another are actually comparable

25

• Longitudinal Studies

• Examine a series of observations over time

• Interrupted – Observations compared before and after some intervention (used in cause-and-effect studies)

• Instrumentation threat to internal validity is likely because changes in measurements may occur over a long period of time

• Often use measures produced by CJ organizations

26

• A large number of variables are studied for a small number of cases or subjects

• Case-oriented research: Many cases are examined to understand a small number of variables (e.g., Boston Gun Project)

• Variable-oriented research: A large number of variables are studied for a small number of cases or subjects

• Case Study Design: Centered on an in-depth examination of one or a few cases on many dimensions

27