Internal Validity & Research Design

Internal Validity: Assuming that there is a relationship in this study, is the relationship a causal one?

Problem: How to increase pet adoptions from the center? Solution: Establish a Facebook presence!

Results (note – this is a fictitious example): Dogs Cats TTL May adoptions (no Facebook page) 100 120 220

Facebook page created June 1

June adoptions 115 150 265

Did the additional communication from the Facebook page cause the increase in pet adoptions?

How do we Establish a Cause-Effect Relationship? 3 criteria

• Temporal Precedence

• Covariation of the Cause and Effect

• No Plausible Alternative Explanations

How do we Establish a Cause-Effect Relationship?

• Temporal Precedence

• Covariation of the Cause and Effect

• No Plausible Alternative Explanations

Research Design R O X O R O O

Notation: O = Observations / Measures Osubscript = measure taken on that occasion X = Treatments / Programs Rows = Groups R = Random assignment to group N = Nonequivalent groups C = Assignment by cutoff Time = moves left to right

Types of Designs

Types of Designs

Single Group Threats to Internal Validity The Single Group Case

Posttest only Pretest-Posttest X O O X O

History Threat History Threat

Maturation Threat Maturation Threat

Testing Threat (primed)

Instrumentation Threat (change test)

Mortality Threat

Regression Threat (regression to the mean)

Regression to the Mean   a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated.

It is a statistical phenomenon. It is a group phenomenon. It happens between any two variables. It is a relative phenomenon. You can have regression up or down. The more extreme the sample group, the greater the regression to the mean. The less correlated the two variables, the greater the regression to the mean.

A real example: Exam 1 Exam 2 Pop. Mean 2.36 2.24 Standard Dev. .79 .80 Correlation rExam1•Exam2 = 0.44 Exam 1 Bottom 10% Mean 1.2 1.4 Distance (Std. Devs.) 1.5 1.0 Exam 1 Top 10%

Mean 3.6 3.3 Distance (Std. Devs.) 1.6 1.3

A real example: Exam 1 Exam 2 Pop. Mean 2.36 2.24 Standard Dev. .79 .80 Correlation rExam1•Exam2 = 0.44 Exam 1 Bottom 10% Mean 1.2 1.4 Distance (Std. Devs.) 1.5 1.0 Exam 1 Top 10%

Mean 3.6 3.3 Distance (Std. Devs.) 1.6 1.3

How do we deal with single group threats to internal validity?

Most common: change the research design Most common change: add a control group (Of course, when you add a control group, you no longer have a single group design.)

Multiple Group Threats to Internal Validity There is one multiple group threat to internal validity: that the groups were not comparable before the study.

A selection bias or selection threat: any factor other than the program that leads to posttest differences between groups.

Pretest-Posttest with Control O X O O O

Selection-History Threat

Selection-Maturation Threat

Selection-Testing Threat (primed)

Selection-Instrumentation Threat (change test)

Selection-Mortality Threat

Selection-Regression Threat (regression to the mean)

Experimental Design • probably the strongest design with respect to

internal validity.

• key to the success of the experiment is in the random assignment (achieves probabilistic equivalence).

Probabilistic Equivalence

Random Selection

vs.

Random Assignment

Sampling Design

External Validity Internal Validity

Social Interaction Threats to Internal Validity Social pressures in the research context that can lead to posttest differences

Groups Research Admin.

Diffusion or Imitation of Treatment ✔

Compensatory Rivalry ✔

Resentful Demoralization ✔

Compensatory Equalization of Treatment ✔ ✔

The simplest of all experimental designs: two-group posttest-only randomized experiment.

Remember this?

Remember this?

This is an Extremely Simple Design Only One Independent Variable (Factor):

Advertising Exposure

What about a design with more than one independent variable?

Independent Variable 1 (Factor 1): Ad Color Two Levels: B&W (1) / CMYK (2) Independent Variable 2 (Factor 2): Humor in Ad Two Levels: Not Humorous (1) / Humorous (2)

Research Design Notation (Post-Test Only) = 4 Groups

R X11 O B&W / Non-humorous R X12 O B&W / Humorous R X21 O CMYK / Non-humorous R X22 O CMYK / Humorous

2 x 2 Factoral Design

Humor

B&W Non-humorous

X11

B&W Humorous

X12 C

olor

CMYK

Non-humorous X21

CMYK Humorous

X22

Understanding Factoral Design Numbering Notation

Design Factors Levels Groups 2 x 2 2 2 / 2 4 2 x 3 2 2 / 3 6 3 x 3 2 3 / 3 9

2 x 2 x 2 3 2 / 2 / 2 8 3 x 2 x 2 3 3 / 2 / 2 12

How Factoral Designs Work

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

Group 1 Mean

Group 2 Mean

Mean Across

Factor 2 Level 2

Group 3 Mean

Group 4 Mean

Mean Across

Mean Down

Mean Down

No Effects (Null Outcome)

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

3

3

3

Factor 2 Level 2

3

3

3

3

3

Main Effect of Factor 1

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

2

4

3

Factor 2 Level 2

2

4

3

2

4

Main Effect of Factor 2

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

2

2

2

Factor 2 Level 2

4

4

4

3

3

Main Effect of Each Factor

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

2

4

3

Factor 2 Level 2

4

6

5

3

5

Interaction Effect (one group differs from all others)

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

3

3

3

Factor 2 Level 2

3

5

4

3

4

Interaction Effect (Crossover)

Factor 1 Level 1

Factor 1 Level 2

Factor 2 Level 1

3

5

4

Factor 2 Level 2

5

3

4

4

4

Adding a Control Group

R X11 O B&W / Non-humorous R X12 O B&W / Humorous R X21 O CMYK / Non-humorous R X22 O CMYK / Humorous R O Control

2 x 2 Factoral Design with Control (not a fully-crossed design)

Incomplete Factoral Design Factor 1

Level 1 Factor 1 Level 2

Factor 2 Level 1

Group 1 Mean

Group 2 Mean

Mean Across

Factor 2 Level 2

Group 3 Mean

Group 4 Mean

Mean Across

Mean Down

Mean Down

Control Mean