Slides to accompany Weathington, Cunningham & Pittenger (2010),

Chapter 4: An Overview of Empirical Methods

1

Objectives

• Internal, statistical conclusion, and external validity

• Empirical methods

• Intact groups and quasi-experimental designs

• Surveys

• Correlational studies

• Single-N methods

• Meta-analysis2

Internal Validity

• Shown by the degree to which a study rules out alt. explanations for IV DV

• Requires ruling out alternative explanations

• Threats include sources of confounding variables

– 4 general categories3

Threats to Internal Validity• Unintended sequence of events

– Carryover effects: drug at Time 1 hurts performance at Time 2 (but the drug is not what we wanted to test)

– Maturation: Changes in answers between 6 and 10 year olds may be due to normal learning rather than a reading intervention

– Intervening events: being burglarized may change your response to a social psychology experiment involving eye witnesses

4

Threats to Internal Validity

• Nonequivalent groups

– Confounds interpretation of cause and effect between IV and DV

– Can be caused by:

•Non-random sampling

•Mortality/attrition

•Subject characteristics (variables)

5

Threats to Internal Validity

• Measurement errors

– Non-valid test

– Low reliability of measurement

– Ceiling and floor effects

– Regression to the mean

• Ambiguity of cause and effect

– Which came first, X or Y?

6

Statistical Conclusion Validity

• Were the proper statistical or analytical methods used when studying the data?

• “Proper” = best allowing the researcher to:

– Demonstrate relationship between IV and DV

– Identify the strength of this relationship

7

Threats to Statistical Conclusion Validity• Low statistical power: increases risk of

missing an effect that really exists

• Violating assumptions of tests: no statistical tests are perfect in all research situations; you need to know your “tools”

• Unreliability in measurement and setting: inconsistencies in the measurement process make it impossible for you to draw valid inferences from the statistics

8

External Validity

• Do our findings/results generalize beyond our sample?

– More likely if representative sample

• Can we generalize our findings to the population?

• Can we generalize our conclusions from one population to another?

9

Internal vs. External Validity

INTERNAL VALIDITY

Interpreting the data for cause

and effectData Population

Generality of findings

Generality of conclusions

EXTERNAL VALIDITY

10

Threats to External Validity

• NOT always just the “lab setting”

• Participant recruitment

– How + who you select to study matters

– Need to be as representative as possible

•May require replication, extension studies

11

Threats to External Validity

• Situation effects

– Where you do the study matters

– Control for what you can and consider replicating in different settings

• History effects

– Be aware that phenomena may change over time

12

True Experiment• Best method for testing cause and effect

• “Easiest” control for internal validity threats

• Not always a practical/ethical option

• You know it is a true experiment if:

1. The IV can be controlled/manipulated

2. Random assignment to conditions occurs

3. Control conditions can be created13

True Experiment

Nonrandom differences among the groups in terms of the measured DV leads us to conclude that the

manipulations of the IV may have caused those differences

Sampling frame

Group 1n = 10

Group 2n = 10

Treatment for Group 1

Group 3n = 10

Treatment for Group 3

Treatment for Group 2

Results for Group 1

Results for Group 2

Results for Group 2

Assuming random assignment into

groups, differences among the groups at

this stage are due to random effects

Separate conditions controlled by the

researcher (different levels of IV)

Differences among groups due to

random effects + effect of treatment

(level of IV)

Random assignment

14

Intact Groups Design

• No random assignment possible

• Multiple samples (by subject variables), from multiple populations

• Cannot establish cause and effect

– Unknown 3rd variable and temporal order

• Can compare differences across samplesIndependent

variableDependent

variable

“Third” variable

Independent variable

Dependent variable

15

Intact Groups Design

Population 1

Group 1n = 10

Group 1n = 10

Group 1n = 10

Results for Group 1

Results for Group 2

Results for Group 2

Groups formed by randomly selecting members of each

population into one of the three

treatment groups

Differences among groups due to

random effects + effect of population (group membership)

Population 2

Population 3

16

Quasi-Experimental Design

• No random assignment; grouping by some other factor

• An IV is manipulated

• One group is treated as a “control”, while the other is exposed to the manipulated IV

• Still problem with unknown 3rd variable and temporal order

17

Quasi-experimental Design

Group 1Baseline

MeasurementMeasurementTreatment

Group 2Baseline

MeasurementMeasurementNo treatment

18

How does the true experiment differ from the intact groups and quasi-experiment design?

19

Surveys

• For estimating population parameters

• Good for large-scale data collection

– Quick and inexpensive

• “Bad” because of respondent error

– Honesty and personal bias

20

Correlational Study

• Usually to estimate population parameters

• Often data from surveys

• Good for initial understanding and “prediction” of complex behaviors

• Bad at supporting cause and effect

– Unknown 3rd variable

– Temporal order issues21

Single-N Methods

• Sometimes better to focus in-depth on one or a few participants

– Single-participant experiment

– Case study

• Good if IV and situational variables are well-controlled

• Bad for generalizability (potentially) and also because of participant bias/error

22

Meta-analysis

• Analysis of multiple outcomes from multiple studies

• Good because takes advantage of more representative sampling of participants and measures/methods

• Bad because depends on which studies are entered

– Principle of GI, GO

23

What is Next?

• **instructor to provide details

24