External ValidityExternal Validity

The External Validity QuestionThe External Validity Question

TheoryTheory

CauseCauseconstructconstruct

EffectEffectconstructconstruct

What you What you thinkthink

Cause-effect constructCause-effect construct

The External Validity QuestionThe External Validity Question

TheoryTheory

CauseCauseconstructconstruct

EffectEffectconstructconstruct

Cause-effect constructCause-effect construct

ObservationObservation

ProgramProgram ObservationsObservationsWhat you What you dodo What you What you seesee

Program-outcome relationshipProgram-outcome relationship

ObservationObservation

ProgramProgram ObservationsObservationsWhat you What you dodo What you What you seesee

Program-outcome relationshipProgram-outcome relationship

ObservationObservation

ProgramProgram ObservationsObservationsWhat you What you dodo What you What you seesee

Program-outcome relationshipProgram-outcome relationship

ObservationObservation

ProgramProgram ObservationsObservationsWhat you What you dodo What you What you seesee

program-outcome relationshipprogram-outcome relationship

Can we generalize to other persons, places, OR times?Can we generalize to other persons, places, OR times?

How Do We Generalize?How Do We Generalize?Model I: SamplingModel I: Sampling

PopulationPopulation

Specified persons, Specified persons, places, and timesplaces, and times

How Do We Generalize?How Do We Generalize?Model I: SamplingModel I: Sampling

PopulationPopulation

SampleSampleDrawDrawsamplesample

DrawDrawsamplesample

How Do We Generalize?How Do We Generalize?Model I: SamplingModel I: Sampling

PopulationPopulation

SampleSample

GeneraLizeGeneraLizebackback

GeneralizeGeneralizebackback

How Do We Generalize?How Do We Generalize?Model II: Proximal SimilarityModel II: Proximal Similarity

OurOurStudyStudy

How Do We Generalize?How Do We Generalize?Model II: Proximal SimilarityModel II: Proximal Similarity

OurOurStudyStudy

TimesTimes PeoplePeople

PlacesPlaces

SettingsSettings

How Do We Generalize?How Do We Generalize?Model II: Proximal SimilarityModel II: Proximal Similarity

OurOurStudyStudy

TimesTimes PeoplePeople

PlacesPlaces

SettingsSettings

LessLesssimilarsimilar

LessLesssimilarsimilar

LessLesssimilarsimilar

LessLesssimilarsimilar

How Do We Generalize?How Do We Generalize?Model II: Proximal SimilarityModel II: Proximal Similarity

OurOurStudyStudy

TimesTimes PeoplePeople

PlacesPlaces

SettingsSettings

LessLesssimilarsimilar

LessLesssimilarsimilar

LessLesssimilarsimilar

LessLesssimilarsimilar

Gradients ofGradients ofsimilaritysimilarity

Threats to External ValidityThreats to External Validity

Maybe it is just Maybe it is just thesethese people.people.

Maybe it is just Maybe it is just thesethese places.places.

Maybe it is just Maybe it is just thesethese times.times.

Interaction of selection Interaction of selection and treatmentand treatment

Interaction of setting Interaction of setting and treatmentand treatment

Interaction of history Interaction of history and treatmentand treatment

How Can We Improve External How Can We Improve External Validity?Validity?

How Can We Improve External How Can We Improve External Validity?Validity?

PopulationPopulation

SampleSample

Random samplingRandom sampling

How Can We Improve External How Can We Improve External Validity?Validity?

PopulationPopulation

SampleSample

OurOurStudyStudy

PeoplePeoplePl

aces

Plac

es

Tim

esTi

mes

Random samplingRandom sampling

Replicate, Replicate, Replicate, Replicate, ReplicateReplicate

How Can We Improve External How Can We Improve External Validity?Validity?

PopulationPopulation

SampleSample

OurOurStudyStudy

PeoplePeoplePl

aces

Plac

es

Tim

esTi

mes

OurOurStudyStudy

timestimes peoplepeople

placesplaces

settingssettings

Random SamplingRandom Sampling

Replicate, Replicate, Replicate, Replicate, ReplicateReplicate

Use theoryUse theory