Foundations of Research 1 10. Basic research designs. This is a PowerPoint Show Click “slide show” to start it. Click through it by pressing any key. Focus

Foundations of Research

110. Basic research designs.

This is a PowerPoint Show Click “slide show” to start it.

Click through it by pressing any key.

Focus & think about each point; do not just passively click.

To print:

Click “File” then “Print…”.

Under “print what” click “handouts (6 slides per page)”.

© Dr. David J. McKirnan, 2014The University of Illinois [email protected] not use or reproduce without permission

mailto:[email protected]

mailto:[email protected]


2Basic experimental designs

This module overviews the core elements of an experimental research design.

We will discuss “pre-experimental” designs

These typically have no control group or may use existing groups

They are often used in preliminary or exploratory research

“True” experiments have several key characteristics:

A control group

Random assignment of participants to groups

Standardized or uniform procedures for each group


3Experimental designs and validity

We will discuss internal and external validity.

Internal validity In experiments we manipulate (induce…) the Independent Variable.

We then measure the Dependent Variable.

Experimental hypothesis: the outcome (the level of the Dependent Variable) is caused by – and only by – the Independent Variable.

Internal validity: How confident are we that the outcome was due only to the Independent Variable.

Confound: A variable other than the IV that caused or influenced the result.

Did the participants in the experimental v. control groups differ on something other than the IV?

Were the procedures biased in some way…?Confound


4Experimental designs and validity

External validity Experimental participants are a sample of the larger population.

The experimental manipulation attempts to accurately induce the Independent Variable.

The outcome measure represents the Dependent Variable.

The experiment is conducted in a specific physical or cultural setting.

External validity:

Does the research sample accurately represent the larger population?

Does the experimental manipulation accurately represent the concept we think causes the outcome or results?

Do the outcome measures accurately represent the phenomenon we are trying to explain?

Is the experimental setting representative of how these processes work in nature?


5External validity: summary

The study structure & context

The research

Setting:

The Dependent

Variable

The research Sample:

Is the sample representative of the larger population?

Is this typical of the natural

settings where the phenomenon

occurs?

Does the outcome measure represent

what we are trying to explain?

Does the experimental manipulation actually create the phenomenon you

are interested in?

The Independent

Variable


6Overview: Basic Designs

“Pre-experimental” designs: no control groupPost-Test Only Design Pre- Post- Test Design

Group assignment

Pre-test Experimental manipulation

Outcome

Experimental Observe2TreatmentObserve1


7Basic Designs


Group assignment


Outcome

Experimental Observe2TreatmentObserve1

True (or Quasi-)experimental designs with a control group

“After only” Control group design

Pre- Post- Group Comparisons

Control Observe2ControlObserve1


8Basic Designs


Group assignment


Outcome

True (or Quasi-)experimental designs with a control group


Pre- Post- Group Comparisons

Multiple group comparison

Experimental Observe1 Observe2

Experimental

Control

Observe1

Observe1

Treatment 2 Observe2

Observe2

Treatment 1

Control


9“Pre-experimental” designsPost-Test Only Design

Treatment MeasureGroup

Only 1 “group”.

A single set of physical measures or observations

In behavioral science typically an existing group: no selection or assignment occurs.

The condition or experimental intervention (“Treatment”) may or may not be controlled by the researcher.

In Earth Sciences we may examine how a geologic formation is associated with historical water flow.

In Behavioral Sciences we may examine naturally occurring or system-wide events

• e.g., socio-economic conditions and racial conflict,

• the effect of a government policy change on foreclosure rates….

Measurement may or may not be controlled by the researcher.

E.g.; existing (archival) climate data.

A survey after an event such as 9/11

Uniform crime rates, hospital admissions, etc.


10“Pre-experimental” designs

Post-Test Only Design

Treatment MeasureGroup

Measure1Treatment Measure1

Group

Only one group;• only group

available?• naturally

occurring intervention?

Measurements from a baseline period and after an intervention or naturally occurring event.

All participants get the same treatment, which may or may not be controlled by the researcher.

Pre- Post- Test Design

Comparing archival climate data from before & after industrialization

Examining school test scores before & after the introduction of the STEM educational approach


11“Pre-experimental” Designs (2)

Allow us to study naturally occurring interventions.

Advantage of “Post-” & “Pre- Post-” Designs:

e.g., test scores before and after some school change,

Crime rates after a policy change, etc.

Having both Pre- and Post measures allows us to examine change.


12“Pre-experimental” Designs (2)

Disadvantage of “Post-” & “Pre- Post-” Designs:

Maturation: Participants may be older / wiser by the post-test

History; Cultural, historical or physical events may occur between pre- and post-test that can represent a confound in our analysis

Mortality: Participants may non-randomly drop out of the study

Regression to baseline: Participants who are more extreme at baseline look less extreme over time as a statistical confound.

Reactive Measurement: Scores may change simply due to being measured twice, not the experimental manipulation.

No control group = many threats to internal validity.


13Experiments“After only” Control group design

Adds a control group. Either…

Observed Groups: Naturally occurring (e.g., Class 1. v. Class 2) or Self-selected (sought therapy v. did not…).

Assigned Groups:

Randomly assign participants to experimental v. control group, or

Match participants to create equivalent groups.

Measure Dependent Variable(s) only at follow-up.

Use experimental or standard measures (e.g., grades, census data, crime reports).

Experimental

Control


Observe2Control


14Advantages of experimental design


Advantage: Lessens the likelihood of confounds or threats to internal validity.

Control group Random assignment

Disadvantage: Existing or self-selected groups may have confounds.

No baseline or pre- measure available: We cannot assess change over time. We cannot assess whether the groups are

equivalent at baseline.

Experimental

Control


Observe2Control


15Basic Designs: True experiments

Pre- Post- Group Comparisons (most common study design)

Two groups:

Observed (quasi-experiment)

orAssigned

(true experiment).

“Groups” can be Different physical

conditions or lab preparations,

Existing blocks of people

Actual experimental groups…

Baseline (“pre-test”) measure of study variables and possible confounds.

Group 1

Group 2

Measure 1

Measure 1


16Basic Designs: True experiments (2)


The group getting the experimental condition is contrasted with a control group..

Naturally occurring

Created by experimenter

Group 1

Group 2

Measure 1

Measure 1

Treatment Measure 2

Measure2

“Post-test” follow-up of dependent variable(s);

Simple outcome Change from

baseline.

Control


17Basic Designs: True experiments (3)


Group 1

Group 2

Measure 1

Measure 1

Treatment Measure 2

Measure2

Advantages: Pre-measure assesses baseline level of Dependent Variable

Allows researcher to assess change

Can find matched pairs of participants or physical samples and assign each to different groups (rather than random assignment).

Can assess whether groups are equivalent at baseline.

Disadvantage: Highly susceptible to confounds if using observed or self-selected groups.

Control


18More Complex Experimental DesignsMultiple group comparison

3 (or more) groups

Typically formed by Random assignment.

Multiple experimental groups, e.g. Low drug dose, High drug dose, Placebo.

or Male therapist, Female therapist, Wait list control.

Group 1

Group 2

Group 3

Measure1

Measure1

Measure1

Treatment #2

Treatment #1

Control


19More Complex Experimental Designs


Group 1

Group 2

Group 3

Measure1

Measure1

Measure1

Treatment #2

Measure2

Measure2

Measure2

Treatment #1

Compare: Experimental group 1 from

experimental group 2. Either / both experimental

groups from the control group.

Control


20More Complex Experimental Designs


Measure2Group 1

Group 2

Group 3

Measure1

Measure1

Measure1

Treatment #2 Measure2

Measure2

Treatment #1

Advantages: Test dose or context effects:

Drug doses, amounts of psychotherapy, levels of anxiety, etc. Increasing dose effect can be tested against no dose.

Diverse conditions to test 2nd hypotheses or confounds, e.g., therapy delivered by same sex v. opposite sex therapist.

Disadvantage:

More costly and complex.

Potential ethical problem with a “no dose” (or very high dose) condition.

Control


21Core components of a research study

Participant Selection

Participant Assignment

Experimental Procedures

Experimental Treatment or Manipulation

Results

We will use this framework to think about the basic elements of an experiment.

Who or what are we studying?

How did we recruit or sample them?

We will have at least one Experimental Group and a Control Group.

How do we assign participants or samples to be in one or the other?

What instructions do we give?

What experimental tasks will participants be performing?

What measures might we be taking?

Experimental & control groups get different conditions.

We hypothesize that this manipulation “causes” the outcome.

What outcomes are we measuring?

What is the experiment trying to explain?


22Experimental design overview





Results

Sample

Group A Procedure Treatment Outcome

Group B Procedure Control Outcome

(Group C) (Procedure ) (Alternate Treatment?) (Outcome)

We recruit a sample of participants from the larger population.

We randomly assign them to groups to ensure the groups are equivalent at baseline.

Procedures for all groups should be exactly the same…

…except the experimental manipulation, i.e., the Independent variable.

Hypothesis: The outcome or Dependent Variable varies only by group.


23Overview of true experimental designs





Results

Sample

Group A Procedure Treatment Outcome

Group B Procedure Control Outcome

(Group C) (Procedure ) (Alternate Treatment?) (Outcome)

Experimental group

Control group


24Overview: experimental designs

Participant Recruitment




Results

Sample

Group A Procedure A Treatment Outcome

Group B Procedure A Control Outcome

Group C Procedure A Alternate Treatment (?) Outcome

Does the sample well represent

the population?

External validity

• Was recruitment biased?

• Is the sample size large enough?

What form of validity is threatened by sample

bias?

What can we do to avoid that threat?

Random selection







Results

Sample




Does the sample well

represent the population?

External validity

Random selection

Are the groups equal at

baseline?

Internal validity

Random Assignment

• Did participants Self-select (in or out) of the study?

• Did we use existing groups?

Validity Threat?

Solution?







Results

Sample






External validity

Random selection

Are the groups equal at baseline?

Internal validity

Random Assignment

Procedures the same for all groups?

Internal validity:

Lack of confounds

• Do both groups have the same expectations?

• Are participants (and researchers) really blind?

• Do we treat both groups the same?

Validity Threat?

Solution?







Results

Sample






External validity

Random selection


Internal validity

Random Assignment


Internal validity:

Lack of confounds

Independent variable faithfully

reflects the construct?

External Validity

Correct IV?

• Does the operational definition really express the construct we are interested in?

• Have we given the correct dose of the IV?

Validity Threat?

Solution?







Results

Sample






External validity

Random selection


Internal validity

Random Assignment


Internal validity:

Lack of confounds



External Validity

Correct IV?

Internal Validity:

Statistical testing

Groups really

different at outcome?

• Is any difference we see actually statistically significant (reliable & meaningful)?

• …or it is due to chance alone..

Validity Threat?

Solution?







Results

Sample






External validity

Random selection


Internal validity

Random Assignment


Internal validity:

Lack of confounds



External Validity

Correct IV?

Internal Validity:

Statistical testing

Groups really different at outcome?


30Why are research methods so important?A case study.

Siyan, S. et al., (2014). The Relationship Between Return on Investment and Quality of Study Methodology

in Workplace Health Promotion Programs. American Journal of Health Promotion, Vol. 28 (6), Pp. 347-363.

Do workplace health programs actually save money?

Over the past 20+ years there has been considerable interest in workplace health promotion:

…dietary, “lifestyle” and exercise advice & resources; …smoking cessation, weight loss programs…


31Why are research methods so important?A case study.


The hypothesis is that healthier employees will save the company money, via lower absenteeism, health insurance costs, etc.

Evidence appears to support that claim,

Slyan et al. took published studies and divided them into four categories:

Randomized Controlled Trials; “true experiments”, the gold standard of research.

Quasi-experimental designs; where participants were able to choose whether to get the health program or not (self-selection into groups).*

Non-experiments; basically anecdotal or observational studies.

Modeling studies; predicting outcomes based on extant data on the general effects of healthier behavior.

or does it?

* We will discuss quasi-experiments next module.


32


Why are research methods so important?A case study.

The results showed clearly that “Return On Investment” (ROI; actual savings) was higher as methodological quality went down.

High quality studies

showed a very modest ROI

Whereas low quality studies

showed substantial

ROI

In low quality studies, companies appeared save more than twice the money they invested in health promotion.


33



Comparing Randomly Controlled Trials (RCTs) to others was particularly damning for the hypothesis….

RCTs showed companies to actually lose

money through health

promotion.

Non-experimental and modeling

studies showed significant ROI

Lower-quality research lead to very misleading results


34



Why this huge difference between randomized controlled trials and non-experiments?

In the non-randomized trials employees were able to choose (self-select) which group they wanted to be in.

It is completely plausible that healthier or more motivated employees would join the health group, not the control group.

Studies with self-selection may be simply showing us that healthier people stay healthy and cost less, not that the actual programs did anything.


35

Experimental design key elements Control group v. non-controlled designs Threats to internal validity:

Maturation History Mortality Regression to baseline Reactive Measurement

“Pre-experimental” designs Pre-post designs Multiple group comparisons

Overview: key termsS

U M

M A

R Y






Exp. Treatment or Manipulation

Results

Does the sample well represent

the population?

External validity

Are the groups equal at

baseline?

Internal validity




Groups really

different at outcome?

External validity

Internal validity

Internal validity

S U

M M

A R

Y


37

Please go on to the Research Designs quiz.

Documents

Foundations of Research 1 10. Basic research designs. This is a PowerPoint Show Click “slide show” to start it. Click through it by pressing any key. Focus