Control in Experimentation & Achieving Constancy

Chapters 7 & 8

Internal Validity

This is the extent to which one can accurately state that the independent variable produced the desired effect. Could the observed effect be due to some

factor other than the independent variable?

Confounding Variable

What is a confounding variable?

Confounding Variable

This is a variable that systematically varies with the independent variable.

Internal Validity

Because confounding variables can obscure the effects of the independent variable, you need to control for confounds in order to achieve a high level of internal validity.

Control of Extraneous Variables

Only need to control for the extraneous variables that are confounds.

Ideal is to eliminate confounds but this is not always possible.

If you can’t eliminate the confound, then ensure that it is constant across all levels of the independent variable.

Challenges to Achieving Constancy in the Confounding Variable

Not all confounding variables can be accurately and precisely measured.

Some confounds change throughout the experiment.

You need to identify the confounds in order to know what needs to be held constant.

How do we achieve constancy?

By controlling the extraneous variables that confound the assessment of the influence of the independent variable on the dependent variable.

Threats to Internal Validity

Natural history Maturation Instrumentation Statistical regression

Origins of the Correlation Coefficient

64” 65” 66” 67” 68” 69”

70” 2 4 5 5

69” 2 3 5 8 9 9

68” 3 6 10 12 12 2

67” 7 11 13 14 13 10

66” 6 8 11 11 8 6

65” 3 4 6 4 3 2

Children’s height

Par

ent’

s he

ight

Correlation between parent’s height and children’s height

Threats to Internal Validity - continued

Selection Selection bias can interact with other

threats to internal validity (e.g., maturation, history) thereby magnifying the problem of confounds.

Example of a Selection Bias

Brady et al. (1958) tested the hypothesis that monkeys given control over pressing a lever to avoid a shock would be more likely to develop stomach ulcers (‘executive monkeys’) than monkeys yoked to them and who received same number of shocks (‘yoked monkeys).

Example of a Selection Bias:Brady et al. 1958

Results – ‘executive monkeys’ were significantly more like to die during the 23-day experiment than the ‘yoked monkeys’ and they were more likely to have stomach ulcers.

Selection bias – monkeys were not randomized. Rather those who pressed the lever the quickest on a baseline trial were put in the ‘executive monkey’ condition.

Threats to Internal Validity - continued

Attrition (mortality or drop-outs) Attrition bias can interact with some of the

other threats to internal validity (e.g., selection bias) further confounding the effects of the other confounding variable.

Threats to Internal Validity

Participant effects that need to be controlled Demand characteristics Positive self-presentation bias (most likely to

immerge when participant believes that his/her true intensions, beliefs, or feelings are being assessed).

Need to ensure constant participant perceptions through all phases of the study.

Threats to Internal Validity

Two types of interaction effects between the motive for positive self-presentation and experimental condition: Intertreatment interaction Intratreatment interaction

Threats to Internal Validity Experimenter effects that need to be

controlled: Experimenter’s desire for certain experimental

effects can influence his/her behaviour to the participant, thereby biasing the participant’s response (expectancy effect).

Experimenter may unintentionally record data in a way to support the hypothesis.

Experimenter may unintentionally misinterpret the results in a way that supports the hypothesis.

Threats to Internal Validity

Experimenter variables that need to be controlled: Biosocial attributes Psychosocial attributes Situational factors

Threats to Internal Validity

Magnitude of the expectancy effects: About 1/3 of studies demonstrate some

degree of an experimenter/participant expectancy effect.

The expectancy effect is often greater than the effect of the experimental condition (e.g., treatment effect).

Threats to Internal Validity

Sequencing effects (e.g., carry-over effects).

Participant sophistication

How to Achieve Constancy Across the Independent Variable

Control of the Effects of Extraneous Variables

Appropriate experimental design Statistical methods (e.g., analysis of

covariance) Incorporate control techniques into the

experimental design.

Control Techniques

Randomization Helps ensure that extraneous variables that

could affect outcome are evenly distributed amongst the experimental conditions.

Random selection Random allocation

Random Selection

Each person in the population of interest has an equal chance of being selected and selecting one person does not affect the selection of another.

What are the challenges to random selection?

Random Allocation

Most effective way of ensuring that influential extraneous variables are balanced amongst experimental conditions.

What are challenges to randomization?

Block Randomization

Each block contains all conditions of the experiment in a randomized order.

E, C, C, E

C, E, C, E

E, E, C, C

ExperimentalGroupN = 6

ControlGroupN = 6

Matching

Increases sensitivity in the study by ensuring that experimental conditions are homogeneous with respect to important confounds. Holding extraneous variables constant Building the extraneous variable into the design Yoked control Equating participants

Building Extraneous Variable into the Design

Handhe ld C om puter ized D ie ta ry A ss is tance D evice(Blackberry Pager)

P ag er N o P ag er

R an d om ized

H S F O W eb s iteV iew ers

P ag er N o P ag er

R an d om ized

C ard iac C lin ics

S am p le o fP artic ip an ts w ith som e in te res tin h eart h ea lth y d ie ta ry ch an g e.

Matching

Increases sensitivity in the study by ensuring that experimental conditions are homogeneous with respect to important confounds. Holding extraneous variables constant Building the extraneous variable into the design Yoked control Equating participants

Matching

Increases sensitivity in the study by ensuring that experimental conditions are homogeneous with respect to important confounds. Holding extraneous variables constant Building the extraneous variable into the design Yoked control Equating participants

Matching by Equating Subjects

Precision control Matching precisely on a number of confounding

variables Frequency distribution control

Determine the frequency distribution of the extraneous variable in one sample (e.g., mean and standard deviation of age) and select subjects for the other group who establishes a similar frequency distribution in the second group.

Matching is Often Used When:

1. Small N and so randomization is risky and might yield unequal groups on influential extraneous variables.

2. Matching variable is expected to be correlated with the dependent variable and so exert an effect on it (confound).

3. There is a way to measure participants on the matching variable.

Control Techniques:Counterbalancing

Counterbalance on the sequence of exposure to the experimental condition/task. Control over order effect Control of carry-over effects

Control Techniques:Intrasubject Counterbalancing

The ABBA technique to control for sequence effects Each subject are exposed to the experimental

condition first in one order (AB) and then in the other order (BA).

This is sometimes referred to as a within subject design or repeated measures design.

Control Techniques:Intragroup Counterbalancing

Groups of participants rather than individuals are counterbalanced. Complete counterbalancing is where all

sequences and orders are represented. Incomplete counterbalancing technique is the

most common intragroup counterbalancing technique.

Example of a Partial Counterbalancing Technique – Latin Square Design

A B F C E D

B C A D F E

C D B E A F

D E C F B A

E F D A C B

F A E B D C

Control of Participant Effects

Need to ensure that participants in each experimental condition have identical perceptions about the experiment, except for the one variable being manipulated. Double-blind placebo model Deception Independent measurement of the dependent

variable

Control of Participant Interpretation

Experimental manipulation check by asking the participant after the experiment is completed what he/she thought the experiment was about. Concurrent verbal inquiry is similar except

participant is asked his/her opinion at the end of each experimental trial.

Think-aloud technique is also similar except participant is simply directed to speak all of his/her thoughts aloud.

Control of Experimenter Effects

Control over recording errors Use multiple recorders. Ensure the recorders are ‘blind’ to the

experimental hypothesis. Use electronic recording device.

Control of Experimenter Effects

Control of experimenter attribute errors Hold the experimenter attributes across

treatment conditions (e.g., same therapist, same gender).

Use multiple experimenters. Control of experimenter expectancy error

Blind technique Partial blind technique