Variance in Research Design

Sources, threats to internal validity, and “Noise”

Sources of Variance

• There are three “sources” of variance:

1) Primary Variance: the variability in the DV that occurs as a result of (or is caused by) the influence of the IV

2) Error Variance: unexplained variance. Variability due to true chance happenings such as moment-to-moment fluctuations in your subject’s attention or fluctuations in your ability to accurately measure your DV due to chance variations in accuracy of equipment.

3) Secondary Variance: Variance in the DV

that occurs as a result of the influence of secondary variables.

• As the experimenter, you want to maximize primary variance, minimize error, and control secondary variance.

• You cannot completely prevent error variance. Some will occur.

• Secondary variance is where the researcher has the best opportunity to improve the chances of obtaining good internal and external validity.

Problems caused by Secondary Variance

• Secondary Variance can cause problems in two ways:

1) If the secondary variable (2nd variable) co-varies along with the Independent variable (IV) then the secondary variable will create a “threat to internal validity” or confound.

2) If secondary variables cause a lot of overall variability in your DV then this may “mask” or hide any effects of the IV. The 2nd variables create “noise” in the data that make it harder to detect an effect of your IV.

Problem #1:Threats to Internal Validity (Confounds) page 314-317 9th ed

• In chapter 10 for ALL versions of text

• Five different threats to internal validity (Campbell and Stanley, 1966)

Two threats resulting in non-equivalent groups

• Occur in independent group (or between subject) type designs.

• Example of threat #1 : The Effects of viewing a violent stimulus on future aggressive tendencies

• Selection: occurs when participants/subjects in one level of the IV differ initially from participants/subjects in another level of the IV, due to systematic selection differences. This is usually the result of the use of “intact groups” or lack of random assignment of subjects to groups. It is a “between-subject” or independent group type issue.

Example of threat #2: Is punishment more effective than reward?

• Mortality/attrition: occurs when you lose subjects from one level of the IV more than from some another level, systematically. Differential loss of subjects from levels of the IV.

• It is a “between-subject” or independent group type issue.

Three threats that involve time-related issues

• Occur in repeated-measures type designs

• Example of threat #3: Effect of a remedial math course for entering freshmen on math grades in college-level Math courses.

• Regression toward the mean (aka statistical regression): extreme scores will become less extreme with repeated measurement.

• Very low scores will regress upward toward the mean and very high scores will fall down toward the mean.

• Not every subject’s score will do this but on average, in the long run, extreme scores will become less extreme and thus they will move toward the mean when subjects are re-measured.

• This is a potential problem in a repeated-measures type design.

• Example of threat #4: The effects of a program on the benefits of alternative energy sources on attitudes toward the use of alternative energy sources.

• History: the occurrence of an EVENT other than the treatment that produces changes in the participants’ behavior. The event is not under the researcher’s control.

• This is a repeated measures problem. The longer the time between measurements, the more likely it is that such an event will occur and internal validity will be jeopardized.

• Example of threat #5: The effects of a pre-school story-time program on adjustment to a school environment.

• Maturation: Changes occur to your participants (older, wiser, weaker, stronger etc.) between measurements.

• These changes are not related to your IV. Practice, boredom, and fatigue are considered threats to internal validity of maturation…even though these are short-term changes.

• this is a potential issue in a repeated-measures type manipulation.

Controlling “Noise”

• Your chances of detecting a real effect of your IV will be better if you reduce variability caused by 2nd variables.

• Two ways in which you can reduce “noise” by controlling secondary variables

1) Isolation: Insolate the research in a “controlled” environment (laboratory). Allows control over many environmental variables (lighting, temperature, noise)

2) Holding constant: Hold some 2nd “individual differences” variables constant to eliminate variability due to 2nd variables

• Isolation and holding constant will eliminate “noise” that might “mask” the effect of your IV

• It will increase “internal validity” by eliminating potential confounds

• But it will also reduce “external validity” (your ability to take the results of your research and make statements about other populations, settings, and conditions)

• Researcher must find the balance between controlling 2nd variables (to increase internal validity but lower external validity) and allowing 2nd variables to occur randomly ( to increase external validity but potentially lower internal validity).