1 of 44

Department of Cognitive Science

PSYCHOMETRICS

Repeated Measures ANOVA

Michael J. Kalsher

MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher

2 of 44

Outline• Review

• Descriptive vs. Inferential Statistics

• Parametric vs. Non-parametric Statistics

• The role of equivalence in research

• The t test– One-group t test

– Independent-groups t test

– Dependent-groups t test

• Analysis of Variance– One-way Independent-groups ANOVA

– One-way Dependent-groups ANOVA

3 of 44

• Describing or characterizing the obtained sample data

• Use of summary measures, typically:– measures of central tendency (mean, median, mode)

– measures of dispersion (range, variance, standard deviation)

Descriptive Statistics

4 of 44

Inferential Statistics

• Used to make inferences about populations based on the behavior of a sample.

• Concerned with how likely it is that a result based on a particular sample, or set of samples, are the same as results that might be obtained from an entire population.

5 of 44

Parametric vs. Non-parametric testsParametric tests• Term used for inferential tests based on the normal distribution.

– Accuracy depends on tests meeting basic assumptions.

• Most outcomes closely follow a known probability distribution.

• Many parametric tests are robust to violations of distributional assumptions, so the assigned p-value will be fairly accurate in many situations.

Non-Parametric tests• A family of statistical procedures that do not rely on the

restrictive assumptions of parametric tests. In particular, they do not assume that data come from a normal distribution.

6 of 44

Assumptions of Parametric Data

1. Normally distributed data.

2. Homogeneity of variance (i.e., variance of each group should be equivalent).

3. Data must be interval or ratio.

4. Independence (i.e., the behavior of one participant doesn’t influence the behavior of another participant).

Most parametric statistics based on the normal distribution have four basic assumptions that must be met for the test to be accurate:

7 of 44

Random assignment: helps to ensure statistical equivalence of groups at the beginning of a study.

Correlated-groups design: assures equivalence by either using the same participants in all groups or participants that have been closely matched. They are usually more sensitive than between-subjects designs to the effects of the IV(s).

- Within-subjects (or repeated measures) designTests each subject under every condition

- Matched-subjects designs

Match participants on relevant variables prior to the study and then randomly assign the matched sets of participants--one member of each matched set--to each group.

Achieving Equivalence: Random Assignment versus Correlated-Groups Design

8 of 44

Characteristics of Within-Subjects Designs

1. Each participant is exposed to all conditions of the experiment, and therefore, serves as his/her own control.

2. Susceptible to sequence effects, so the order of the conditions should be “counter-balanced”. In complete counter-balancing:

a. Each participant is exposed to all conditions of the experiment.

b. Each condition is presented an equal number of times.

c. Each condition is presented an equal number of times in each position.

d. Each condition precedes and follows each other condition an equal number of times.

3. The critical comparison is the difference between the correlated groups on the dependent variable.

9 of 44

Dependent or Matched Pairs t-test

t = Mean Difference

Standard Error (of the mean difference)

Calculation of t:

The dependent, or matched-pairs t-test, is designed for situations in which the same participants are used in both experimental conditions. Thus, each participant contributes two scores.

10 of 44

Assumptions

The dependent groups t test requires the following statistical assumptions:

1.Data are from normally distributed populations. Note: Independent samples t-test is robust against violation of this assumption if n > 30 for both groups.

2.Data are measured at least at the interval level.

11 of 44

Dependent t-test: An Overview

Books Read Mean N Std. Deviation

Std. Error

Mean

Marital Bliss: A Practical Approach

Statistics 101

20.018

18.490

500

500

9.98123

8.99153

.44637

.40211

Paired Differences

Mean Std. Std. Error 95% Confidence Int. Sig.

Diff. Deviation Mean Lower Upper t df (2-tailed)

Pair 1 1.5280 12.62807 .56474 .4184 2.6376 2.706 499 .007

A researcher wonders whether self-help books actually work and asks a group of participants to read two books: one written for the purpose of increasing relationship happiness and another that is (hopefully!) irrelevant for this purpose. After reading each book, she asks participants to fill out a survey that measures relationship happiness.

12 of 44

Critical Values: Dependent Groups t testNote: Degrees of Freedom = ND - 1 (where ND = the number of difference scores)

Our value: 2.706

13 of 44

Effect Sizes

R2 = SSM

SST

The statistical test tells us whether it is safe to conclude that the means come from the same—or different—populations. It doesn’t tell us how strong these differences are.

r2 (r-Square), or the coefficient of determination, is one metric for gauging effect size. It represents the proportion of variance in the Sums of Squares Total that is accounted for by the treatment.

Rules of Thumb regarding effects sizes:

Small effect: 1-3% of the total variance

Medium effect: 10% of the total variance

Large effect: 25% of the variance

14 of 44

Dependent t-test: Calculating the Effect Size

Statistical vs. Practical significance.

Formula: r2 =

t2

t2 + df

Relationship Happiness sample computation:

r2 = (-2.706)2 = 7.32 = .01 (small effect size)

(-2.706)2 + 499 506.32

Degrees of freedom:

(df) = N-1 = 499

15 of 44

Dependent t-test: Reporting the Results

On average, the reported relationship happiness after reading Marital Bliss: A Practical Approach (M = 20.02, SE = .45), was significantly higher than after reading the introductory statistics book (M = 18.49, SE =.40), t(499) = 2.71, p < .01, r2= .01. However, the small effect size estimate indicates that this difference was not practically significance.

16 of 44

Sample Problem A psychologist believes that children of parents who use positive verbal statements (polite requests and suggestions) are more socially accepted and more positive in interactions with their peers. Although children acquire behavioral information from sources other than parents (TV, peers, and so on), more induction (coaching children by introducing consequences for behaviors and supplying rationales that support them) on the part of parents, as opposed to more power-assertive and permissive types of discipline, facilitates a pro-social behavioral orientation in children that, in turn, leads to greater competence and greater acceptance by peers.

Twenty first-grade children who were rated by teachers and peers as aggressive and their parents are asked to participate in a study to determine whether a seminar on inductive parenting techniques improves social competency in children. The parents attend the seminar for one month. The children are tested for social competency before the course and then retested six months after their parents’ completion of the course. The results of the social competency test are shown on the following page with higher scores indicating a higher level of social competency.

In this problem, we are testing the null hypothesis that there is no difference between the means of pre- and post-seminar social competency scores.

What is the IV? What is the DV? Was the seminar effective?

17 of 44

Social Competency Problem Data Set

Child Pre Post Child Pre Post 1 31 34 11 31 28

2 26 25 12 27 32

3 32 38 13 25 25

4 38 36 14 28 30

5 29 29 15 32 41

6 34 41 16 27 37

7 24 26 17 37 39

8 35 42 18 29 33

9 30 36 19 31 40

10 36 44 20 27 28

18 of 44

T-TEST PAIRS=PreSeminar WITH PostSeminar (PAIRED)

/CRITERIA=CI(.9500)

/MISSING=ANALYSIS.

19 of 44

SPSS Output

Paired Samples Descriptive Statistics

Mean N

Std. Deviation

Std. Error Mean

Pre-Post 30.45

34.20

20

20

4.019

6.066

.899

1.356

Paired Samples Correlations

N Correlation Sig.

Pre & Post 20 .771 .000

Paired Differences

t df

Sig.

(2-tail)

Mean Std. Dev.

Std. Error Mean

95% Confidence Interval of the

Difference

Lower Upper

Pre-Post -3.750 3.919 .876 -5.584 -1.916 -4.280 19 .000

Paired

Samples Test

20 of 44

One-way

Repeated-measures

ANOVA

21 of 44

What is it?

• Used when testing more than 2 experimental groups.

• In dependent groups ANOVA, all groups are dependent: each score in one group is associated with a score in every other group. This may be because the same subjects served in every group or because subjects have been matched.

• Within-subjects design equates conditions prior to experiment by using same participants in each condition or participants matched on some variable(s) of interest. This removes the single largest contributing factor to variance: individual differences.

22 of 44

Suppose we want to test whether participants can identify a target faster if there are fewer distractor items. Participants must find the single target (T or F) in a letter array and then push either the T or F button on the keyboard.

Each person is tested under three conditions: 10, 15, and 20 distractor items. There are 10 trials at each of the three levels of distraction, and the 10 trials are summed to give the total search time for each distraction level.

Examining Sources of Variance in repeated measures ANOVA: An Example

23 of 44

One-Way Repeated Measures ANOVA

• Used when testing 3 or more experimental groups.• Each person contributes more than one score

(i.e., every participant is exposed to every treatment).

• Within-subjects designs:- equate conditions by using same participants in each condition.

- variance is partitioned into SST, SSM and SSR

- in repeated-measures ANOVA, the model and residual sums of squares are both part of the within-participant variance.

SST

SSBG SSWG

SSModel SSR

24 of 44

Source Df SS MS F p

Between 2 83.69 41.85 32.25 <.001

Subjects 5 95.85 19.17

Error 10 122.997 1.30

Condition (time estimation in seconds)

Participants Order of Presentation

A

(10)

B

(15)

C

(20)

1 ABC 18.33 22.39 24.97

2 ACB 15.96 20.72 21.79

3 BAC 19.02 22.78 25.46

4 BCA 25.36 27.48 27.91

5 CAB 19.52 24.64 26.75

6 CBA 23.27 24.96 25.49

Mean Scores

20.24 23.83 25.40

25 of 44

Independent vs. Dependent Groups ANOVA: A few points worth noting …

• In independent-groups ANOVA, accuracy of the F-test depends upon the assumption that the groups tested are independent.

• The relationship between treatments in a repeated-measures design causes the conventional F-test to lack accuracy, which leads to an additional assumption.

• Sphericity: refers to the equality of variances of the differences between treatment levels.

• Mauchly’s test statistic• Corrections applied to produce a valid F-ratio:• Sphericity estimates < .75, use Greenhouse-Geisser estimate• When sphericity esimates > .75, use Huynh-Feldt estimate

26 of 44

The Sphericity Assumption: Equality of variances of the differences between treatment levels

Accuracy of the F test in independent-groups ANOVA depends upon the assumption that the groups tested are independent.

The relationship between treatments in a repeated-measures design causes the F test to lack accuracy. This requires an additional assumption termed sphericity.

If we were to take each pair of treatment levels and calculate the differences between each pair of scores, then it is necessary that these differences have equal variances.

Mauchly’s test examines the hypothesis that the variances of the differences between conditions are equal. The effect of violating sphericity is a loss of power (increased Type II error).

27 of 44

One-Way Dependent-Groups ANOVA: An Example

Suppose we were testing the idea that consuming an increasing amount of alcohol will make it more likely people will “eye-up” members of the opposite sex. IV: Over 4 nights, people are given either 1, 2, 3 or 4

pints of beer to drink.

DV: How many people the drinkers “eye-ball” (as measured by specialized eye-tracking goggles).

28 of 44

Number of Pints

Num

ber

of p

eopl

e o

gled

“Ogling” after 1 and 2 pints seems similar

“Ogling” increases after 3 pints!

Descriptive StatisticsMean Std. Dev. N

1 Pint 11.7500 4.31491 20

2 Pints 11.7000 4.65776 20

3 Pints 15.2000 5.80018 20

4 Pints 14.9500 4.67327 20

Mean Difference = 3.5

29 of 44

-- Compute SSM (the variability explained by the experimental effect)

-- Compute SSR (amount of unexplained variation across the conditions of

the repeated measures variable)

-- Divide by the appropriate df: (1) df for SSM = levels of the IV minus 1 (or k - 1);

(2) df for SSR = (k - 1) x (n - 1) [n = number of participants in each group]

-- F = MSM/MSR = the probability of getting a value like this by chance alone.

-- Check to see whether Mauchly’s test is significant!

Steps in the Analysis

30 of 44

Mauchly’s Test of Sphericity

Within-subjects Epsilon

Effect Mauchly’s W Chi-Square df1 Sig. GG HF LB

Alcohol .477 13.122 5 .022 .745 .849 .333

SPSS Output

Source Type III SS df Mean Square F Sig.

ALCOHOL Sphericity Assumed 225.100 3 75.033 4.729 .005

Greenhouse-Geisser 225.100 2.235 100.706 4.729 .011

Huynh-Feldt 225.100 2.547 88.370 4.729 .008

Lower-bound 225.100 1.000 225.100 4.729 .042

Error Sphericity Assumed 904.400 57 15.867

Greenhouse-Geisser 904.400 42.469 21.296

Huynh-Feldt 904.400 48.398 18.687

Lower-bound 904.400 19.000 47.600

Note: If the Epsilon estimates <.75, use Greenhouse-Geisser (GG);

If the Epsilon estimates > .75, use Huynh-Feldt (HF).

31 of 44

SPSS Output: Post hoc tests

(I) Alcohol (J) Alcohol

Mean

Difference

(I - J) Std. Error Sig.

Lower Bound Upper

Bound

1 2

3

4

5.000E-02

-3.450

-3.200

.742

1.391

1.454

1.000

.136 .242

-2.133

-7.544

-.7.480

2.233

.644

1.080

2 1

3

4

-5.000E-02

-3.500

-3.250

.742

1.139

1.420

1.000

.038

.202

-2.233

-6.853

-7.429

2.133

-.147

.929

3 1

2

4

3.450

3.500

.250

1.391

1.139

1.269

.136

.038

1.000

-.644

.147

-3.485

7.544

6.853

3.985

4 1

2

3

3.200

3.250

-.250

1.454

1.420

1.269

.242

.202

1.000

-1.080

-.929

-3.985

7.480

7.429

3.485

32 of 44

2 = MSM - MSR

MSM + ((n-1) x MSR)

Calculating Effect Size

2 = 75.03 - 15.87 = 59.16 = 59.17 = .16

75.03 + ((20 - 1) x 15.87) 75.03 + 301.53 376.56

33 of 44

The results show that the number of people eyed-up was significantly affected by the amount of alcohol consumed, F(2.55, 48.40) = 4.73, p<.05. Mauchly’s test indicated that the assumption of sphericity had been violated, 2(5) = 13.12, p<.05, therefore degrees of freedom were corrected using Huynh-Feldt estimates of sphericity ( = .85). The effect size indicated that the effect of alcohol consumption on ogling was substantial. Bonferroni post-hoc tests revealed a significant difference in the number of people eyed-up only between 2 and 3 pints, p<.05. No other comparisons were significant (all ps > .05).

Reporting the Results

34 of 44

Sample Problem: One-way Repeated-measures ANOVA

A stress management therapy group instructor conducts a study to determine the most effective relaxation technique(s) for stress reduction. 20 members of his stress management group participate in the study. The heart rate of each participant is monitored during each of five conditions. Each participant experienced all five conditions during the same session to control for variations in the amount of stress experienced from day to day.

The five conditions are as follows: (1) baseline (subjects sat quietly for 15-minutes); (2) guided meditation (subjects listened to a tape instructing them to close their eyes, breathe deeply, and relax their muscles for 15 minutes while concentrating on a single word or phrase); (3) comedy (subjects listened to the act of a stand-up comedian on a tape cassette for 15 minutes); (4) nature (subjects listened to a tape for 15 minutes of various sounds of nature, including the sounds of the ocean, wind, rain, leaves rustling, and birds chirping); and (5) music (subjects listened to a tape of a collection of easy-listening music for 15 minutes). Every subject experienced the baseline condition first; however, the four treatment conditions were counterbalanced to alleviate the possibility of order effects.

Each subject’s heart rate was monitored continuously during each of the 15-minute periods. The mean heart rate (beats per minute) for each subject during each condition is presented on the following page.

In this problem, we are testing the null hypothesis that, on average, the heart rates of subjects remain the same during each of the five conditions - or - that these conditions do not influence heart rate differentially. What is the IV? What is the DV?

35 of 44

Sample Problem Data Set

1 85 70 75 71 74

2 79 69 73 70 72

3 91 82 87 83 86

4 93 80 85 79 84

5 92 80 86 81 87

6 87 79 83 80 81

7 84 72 77 73 76

8 78 69 74 71 73

9 79 69 73 70 72

10 80 71 74 72 73

11 80 72 76 74 75

12 97 80 89 82 87

13 88 78 82 80 82

14 94 79 84 80 84

15 75 60 68 62 66

16 76 67 72 69 70

17 90 77 83 76 83

18 86 75 80 77 80

19 94 84 88 85 87

20 70 59 64 58 62

Subject

Baseline

Meditatio

n

Comedy

NatureMusic

Subject

Baseline

Meditatio

n

Comedy

NatureMusic

36 of 44

37 of 44

38 of 44

Move each of the within subjects variables over to the “Within-

Subjects Variables” box.

39 of 44

40 of 44

41 of 44

Which estimate do you use?

Remember!: If the Epsilon estimates <.75, use Greenhouse-Geisser (GG);

If the Epsilon estimates > .75, use Huynh-Feldt (HF).

42 of 44

SPSS OutputSource SS Df Mean

SquareF Sig.

Exp. Cond

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

1573.100

1573.100

1573.100

1573.100

4

1.654

1.792

1.000

393.275

951.015

877.881

1573.100

235.531

235.531

235.531

235.531

.000

.000

.000

.000

Error

Sphericity Assumed

Greenhouse-Geisser

Huynh-Feldt

Lower-bound

126.900

126.900

126.900

126.900

76

31.428

34.047

19.000

1.670

4.038

3.727

6.679

Within-Subjects

Effects

Descriptive

Statistics

Condition Mean Std. Dev N

Baseline

Meditate

Comedy

Nature

Music

84.90

73.60

78.65

74.65

77.70

7.511

6.969

7.036

7.006

7.420

20

20

20

20

20

43 of 44

44 of 44

Western people can become obsessed with body weight and diets, and because the media continues to glamorize stick-thin celebrities, we end up depressed that we’re not perfect. This gives corporate moguls in the fashion industry the opportunity to jump on our vulnerability by making loads of money on diets that will apparently help us to attain beautiful bodies. A European company bursts onto the scene with a diet called the Mediterranean-Monaco diet. The basic principle is that you eat no meat, drink lots of green tea, eat lots of bread dipped in extra virgin olive oil, eat chocolate at least once per day, and drink red wine (for the health benefits, of course) at the rate of at least 1.5-oz. glass per day. Ten people in need of losing weight agree to try the diet for two months. Their weight was measured in Kilograms at the start of the diet and then after 1 month and 2 months. Did the diet work?

Before Diet

After 1 Month

After 2 Months

63.75 65.38 81.34

62.98 66.24 69.31

65.98 67.70 77.89

107.27 102.72 91.33

66.58 69.45 72.87

120.46 119.96 114.26

62.01 66.09 68.01

71.87 73.62 55.43

83.01 75.81 71.63

76.62 67.66 68.60

Sample Problem