One-way ANOVA (Single-Factor CRD) - University of Iowahomepage.stat.uiowa.edu/.../notes/2-10_one_way_ANOVA.pdfOne-way ANOVA We have already described a completed randomized design

One-way ANOVA(Single-Factor CRD)

STAT:5201

Week 3: Lecture 3

1 / 23

One-way ANOVA

We have already described a completed randomized design (CRD)where EUs are randomly assigned to treatments. There is no blockingand no nesting in a CRD.

We will now take a closer look at the model for a CRD when there isonly one factor.A /Wf N/4U- e~~ '-~ tdd/0;

tfJh,o; ~ CL- 2- -/ adfL £;Crl/;wJ--.

/\

~/ = fl,_ W-e- ~)uJ C~~5 a: r~p.IC!r:d2tJ'(01 ~(;~J,6n I:uC'£uv~ W-.e w£ d-pCv1.~ 10 du~,'& d nv~

~ ~~ fYcvt~ cI~ ~reJ1fY' tuLea: for C>-(..V .[ ~ CL ~o.l-~ £/VLR c:vr; W ~ r ~-u-: &--0-.qa.X.{J ~CtA-")) II.or ~ 1a/L 0{. ~~hvcv"l u:> -J ~ CVV! '" -e.: ~J~ Y"\.P a:A-1. / e:»w-e- ~ ~ +L ~JUVYlcd-idV'- fin?? ~ --C;;. k )

A ;[z..(}/-.- y-: )'2-/J '2.-::: i j '~ If

N-J

We will consider two models (or parameterizations) for describing thesingle-factor CRD here. The first is called the cell means model andthe second is called the effects model. We will mostly use the latter.

2 / 23

Notation: ‘dot and bar’ (sum and average)

Let Yij be the jth response in treatment i . We have i = 1, 2, . . . , ggroups and j = 1, 2, . . . , ni where the number of observations fromeach group does not have to be the same.

Let Ȳi · =∑ni

j=1 Yijni

be the mean response in the ith treatment group(stated as “Y-bar” or a ‘cell mean’).

Let Ȳ·· =∑g

i=1

∑nij=1 Yij

N be the grand mean response or the overallmean.

N =∑

i ni is the total number of observations in the study.A /Wf N/4U- e~~ '-~ tdd/0;tfJh,o; ~ CL- 2- -/ adfL £;Crl/;wJ--.

/\

~/ = fl,_ W-e- ~)uJ C~~5 a: r~p.IC!r:d2tJ'(01 ~(;~J,6n I:uC'£uv~ W-.e w£ d-pCv1.~ 10 du~,'& d nv~

~ ~~ fYcvt~ cI~ ~reJ1fY' tuLea: for C>-(..V .[ ~ CL ~o.l-~ £/VLR c:vr; W ~ r ~-u-: &--0-.qa.X.{J ~CtA-")) II.or ~ 1a/L 0{. ~~hvcv"l u:> -J ~ CVV! '" -e.: ~J~ Y"\.P a:A-1. / e:»w-e- ~ ~ +L ~JUVYlcd-idV'- fin?? ~ --C;;. k )

A ;[z..(}/-.- y-: )'2-/J '2.-::: i j '~ If

N-J

3 / 23

One-way ANOVA: Cell means model

Cell Means Model

Yij = µi + �ij

with �ijiid∼ N(0, σ2)

for i = 1, . . . , g and j = 1, ..., ni

We have one mean parameter µi for each cell, or separate group.

This is the same as Yij ∼ N(µi , σ2) .

4 / 23


Cell Means Model

Yij = µi + �ij with �ijiid∼ N(0, σ2) for i = 1, . . . , g and j = 1, ..., ni

The estimates for the mean-structure parameters are simply the cellmeans, or µ̂i = Ȳi ·

Estimate for the noise: σ̂2 =∑

i

∑j (Yij−Ȳi·)2

N−g where N =∑

i ni

Positive CharacteristicsWe do not need any constraints or restrictions for estimation becausewe use g parameters to describe g means. µ̂i is the estimated groupmean.Estimates are easy, very intuitive.

Negative CharacteristicThe estimated parameters don’t directly tell us how far a treatmentmean is from the overall mean, nor how far a treatment mean is fromanother treatment mean (but we can get this information from our µ̂ivalues.) 5 / 23


Cell Means Model

Yij = µi + �ij with �ijiid∼ N(0, σ2) for i = 1, . . . , g and j = 1, ..., ni

The Design matrix X is of full rank for the cell means model.Again, very easy to work with, intuitive.

Example (1-way ANOVA with g = 3 and n = 2)

Suppose we have a one-way ANOVA framework with g = 3 and n = 2 foreach group. In the cell means model, the design matrix is of rank 3 andhas 6 rows and 3 columns.

µ1 µ2 µ3

X =

1 0 01 0 00 1 00 1 00 0 10 0 1

Letting Y = Xµ+ � using OLS we have

µ̂ =

µ̂1µ̂2µ̂3

= (X ′X )−1X ′Y = Ȳ1·Ȳ2·

Ȳ3·

6 / 23

One-way ANOVA: Effects model

Effects Model

Yij = µ+ αi + �ij

with �ijiid∼ N(0, σ2)

for i = 1, . . . , g and j = 1, ..., ni

In this model, we use g + 1 parameters to describe g means. This isan overparameterization.

- One parameter µ for the overall mean.- One parameter αi for each group.

This is the same as Yij ∼ N(µ+ αi , σ2) .

7 / 23


Effects Model

Yij = µ+αi + �ij with �ijiid∼ N(0, σ2) for i = 1, . . . , g and j = 1, ..., ni

“µ+ αi” represents the mean of a group.

Because this is an overparameterization, we need a constraint tomake the parameters ‘identifiable’ (i.e. uniquely determined).

One option is to use the sum-to-zero constraints which providesintuitive interpretation of the parameters. For balanced data, this is∑g

i α̂i = 0 and we use the estimates of

µ̂ = Ȳ·· α̂i = Ȳi · − Ȳ·· σ̂2 =∑

i

∑j (Yij−Ȳi·)2

N−g .

Here, µ̂ represents the overall mean and α̂i represents the distancethat group i is from the overall mean. Some α̂i values will be positiveand some will be negative.

8 / 23


Effects Model

Yij = µ+αi + �ij with �ijiid∼ N(0, σ2) for i = 1, . . . , g and j = 1, ..., ni

Positive Characteristics

In the sum-to-zero constraints, the estimated parameters directly tellus how far a treatment mean is from the overall mean. The effectsare just deviations from the grand mean.

Negative Characteristic

We need a constraint or restriction on the parameters for estimationdue to overparameterization.

* No statistical software uses sum-to-zero-constraints by default, but wewill use these when calculating estimates by hand (it’s easiest).

* SAS uses a constraint that sets the last α̂i = 0. By default, R sets thefirst α̂i = 0. In these constraints µ no longer represents the overallmean, but the mean of a specific ‘reference’ group.

9 / 23

One-way ANOVA

The choice between these models (cell means model or effects model)or constraints does affect the interpretation of the parameters, butthe important estimates are the same under any of these choices...

* Fitted Ŷ values* Differences between groups or µ̂i − µ̂j* Residual �̂ij values

In a one-way ANOVA, we perceive a scenario where we have distinctgroup means, with normally distributed errors around the means, andwe are interested in comparing group means. This perception is thesame regardless of the model and constraint choices above.

10 / 23

One-way ANOVA

Unbalanced data in One-way ANOVA

If you have unbalance data, nl 6= nk for some l , k and you are usingthe effects model, then the grand mean

µ̂ = Ȳ·· =∑

i

∑j Yij

N =∑

i

∑j Yij∑

i ni=

∑i ni µ̂iN

looks like a weighted average of the group means.

µ̂ will be pulled toward the larger groups.

The sum-to-zero constraints are∑g

i ni α̂i = 0

Estimates of the effects are shown with the same formula asdeviations from the grand mean which is α̂i = µ̂i − µ̂

But most of the time we will have balanced data, so I will usuallystate the constraints on the board as

∑gi α̂i = 0

NOTE: if ni = nj for all i , j then∑g

i ni α̂i = 0 ⇒∑g

i α̂i = 0.

11 / 23

One-way ANOVA: Sums of Squares

ANOVA - The partitioning of the sums of squares is called Analysis ofVariance, or ANOVA.

In an ANOVA, we break down the total variability in the data intocomponent parts, i.e. into the differing sources of variation.

Consider the one-factor experiment:

Yij = µ+ αi + �ij with �ijiid∼ N(0, σ2)

for i = 1, . . . , g and j = 1, ..., ni

We analyze such data as a “1-way ANOVA” with only one factor andthe hypothesis test of interest isH0 : µ1 = · · · = µg vs. H1 : at least one group is not equal

If we reject this null hypothesis, we usually do follow-up comparisonsto see which of the groups are statistically significant from each other.

12 / 23


Total Variation: SSTOT =∑

i

∑j(Yij − Ȳ··)2

This is the total sum of squares (corrected for the mean).

Variation due to Treatment: SSTRT =∑

i ni (Ȳi · − Ȳ··)2This is the treatment sum of squares.

This quantifies how far the groups means are from the overall mean.

Unexplained Variation: SSE =∑

i

∑j(Yij − Ȳi ·)2

This is the sum of squares for error.

This quantifies how far the individual observations are from their group mean.

We know SSTOT = SSTRT + SSE

*Fundamental ANOVA identity

13 / 23


At a minimum, an ANOVA table will list the sources of variation inthe experiment and their degrees of freedom. We usually also includethe sum of squares (SSx) and the related mean squares (MSx).

Here is a general layout for a 1-way ANOVA:

14 / 23

One-way ANOVA: Correcting for the mean

We almost always estimate an overall mean in our model, so we lose1 degree of freedom (d.f.) right away.

Thus, we essentially start with N − 1 d.f., and say the total sum ofsquares is “corrected for the mean”.

Once we have our overall mean estimated (or µ̂), then we only needg − 1 more parameters to describe the mean structure (i.e. todescribe the g cell means). Thus, we use g − 1 d.f. for Treatment.

The leftover N − g d.f. are given for estimation of the error.

15 / 23

One-way ANOVA: Example

Example (Response time for circuit types)

Three different types of circuit are investigated for response time inmilliseconds. Fifteen are completed in a balanced CRD with the singlefactor of Type (1,2,3).

Circuit Type Response Time

1 9 12 10 8 152 20 21 23 17 303 6 5 8 16 7

From D.C Montgomery (2005). Design and Analysis of Experiments. Wiley:USA

16 / 23



17 / 23



See handout for annotated output.18 / 23

One-way ANOVA: MSTRT and MSE

Why is the ANOVA table useful?The MS values will be used to perform statistical tests.


PROC GLM automatically generated plot in HTML output:

/*Fit the 1-way ANOVA model*/

proc glm data=circuits plot=diagnostics;

class type;

model time=type;

output out=diagnostics p=predicted r=residual;

run;

The GLM Procedure

Class Level Information

Class Levels Values

type 3 1 2 3

Number of Observations Read 15

Number of Observations Used 15

Dependent Variable: time

Sum of

Source DF Squares Mean Square F Value Pr > F

Model 2 543.6000000 271.8000000 16.08 0.0004

Error 12 202.8000000 16.9000000

Corrected Total 14 746.4000000

2We need to know what we EXPECT to get from MSTRT and MSE ...

E (MSTRT ) = σ2 +

∑gi niα

2i

g−1 E (MSE ) = σ2

If H0 : µ1 = µ2 = µ3 ⇐⇒ αi = 0 ∀i is true, then E (MSTRT ) = σ2,and MSTRT and MSE should be similar.

If HA : αi 6= 0 is true for at least one i , then MSTRT > MSE .19 / 23


We base our statistical test on the ratio of MSTRTMSE .

Under H0 true, Fo =MSTRTMSE

∼ F(g−1,N−g) and we expect a value near1 for our Fo (in general).

Under HA true, Fo has a stochastically greater distribution thanF(g−1,N−g) and we reject the null if Fo > F(g−1,N−g ,0.95)

20 / 23



Circuit data

Fo =271.816.9 = 16.08 compared to F(2,12) to get p-value.

p-value is 0.0004↑

Only valid if model assumptions are met(we’ll return to checking the assumptions for this model soon).

21 / 23

One-way ANOVA: Full vs. Reduced Models

The overall F -test in a 1-way ANOVA is actually a test for comparinga full model and a reduced model that is “nested” in the full model.

- A reduced model is nested in a full model if it is aparticular case of the full model.

NOTE: we will use the design term nested in another waylater, so be aware of this.

The ANOVA table in the 1-way ANOVA compares a full model(requiring g parameters to describe the mean structure) and a reducedmodel (requiring only 1 parameter to describe the mean structure).

Thus, we can think of the F -test as comparing a full and reducedmodel.

22 / 23

SIDENOTE: SAS Settings

1 On the first line of all my SAS code files, I set the following options:

options linesize = 79 nocenter nodate formchar = "|----|+|---+=|-/\*" ;

2 I set my preferences to have SAS output the results in both ‘listing’ and HTMLformat.

The HTML output is nice because you automatically get HTML graphicsgenerated, but I’ve also found the HTML output difficult to deal with at times aswell (like when I’m trying to save pieces of it).

Therefore, I also generate all my output as a ‘listing’. If you are on virtual desktop,I know you can choose this option by going to...Tools → Options → Preferences...Click the Results tab, and check the box that says ‘Create Listing’. Then OK.

This listing output is just text and you can easily copy and paste the pieces intoLaTeX and use the ’verbatim’ environment to present it. If you copy and past intoWord, you might use a monospace font, such as Andale Mono or SAS monospace.If you save your listing output it will be as a .lst file.

23 / 23

First section

Documents

One-way ANOVA (Single-Factor CRD) - University of Iowahomepage.stat.uiowa.edu/.../notes/2-10_one_way_ANOVA.pdfOne-way ANOVA We have already described a completed randomized design