Introduction to Design and Analysis of Experiments:

a LISA short course

Jonathan Stallings, MSSeptember 20, 2011

My Qualifications

3rd Year PhD Statistics Student BS in Mathematics, UMW MS in Statistics, VT Main research interest is Experimental Design

I enjoy helping researchers answer their specific questions by giving them the best possible “tools” to collect and analyze their data.

A little about LISA

Laboratory for Interdisciplinary Statistical Analysis Free Collaboration:

– Experimental Design – Data Analysis – Softward Help

Interpreting Results – Grant Proposals Free Walk-In Consulting for quick statistics questions Free Short Courses

– JMP and R tutorial; Case Studies

Improve research quality through project collaboration and statistical consulting

Requesting a LISA Meeting

To request a collaboration go to: www.lisa.stat.edu Sign in using VT PID and password Enter your information (e-mail, college, etc.) Describe your project (project title, research goals,

specific research questions, do you have data, etc.) Contact assigned LISA collaborators as soon as possible to

schedule meeting

We prefer to meet with you before data collection!

Short Course Goals

Introduce terminology used by statisticians to make collaboration easier

Detail fundamentals of a good design Understand when one design is better than another Explain how to analyze data to answer research

questions

What we won't be talking about

Survey Design (Observational Study) Finding necessary sample sizes Assumption Checking Measurement Error

These are important design questions, but involve more advanced design. LISA collaboration meetings are ideal for these questions.

Observational Studies

When the researcher has no control over what's given to subjects and simply observes what's happening

Examples– Surveys

– Investigating effects of cancer on human subjects

– Observing any natural phenomenon These types of studies relate to the statement:

Correlation does not imply causation

Designed Experiment

The researcher can assign any treatment to any subject Example:

– Assign different medications to subjects with a similar illness

– Assign different credit card limit rates to customers with a similar financial situation

– Assign different amounts of carcinogen to lab rats.

Why Design an Experiment?

With a designed experiment we– Control over what we give subjects– Account for sources of variation– Control → Causation

By source of variation we mean some effect (inherent or applied) that may cause the response to change.

By designing an experiment, the researcher can maximize information that answers their specific research questions.

Design Example Your child comes home from school and shows you what they

learned in class. He/she asks for a film canister and an Alka-Seltzer tablet. They

fill up the canister with a little water, put the tablet in the water, close the canister and turn it upside down.

After a few seconds, the canister flies in the air! Your child wants to know how to make the canister fly as high as possible.

= BOOM!

http://www.youtube.com/watch?v=Gtbane7BBdQ&feature=related

Design Example

Question: Does the amount of alka-seltzer affect flight time? Which amount gives the best time?

The different amounts of alka-seltzer are– 1 1/2 tablets– 1 tablet– 1/2 of a tablet

For now, we will reuse the same film canister The response is the amount of time from liftoff to

landing in seconds

Design Example

What are some sources of variation?– Amount of alka-seltzer (we control this)– Amount of water– Film canister seal– Time Measurement– Angle of liftoff

There may be more, let's choose the ones that we think will be most signficant and difficult to control

Design Example

In order to answer our research goals, we need to “remove” the significant sources of variation other than the alka-seltzer amount.

We need to assign alka-seltzer amounts to the canisters in a way that will minimize the variance of the effect estimates.

A design must do both of these things in order for the researcher to confidently reach conclusions!

Fundamental Principles

A treatment is something controlled and administered by the researcher to an experimental unit (EU)

– Emphasis on the researcher administering the treatment!– An experimental unit can also be thought of the physical

entity assigned to a treatment (very broad and circular definition)

Essentially a design is the proposed allocation of treatments to experimental units (or vice-versa)

The purpose of the design should be to – Detect differences among different treatments– Estimate and compare treatments effects

Fundamental Principles

There are three fundamental concepts to any design:− Replication of treatment− Randomization of treatment assignment− Blocking of EU's (Experimental Units)

Neglecting to acknowledge these will result in unreliable results and immediate skepticism

Film Canister Experiment

Treatments: Three different amounts of Alka-Seltzer EU's: For now, we reuse the same film canister. The

EU's will technically be the time we use the film canister, so 1st time, 2nd time, 3rd time, etc.

Again, we assume that the EU's can be reused without any after effects, i.e. they are independent

How do we use the fundamental principles to design this experiment?

Replication

Replicating a treatment means assigning that treatment to multiple EU's

Will reduce variance of estimates of that treatment's effect

If we have equal interest in all the treatments, we want to try to equally replicate the number of treatment assignments

FC Example: There are three treatments (tablet size) and say we reuse the canister 9 times. So 9/3=3 reps

Randomization

Randomly selecting which EU gets a treatment How we randomize depends on the type of design Clearly we must randomize before measurements are

taken Reduces most types of bias (systemic bias especially) FC Example: We know that we want to replicate each

tablet size 3 times. A biased, but almost natural, way to do this is to start with the one treatment, do it 3 times, move to the next, etc. Randomizing the order would remove any systemic bias.

Blocking

Technique that groups EU's so that each block contains EU's that are more “homogeneous”

If done properly, this can account for variance that would otherwise be considered as “noise” or “error”

Block effects are referred to as nuisance parameters because they are “getting in the way” of the estimation of treatment effects

BLOCK EFFECTS CANNOT BE ASSIGNED

Blocking

FC Example: Maybe we want to use three film canisters which we feel may be significantly different from each other. So we block by canister.

BlockEach box represents an EU with the block trait

Each box represents an EU with the block trait

9 EU's in each block, call this “block size”

Analysis is based on this model Basic Components:

− Response: yij where i is treatment, j rep number

− Mean: μ i

− Variance: eij random variable, Normal(0,σ2)

yij = μ i + eij

The Linear Model

We can “decompose” thisparameter into a sum of different effects.

This is random! What we want to know from this is the variance.

Least Squares

To find estimates for the parameters we use the method of least squares, i.e. we try to minimize the function

Σ( yij – μi )2

with respect to the parameters μi .

Later we'll see that the common way to decompose the mean parameter yields biased and non-unique solutions.

There are ways to still get information pertinent to the experiment despite this using contrasts.

One Source of Variation: The CRD

The simplest design assumes all the EU's to be similar so the only significant source of variation would be the different treatments

A completely randomized design (CRD) will randomize all the treatment-EU assignments for the specified number of treatment replications

Result: Getting as close as possible to equally

replicating the treatments will minimize the variance of estimates

CRD Example: FC ExperimentThe Design Plan:

Before Randomization

These are “similar” EU's

1/2 Tablet 1 Tablet 1 1/2 Tablet

1

2

3

4

5

6

7

8

9

Time Order

CRD Example: FC ExperimentThe

Implemented Design

1

2

3

4

5

6

7

8

9

CRD: The Linear Model

Say we have v treatments, replicated r times so we need vr EUs

yij = μi + eij where i=1,...,v j=1,...,r

We decompose mean as μ + τi

yij= μ + τi + eij

μ is the overall effect, τi is treatment effect

The first thing we want to know is if there is any difference between the v treatments

We test the null hypothesis that there is no difference using the following identity:

Σ (yij – y..)2 = Σ r(yi. - y.. )

2 + Σ (yij – yi. )2

SSTot = SSTrt + SSErr

Analysis of CRD: ANOVA Review

This is the main part of the variance estimate if the treatment means are the same.

Variance due to differences in treatment

Variance we attribute to other sources of variation

Analysis of CRD: ANOVA Table

Effect df SS MS E(MS)

Trt v-1 SSTrt SSTrt / (v-1) σ2 + Q( τi )

Error n-v SSErr SSErr / (n-v) σ2

Total n-1 SSTot Q(τi)= [Σ r(τi - τ. )

2]/(v-1)

If all the treatments are equal then Q(τi)=0 and the ratio MSTrt/MSErr ≈ 1.

MSTrt/MSErr follows an F-distribution which gives us p-values

Knowing that a difference exists is not enough; we want to estimate the treatment effects and compare them to each other.

Using least squares to estimate the parameters, we find that the estimates are biased and not unique! So what can we estimate?

It turns out that E(yij )= μ + τi is unbiased and unique regardless of individual parameter estimates. The best estimator for this is the average of replications for treatment i.

Analysis of CRD: Estimability

Knowing that expected values of responses are unbiased, we know that we can estimate any linear combination of them using linear combinations of the treatment averages.

Σ bi E(yi.) = Σbi(μ + τi)= μ Σ bi + Σ biτi

The most important type of linear combinations are when Σ bi = 0. These are called treatment contrasts.

Examples: τ1 - τ2, (τ1 + τ2)/2 – (τ3 + τ4)/2

Analysis of CRD: Estimability

Analysis of CRD: Example

Fill the canister halfway with water for each run– Replicate each treatment 6 times

Analysis should start with plotting data How to read ANOVA table Inference and treatment comparisons

Contrast Comments

Be careful if doing multiple hypothesis testing of contrasts and other estimable functions– Alpha level must be lowered because overall Type I

error rate increases– Scheffe, Bonferroni, and Tukey methods for

multiple comparisons frequently used Advanced technique: trend contrasts used when the

treatments are numerical and evenly spaced

CRD Extension: Factorial Experiments

We may be able to attribute multiple factors with different levels to a single treatment.

Example: For the FC experiment we may also vary water amount (low/medium/high). In this case one “treatment” is actually a combination of tablet and water amount

The different tablet amounts and water amounts are called the levels of the tablet factor and water factor, respectively.

Factorial Example: FC Experiment1/2 tablet / Low water

1 1/2 tablet / High Water

1

2

3

4

5

6

7

8

9

Factorial Example: FC Experiment

1

2

3

4

5

6

7

8

9

Linear Model: Factorial

We “reparameterize” the CRD model asyijk= μ + τij + eijk

yijk = μ + αi + βj + (αβ)ij + eijk

Key Idea: Consider a treatment effect being composed of different factors.

αi and βj are called main effects

(αβ)ij is the interaction effect

Interpreting Interactions

If two factors interact then the effect of one factor level depends on the level of the other factor.

Interaction plots:

Parallel lines imply the treatment effect is about the same regardless of the other effect

The effect of Subordinate Performance clearly depends on the other treatment

Factorial Analysis

We use the same ANOVA approach, but decompose SSTrt to account for different factors.

SSTrt=SSA+SSB+SSABΣiΣj(yij. - y...)

2 = brΣi(yi.. - y...)2 + arΣj(y.j. -y...)

2

+ rΣiΣj(yij.- yi.. - y.j. + y...)2

Contrasts of main effects will include the interaction effect.

Factorial Analysis: Example

Vary water and alka-seltzer amount (3 levels each) so 3x3=9 combinations total

Assume we only do 9 runs, what can we do?– This is called “saturated” because the number of

parameters is greater than the number of observations

Factorial Comments

The minimum number of EU's needed is ab where a and b are the number of levels in factor A and B.

We may extend to more than two factors, but the number of EU's grows rapidly!

There exist design techniques for cases where the number of factors large and only few EU's called fractional factorials.

10 minute break

Find the bi for contrasts (say we have 4 total trts)

τ1 - τ2

(τ1 + τ2)/2 – (τ3 + τ4)/2 Name the EU's and treatments in this experiment

– Have 9 similar “chambers” each holding 4 similar plants. We fill each chamber with one of 3 different types of gas and measure the plant growth before and after.

Reducing Variance by Blocking

Recall that we group EU's so that each block (group of EU's) is more homogeneous

If we compare treatments within blocks or are able to “remove” the block effect then we will get more consistent results

If we didn't block then it may be that heterogeneity of the EU's “gets in the way” of the treatment effects.

If blocking has a significant effect, we can greatly reduce the variability of the treatment effects.

Block Examples

From FC example, we blocked by canister Male and Female Plots in a field (close together more similar) Note that in all of these cases, we cannot assign a

block to an EU, it is an inherent property of the EU

Assigning treatment to blocks

Remember, we don't want to estimate the block effects! We want to “remove” their effect from treatment effects

If we use the same treatment in all the blocks then we are confounding the treatment effect with the block effect (we can't separate treatment effect from block effect)– E.g. Assign 1/2 tablet to all-black canister.

The simplest block design happens when we can assign each treatment once to each block.

Block Design: RCBD

If the block size equals the number of treatments we call this a randomized complete block design.

You can almost think of this as separate CRD's for each block. By that I mean we know we want all the treatments once in each block and we

RANDOMIZE TREATMENTS IN EACH BLOCK In the corresponding ANOVA, we have SSBlock but

we cannot assess the significance of them!

RCBD Analysis: FC Example

1 11 2

1 32 12 2

2 33 13 23 3

1 11 2

1 32 12 2

2 33 13 23 3

1 11 2

1 32 12 2

2 33 13 23 3

Recall, the EU's in the blocks are the time order of reuses of same canister

1 1 means 1/2 tablet, low water; 3 3 means 1 1/2 tablet, high water

Recall, we randomize within each block (3 total randomizations)

RCBD Analysis: FC Example

1 22 3

3 12 11 1

2 23 31 33 2

2 13 1

3 22 31 3

1 21 12 23 3

1 13 1

3 33 22 1

1 32 31 22 2

RCBD: Linear Model

Add block effect βj to CRD model:

yij = μ + βj + τi + eij

j=1,..., # blocks i=1,..., # treatments Again, each treatment in a block once so the number

of reps for a treatment equals the number of blocks

RCBD Contrasts

Note E( yi. ) = μ + β. +τi = constant + τi

So the contrasts we looked at for CRD still apply since the constant will cancel out!

Question: What about contrasts of the block effects?

No! Block effects have little statistical purpose outside of reducing variance!

Assessing block efficiency

If we can't test for block significance how do we know blocking was a good idea? We hope that blocking will account for some of the SSErr.

Quick check is to compare the MSBlocks and MSErr although there is no statistical foundation for this. If MSBlocks about same as MSErr, then blocking was probably a bad idea.

Can we do analysis like CRD if we find out blocking was a bad idea? No! The randomization did not follow CRD protocol.

RCBD: Analysis

Here we use 6 different film canisters (blocks) and have 9 runs for each block = 54 totals observations

We assess blocking efficiency the best we can

More block designs

There are many more types of block designs that fit more real life situations

Block size:– # of EU's in block < # treatments

– # of EU's in block > # treatments

– # of EU's in any block are not equal Block factors:

– More than one block factor

These are advanced design scenarios and we highly recommend visiting LISA to discuss what to do.

Incomplete Block Design: Analysis

Block size: # EU's in block < # treatments Useful if # treatments large (factorials) because large

blocks tend to be less homogeneous Key Concept: Balance:

– We want the treatments to be equally replicated across all blocks and each pair of treatments to appear in the same block an equal number of times.

For any given number of treatments, blocks, and block size, it is usually difficult to find such a design.

IBD Example: Balanced IBD

This is a Balanced IBD with

– 8 treatments

– 14 blocks, 4 EU's in a block

For the Balanced Incomplete Block Design, we have equal variance for all pairwise comparisons so their confidence intervals have the same length.

123

4

125

6

127

8

135

7

136

8

145

8

146

7

235

8

236

7

245

7

246

8

345

6

347

8

567

8

# reps = 7“block concurrence” = 3

Two blocking factors

Similar to multiple treatment factors If we think block factors interact then we need many

EU's in order to estimate the interaction If no block interaction, then only one EU per block

combination is needed. We call these Row-Column block designs

Row-Column Design

We consider two types:– Latin Square designs: the number of blocks

for the two blocking factors equals the number of treatments

– Youden Square designs: number of blocks for one block factor equals # treatments, the other block factor has less blocks

It's useful to think of these as combinations of RCBDs and IBDs.

Other Design Techniques

Analysis of Covariance:– Each EU has an inherent quantitative variable we may consider to

be a source of variation (e.g. weight)

Repeated Measures:– We take repeated measurements on the same EU (usually a human

patient) so that each subject serves as its own control

Fractional Factorials:– The total number of EU's is less than the number of factor

combinations (usually because we have lots of factors)