Keep in touch - University of Cape Town€¦ · Observational Experimental # # random sample randomize # # representative equalize groups (average out all other factors) # # make

Keep in touch

@SEEC UCT

SEEC.UCT

ask to be added to mailing list([email protected])

www seec.uct.ac.za

Experimental and Survey Design

I Why?

I What?

I How?

Yoccoz, N. G., J. D. Nichols, and T. Boulinier. 2001. Monitoring of biological diversity in space and time. Trends

in Ecology & Evolution 16:446 - 453.

Experimental and Survey Designs

from Mackenzie et al 2006, Academic Press

6 / 55

Experiments: what they are and why we need them

I carefully controlled environment

I everything stays constant

I except the factor of interest (treatment factor)

I measure change in response

I infer causal relationships

7 / 55

Observational vs experimental studies

Hypothetical example: Vegetation in relation to temperature andaltitude

observational

●

●

●

●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

temperature

altit

ude

low high

low

high

experimental

temperature

altit

ude

low high

low

high

* * *

* * *

* * *

8 / 55

Observational vs experimental studies

Hypothetical example: Vegetation in relation to temperature andaltitude

observational

●

●

●

●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

temperature

altit

ude

low high

low

high

experimental

temperature

altit

ude

low high

low

high

* * *

* * *

* * *

9 / 55

Summary

Observational Experimental

↓ ↓

random sample randomize

↓ ↓

representativeequalize groups (average out

all other factors)

↓ ↓

make statements about thepopulation

causal inference (cannot sayanything about population in

general)

10 / 55

Three principles in experimental design

1. Randomization

2. Replication

3. Reduce (unexplained variation)

What is the experimental unit?

11 / 55

Three principles in experimental design

1. Randomization

2. Replication

3. Reduce (unexplained variation)

What is the experimental unit?

12 / 55

ED Principle 1: Randomization

Randomization

is the assignment of treatments to experimental units at random.

13 / 55

What happens if we don’t randomize?

14 / 55

What happens if we don’t randomize?

15 / 55

What happens if we don’t randomize order?

time

A A A B B B C C C

16 / 55

What happens if we don’t randomize order?

time

A A A B B B C C C

tem

pera

ture

17 / 55

Summary: Why is randomization important?

I average out systematic effects (anticipated or not), extraneousfactors not directly controlled (prevent confounding withunknown factors)

I ensures that treatment is independent of experimental unit’sproperties

18 / 55

ED Principle 2: Replication

each treatment is assigned to > 1 experimental unit, randomly andindependently

19 / 55

Uncertainty – Variability

If all experimental units were exactly identical, it would be easy toestablish cause and effect relationships.

A B

treatment

resp

onse

●

●

A B

treatment

resp

onse

●●●●●

●●●●●

A B

treatment

resp

onse

●

●

●

●

●

●●

●

●

●

20 / 55

Uncertainty – Variability

If all experimental units were exactly identical, it would be easy toestablish cause and effect relationships.

A B

treatment

resp

onse

●

●

A B

treatment

resp

onse

●●●●●

●●●●●

A B

treatment

resp

onse

●

●

●

●

●

●●

●

●

●

21 / 55

Is this proper replication?

22 / 55


23 / 55


24 / 55


25 / 55

ED Principle 3: Reduce unexplained variation

1. keep all other factors constant

2. blocking

Blocking:

is the grouping of similar (homogeneous) experimental units intoblocks.

I The experimental units in one block aresimilar.

I Each treatment occurs once in each block.

I Common blocking factors: age, spatiallocation, time (day or year)

I each block is a replicate

26 / 55

ED Principle 3: Reduce unexplained variation

1. keep all other factors constant

2. blocking

Blocking:

is the grouping of similar (homogeneous) experimental units intoblocks.

A

C

B

A

B CI The experimental units in one block are

similar.

I Each treatment occurs once in each block.

I Common blocking factors: age, spatiallocation, time (day or year)

I each block is a replicate

27 / 55

Controls

I to measure the effect of a treatment, we need to compareresponse with to without treatment

I eliminate alternative explanations for results

28 / 55

Part 2: Constrained Design Studies

One or several principles of experimental design cannot be adheredto:

I impact studies

I observational studies

29 / 55

Sampling Designs for Impact Assessment

from Green 1979, John Wiley & Sons

30 / 55

Example: Impact of a new wind energy farm

Problem formulation:I not an experiment, but impact assessmentI cannot randomly choose locationI only one farm (no replication of ‘treatment’)I impact: effect of wind farm on bird behaviour around turbinesI hypothesis: change in the local density of raptors with

presence of wind farm compared to abscence of wind farmI sampling method: vantage point surveys

31 / 55

Control and Replicate Time and Location

before

temporal control

o o o: temporal replication

spatial control


after

o o o

spatial control

o o o

32 / 55


before

temporal control


spatial control


after

o o o

spatial control

o o o

33 / 55


before

temporal control


spatial control


after

o o o

spatial control

o o o

34 / 55


before

temporal control


spatial control


after

o o o

spatial control

o o o

35 / 55

Random Sampling

Purpose: sample must be representative (of the population)

I where: where to place the vantage points

I when: times of day, season, weather conditions

36 / 55

Observational studies

No manipulation

I Cause and effect unclear

I Influence of unmeasured variables can never be ruled out(confounding)

Two things to worry about:

1. Observation process (Solution: distance sampling, repeatedvisits, multi-observer designs)

2. Representativeness


No manipulation

I Cause and effect unclear

I Influence of unmeasured variables can never be ruled out(confounding)

Two things to worry about:

1. Observation process (Solution: distance sampling, repeatedvisits, multi-observer designs)

2. Representativeness


1. Identify target population

2. Random sampling: no assumptions about the populationneeded

I every unit has a known probability of being in the sampleI sample is drawn according to these probabilitiesI selection probabilities are used when making estimates

3. Model-based sampling: unbiased if model describespopulation well
















Random sampling

I Simple random sampling

I Stratified random sampling

I Cluster sampling

I Multistage sampling

I Adaptive sampling

Density of hadeda ibises

What is the average density of hadeda ibises in this landscape?

Simple random sampling

x x

xx

xx

x x

x x

xx x x

x

xx x x x

µ = 1.98x̄ = 1.3595%CI : 0.21 − 2.49

Unbiased regardless of how densities are distributed.

Convenience sampling

x

x

xx x

x

x

x

x

x

x

xx

x

x

xx

x

x

x

µ = 1.98x̄ = 3.0595%CI : 1.91 − 4.18

Preferentially sampling high-density areas leads to estimates thatare biased high.

Stratified random sampling

x

x

xx x

x

x

x

x

x

x

xx

x

x

xx

x

x

x

µ = 1.98

x̄ = 1N

∑Lh=1Nhx̄h = 1.75

s2 =∑L

h=1(NhN )2(Nh−nh

Nh)s2hnh

95%CI : 1.05 − 2.44

Strata should be as homogeneous as possible; blocking is a relatedidea in experimental design.

Model-based sampling

x

x

xx x

x

x

x

x

x

x

xx

x

x

xx

x

x

x

x̄ = 1.71

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ●

●

●

habitat gradient

dens

ity

1 5 10 15 20

0

2

4

6

8

10

Can be unbiased if the model is good; no random sampling needed.

Cluster sampling

x

xx

x

x

x

xx

x

x

x

xx

x

x

x

xx

x

x

x

xx

x

x

x

xx

x

x

x

xx

x

x

x

xx

x

x

I Clusters selected at randomI Most efficient if variability within cluster is high and variability

among clusters is low

Multistage sampling

xx

xx

xxx

x

xx

x

x

x

x

x xx x

xx

1. Select large units at random

2. Select small units at randomwithin selected large units

Adaptive sampling

x

x

x

x

x

x

x

x

x

x

x

xx

xx

x

x x

x

x

●

●●

● ● ●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

x

x

x

x

x

x

x

x

x

x

x

xx

xx

x

x x

x

x

●

●●

● ● ●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

x x xx x xx x x xx x x xx x x x

1. Select units at random

2. If species detected, sample surrounding units

Random sampling

I Probability of being in the sample is known for each unit

I This probability is used when making inference

I Lots of designs to choose from

I Some designs are more effective than others for your situationI Things to think about:

I Scale of variability and spatial correlation (do a pilot)I Do you have informative covariates? (stratification;

model-based sampling)I Costs of travelling to sites

I Things get complicated quickly

Summary: Nine principles of good study design

1. formulate problem (design, predictors, hypothesis ...)

2. replicate time, location

3. random sample

4. control (space and time)

5. conduct a pilot study

6. worry about observation process

7. subsampling, stratified sampling where necessary

8. appropriate sample unit size

9. use an appropriate statistical modelAdapted from: Green 1979, John Wiley & Sons

Key references

Experimental design:

I Underwood, A. J. 1997. Experiments in ecology: their logical design andinterpretation using analysis of variance. Cambridge University Press.

I Montgomery, D. C. 2012. Design and Analysis of Experiments, 8th edition.Wiley.

Constrained designs / impact studies:

I Green, R. H. 1979. Sampling design and statistical methods for environmentalbiologists. John Wiley & Sons.

Survey design:

I Thompson, S. K. 2012. Sampling. 3rd edition. John Wiley & Sons.

Documents

Keep in touch - University of Cape Town€¦ · Observational Experimental # # random sample randomize # # representative equalize groups (average out all other factors) # # make