Experimental Design for Biologists Lecture Topicsstaging.postdoc.hms.harvard.edu/slides/ExpDesign-Slides.pdf · Newton: Second edition of Principia: Hypotheses non fingo: “Hypotheses

1/17/13 4:13 PM 1

Experimental Design for Biologists

Lecture Topics

A. A little history & philosophy B. Frameworks C.  System Design; System Analysis

D.  Experimental Controls

E.  Model Building & Verification

2

Experiment 2

Experimental Program: ... the process of doing science

Experiment 2

Experiment 1

Project A

Experiment 1

Project B

Experiment 1

Experiment 2

Experimental Program: Includes Distinct Projects, performed under particular “Frameworks”

Project B

Experiment 1

Experiment 2

FRAMEWORK

Experiment 1

Experiment 2

Project A

Experiment 1

Experiment 2

FRAMEWORK


Project B

Experiment 1

Experiment 2

FRAMEWORK

Experiment 1

Experiment 2

Project A

Experiment 1

Experiment 2

FRAMEWORK Constructed by Philosophy

1/17/13 4:13 PM 6

What is a Framework?

Refers to multiple aspects of the project...

1)  Why is the experiment being considered? 2)  What type of experiments need to be done? 3)  How will the experiment be designed? 4)  How will the data be analyzed? 5)  What can be obtained from an experiment? 6)  How does one use the experimental results?

1/17/13 4:13 PM 7

Beginning of “Modern Science”

Galileo: Emphasis on the Experiment, even though in his case it was mostly ‘thought experiments”, as a way of explaining an idea

However, his process was more in line with Aristotle/Archimedes View of the world, leading to experiment/observation, conclusion

Bacon: Emphasis on the Experiment, as a way of building a model - then

generalization via “induction”.

Suspicion of Intuition, or current views as to how things work.

Legalistic/process driven approach, as opposed to requirement for absolute, mathematical certainty (thus perhaps more appropriate for biology).

1/17/13 4:13 PM Experimental Design

8

What is Inductive Reasoning?

Two definitions of induction: prediction of the future; generalization to a broader case Emphasized by Francis Bacon, in “Novum Organum” (1621) and then again by Isaac Newton in the Principia, and in his Optiks Inductive Reasoning: Application of finding X to “broader” or “general” case - as in the future case.

1/17/13 4:13 PM 9

Francis Bacon on Inductive Reasoning

XIV. The syllogism consists of propositions, propositions consist of words, words are symbols of notions. Therefore if the notions themselves (which is the root of the matter) are confused and over-hastily abstracted from the facts, there can be no firmness in the superstructure. Our only hope therefore lies in a true induction.

XIX. There are and can be only two ways of searching into and discovering truth. The one flies from the senses and particulars to the most general axioms, and from these principles, the truth of which it takes for settled and immovable, proceeds to judgment and to the discovery of middle axioms. And this way is now in fashion. The other derives axioms from the senses and particulars, rising by a gradual and unbroken ascent, so that it arrives at the most general axioms last of all. This is the true way, but as yet untried.

NOTE: Critique of a Hypothesis-based methodology, but in the context of VERIFICATION of an unproven premise, vs evidence-based model-building.

1/17/13 4:13 PM 10

Newton on the Hypothesis

Isaac Newton started with hypotheses, but came to reject them, accepting Bacon’s approach… Newton: Second edition of Principia:

Hypotheses non fingo: “Hypotheses I make not”

Optiks: “Hypotheses have no place in experimental science.” The idea was that one should not establish something as a hypothesis in advance of an experiment, or an explanation; and once you had such evidence, it was no longer “hypothetical”.

1/17/13 4:13 PM 11

David Hume - early 1700s, after Bacon & Newton

“Treatise of Human Nature”: expressed extreme skepticism that the past could be said to rationally predict the future Held that one could only know what was true in observed phenomena, meaning via experience... ...but experience is not applicable to the future, according to Hume. Thus he rejected Inductive Reasoning.

1/17/13 4:13 PM 12

Hume’s Rejection of Inductive Reasoning 1739, David Hume wrote a Treatise of Human Nature, and introduced “radical skepticism” - a rejection of probability derived from past experience as predicting the future. “Your appeal to past experience decides nothing in the present case; and at the utmost can only prove, that that very object, which produced any other, was at that very instant endowed with such a power; but can never prove, that the same power must continue in the same object or collection of sensible qualities; much less, that a like power is always conjoined with like sensible qualities. Should it be said, that we have experience, that the same power continues united with the same object,and that like objects are endowed with like powers, I would renew my question, why from this experience we form any conclusion beyond those past instances, of which we have had experience.” Hume thus rejected “Inductive Reasoning”, that from the experience A causes B, we can infer that in the future A’ will cause B’, saying that the “future A’ ” may not resemble the present A.

1/17/13 4:13 PM 13

Hume’s Rejection of Inductive Reasoning gave rise to Popper’s Hypothesis, which was different (falsification)

Popper proposes “Critical Rationalism” - and suggests the following: 1.  Verification cannot be assumed from past experience. 2.  Falsification only can be demonstrably shown.

Question to think about: If one is truly limited to falsification, how can one rationally build on a past experiment, so as to design a follow-up? (Popper’s answer & Hume’s answer)

1/17/13 4:13 PM 14

The Dominant Framework of Scientific Experimentation involves the “testing of falsifiable hypotheses”

1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism.

1/17/13 4:13 PM 15


1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism. 2. Hypothesis a) statement about the observable universe that is formulated in such a manner that it can be tested.

b) it is a tentative explanation of events or of how something works.

1/17/13 4:13 PM 16

1. Falsifiable Hypothesis - developed by Karl Popper, under a theory called Critical Rationalism. 2. Hypothesis a) statement about the observable universe that is formulated in such a manner that it can be tested.

b) it is a tentative explanation of events or of how something works.

3. Key component - the statement must be falsifiable.

Oft unmentioned requirement of the hypothesis under Critical Rationalism - rejection of verification


1/17/13 4:13 PM 17

Advantages of the Hypothesis, when used in the sense of falsification

1.  Forces restriction of experiments to ideas that can be falsified (establishes what is testable) 2.  Instills a sense of skepticism, which may be helpful in critically analyzing data 3.  Teaches the concept of a “framework” - that the data one produces might be limited in their applicability

1/17/13 4:13 PM 18

Published Criticisms of Critical Rationalism

1.  Critical Rationalism does not avoid inductive reasoning. 2.  Extreme skepticism can be paralyzing, if it does not allow one to say it is rational to look at the past as a predictor of the future. (identity ex) 3.  The past is a rational predictor of the future, since there is nothing about time which would give a mechanism as to why “laws of nature” should start working differently at a particular point. 4. “Falsificationalism” gets you nowhere.

1/17/13 4:13 PM 19

Statistical Hypothesis- testing is distinct from the philosophical use of the term

1.  Role of statistics: to determine if there is a difference between groups of data 2.  Type I statistical error: when a difference is claimed when there is none (false positive) 3.  Type II statistical error: when a difference exists but is not recognized. (false negative) Question to think about: Role of repetition in determining statistical significance, and how repetition functions in different philosphical contexts.

1/17/13 4:13 PM 20

Statistical Hypothesis-Testing

none of the criticisms of Critical Rationalism should be confused with the term “hypothesis” as it is used in the context of Statistics. HOWEVER… it is controversial as to where statistical verification can be said to be predictive (traditional vs Beyesian statistics)

1/17/13 4:13 PM 21

Additional Criticisms of the Hypothesis as an Experimental Framework (filtration criticism)

consider this example...

1/17/13 4:13 PM 22

Hypothesis: The Sky is Red Experimental Design: First Determine the wavelength of light which causes a “red” readout, using a positive control known to be red. Set a wavelength-meter to signal when the color red is detected As a negative control set a second wavelength meter to detect any color other than red. Collect data over 24 hours. See if you got a “positive” readout. Repeat the experiment ten times times. Nine out of ten times, the color red has been positively detected. Conclusion: The sky is red (or the hypothesis is not falsified)

1/17/13 4:13 PM 23

Hypothesis: The Sky is Red Experimental Design: First Determine the wavelength of light which causes a “red” readout, using a positive control known to be red. Straw Man? Does the hypothesis force a reading of any other color except “not red?” Compare to a hypothesis where you don’t have as much inductive information: Activation of NF-kB induces inflammation Ingestion of caffeine induces the likelihood of cancer

1/17/13 4:13 PM 24

Might one conclude from this experiment that the sky is red? Issues: 1)  Need to establish the “usual case” 2)  Need to establish definition of terms

does the hypothesis mean “red appears in the sky?” or does it mean “the sky is usually red?”

3) Scientific disagreements are often about “semantics”

Semantics are important

1/17/13 4:13 PM 25

The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1)  It demanded only one type of measurement

1/17/13 4:13 PM 26

The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1)  It demanded only one type of measurement 2)  It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction

1/17/13 4:13 PM 27

The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1)  It demanded only one type of measurement 2)  It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter)

1/17/13 4:13 PM 28

The Hypothesis “the sky is red” might have “biased” the conclusion in the following ways: 1)  It demanded only one type of measurement 2)  It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period)

1/17/13 4:13 PM 29

The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1)  It demanded only one type of measurement 2)  It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period) 5) Question: Can the conclusion as stated be criticized?

1/17/13 4:13 PM 30

The Hypothesis “the sky is red” might have biased the conclusion in the following ways: 1)  It demanded only one type of measurement 2)  It caused all other color measurements to be grouped as “negative data”, since they did not prove the hypothesis “true” - establishing a binary “positive”/”negative” distinction 3) It allowed for validation(or lack of falsification) even though the result did not represent the “normal” situation (it did not control for rare or “usual” events) (question: does this matter) 4) Sampling times were not altered (it did not control for hidden variables, like color changes within the 24 hour time period) 5) Question: Can the conclusion as stated be criticized? 6) Exercise: compare to a more familiar biological hypothesis, such as “Raf and akt synergize”. Does this hypothesis escape the problems of the “sky is red” hypothesis.

1/17/13 4:13 PM 31

Other issues with framing experiments with hypotheses: 1)  The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven

1/17/13 4:13 PM 32

Other issues with framing experiments with hypotheses: 1)  The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low

1/17/13 4:13 PM 33

Other issues with framing experiments with hypotheses: 1)  The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded

1/17/13 4:13 PM 34

Other issues with framing experiments with hypotheses: 1)  The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded 4) Experiments are aimed at confirmation, because of the requirement to measure the “positive” result - therefore there is a tension between the experiment and the philosophical paradigm framing the experiment

1/17/13 4:13 PM 35

Other issues with framing experiments with hypotheses: 1)  The hypothesis mirrors the conclusion: Hypothesis: The Sky is Red Conclusion: The Sky is Red Hypothesis: Raf and Akt synergize Conclusion: Raf and Akt synergize This formulation can make the experimenter confuse the unproven with the proven 2) The hypothesis by definition is a statement without inductive basis - odds of “correctness” therefore low 3) Falsification is not rewarded 4) Experiments are aimed at confirmation, because of the requirement to measure the “positive” result - therefore there is a tension between the experiment and the philosophical paradigm framing the experiment 5) The negative set may be inappropriately used as a filter that = lack of positive data, as opposed to “not positive”

1/17/13 4:13 PM 36

Settings where the Hypothesis is not practical: “Big Science”

1.  Genome sequencing (what would be a hypothesis seeking to be falsified

for this experiment?) (difference between goals, motivations and hypotheses)

1/17/13 4:13 PM 37




2.  Snp search vs snp test

distinction between “hypothesis producing” vs “hypothesis testing” settings

Point: the “hypothesis producing” experiment is still an experiment, and is still governed by all the necessary requirements of an experiment...

1/17/13 4:13 PM 38




2.  Snp search vs snp test

distinction between “hypothesis producing” vs “hypothesis testing” settings

Point: the “hypothesis producing” experiment is still an experiment, and is still governed by all the necessary requirements of an experiment...

3.  Microarray experiment vs follow-up experiments

First experiment: compare expression differences between conditions x & y (falsification is not the goal for the hypothesis that there are differences between x & y) Follow-up experiments: ask or hypothesize whether a particular difference plays a mechanistic role in the condition y

1/17/13 4:13 PM 39

“Big Science” can be framed by questions

1.  Genome sequencing - what is the sequence of X?

2.  Snp search - what are the sequence differences between individuals of phenotype x and y?

3.  Microarray experiment - what are the expression differences

between tissue treated with x vs not treated? EXERCISE: compare these questions to hypotheses as

frameworks for this type of science - which “works” better?

1/17/13 4:13 PM 40

1.  A process of iterative querying, from a broad question to a series of increasingly narrow questions, or a series of yes/no questions. Example: What color is the sky?

Is the sky red? Is the sky blue? etc

Example: How do you get to Carnegie Hall? Does route X take you to Carnegie Hall? Does route Y take you to Carnegie Hall?

Example: What is the fastest way to Carnegie Hall? Is route X faster than an other route Y, Z, etc?

Example: What are the effects of Raf? What are the effects of Raf on proliferation? Does activation of raf perturb proliferation? How does raf perturb proliferation? Does raf perturb proliferation via Erk1?

The Problem/Question as a Framework for Experimental Projects

41

It can accept the idea of inductive reasoning

Features of The Problem/Question Framework

42

It can accept the idea of inductive reasoning It does not make a distinction between “positive” or “negative” data (but models

are either verified or they are not)


43



It is framed with an open-ended question, which should allow an answer


44




The question is a different structure than the answer


45





Answering the question is rewarded


46





Answering the question is rewarded

It is somewhat more flexible: you can ask questions like “How...?” “What...?” Reframing these as hypotheses may obscure what the scientist wants to achieve from the experiment. It is appropriate for Big Science


1/17/13 4:13 PM 47

1. The question is a different structure than the answer

2.  Can be phrased such that no answer is deemed “negative”.

3.  Can accept the idea of inductive reasoning

4.  Can accept the idea of verification, via model-building and tests of a model (“Is the model true?”)

5. Statistics can be used to determine model verification, based on the answer to the question


1/17/13 4:13 PM Experimental Design Harvard Medical School

48

Project Schematic in The Problem/Question Framework

Model

Query Model Verification?

Question Answer step 1

Question Answer step 2; repeat

step 3

Perturbed Model

1/17/13 4:13 PM Experimental Design Harvard Medical School

49

Project Schematic in Hypothesis-Testing Framework

Conclusion

Test Conclusion Lack of Falsification (is model building valid without inductive reasoning?)

Hypothesis Lack of Falsification step 1

Hypothesis Lack of Falsification

step 2; repeat

step 3

What type of conclusion is appropriate under Critical Rationalism? Only lack of falsification.

Model


Project B

Experiment 1

Experiment 2

FRAMEWORK

Experiment 1

Experiment 2

Project A

Experiment 1

Experiment 2



Project B

Experiment 2

FRAMEWORK

Experiment 1

Experiment 2

Project A

Experiment 2


Sy

Experiment 1

System

Experiment 1

System

1/17/13 4:13 PM Experimental Design Harvard Medical

School

52

System Issues Capability - Can you measure/do the thing you want to

measure/do? – answered by these other properties:

Sensitivity - How much of the thing can you measure?

To what degree of difference can you measure it? (bp ex)

Specificity/Selectivity - Can you measure/do the thing you

want such that you can discriminate between X & Y, where X is the thing you want to measure & Y are other things which might interfere with you measuring X; X & Y are the signal to noise represented by the positive & negative controls, respectively

Fidelity/Stability - Can you measure the thing consistently?

Do you get the same answer under the same conditions.

Two an'bodies for detec'ng MuSK from Santa Cruz, inc.

An'body A An'body B

Experimental Design Harvard Medical School

Two an'bodies for detec'ng MuSK from Santa Cruz, inc.

An'body A An'body B

Experimental Design Harvard Medical School


School

55

System Issues for an antibody Capability - Can you measure/do the thing you want to

measure/do? – answered by these other properties: (also issues like IP vs IB)

Sensitivity - How much of the thing can you measure?

To what degree of difference can you measure it? Dose down lysate or peptide

Specificity/Selectivity - Can you measure/do the thing you

want such that you can discriminate between X & Y, where X is the thing you want to measure & Y are other things which might interfere with you measuring X; X & Y are the signal to noise represented by the positive & negative controls, respectively; Peptide alone vs lysate; KO)

Fidelity/Stability - Can you measure the thing consistently?

Do you get the same answer under the same conditions. (Stability of ab; stab of lysate)


School

56

Examples of Systems that require validation Animal models

Genetic model Treatment model Pharmacologic model Translatability? Genetic variation? Discrete markers?

Cellular models

Genetic perturbation Treatment model Pharmacologic model Translatability to the normal cell? Inductive ability?

Biochemical/Test-tube experiments

(PCR, Westerns, Southerns etc) Conditions used Reagents


School

57

The use of the Positive Control establishes that the system is working.

The positive control in an experiment is a system control.

System validation is the process of understanding the

parameters of the positive control in the context of your model.

Additional issues in system validation depend on why you’re

using the system; for example, the issue of the generalizability of your results from your system to another

needs to be validated..


School

58

System Issues (capability, sensitivity, specificity) Need to be Validated with ALL CONTRLS Used in the Experiment

Negative Control - e.g. No antibody; irrelevant antibody Positive Control – e.g. Different, previously validated

antibody; recombinant protein to demonstrate antibody works

Sensitivity Controls – discussed further today Specificity Controls – “ Selectivity Controls - “ Thus there is a circularity to System Validation – the Positive control in an experiment is a way of asking “did the system work” each time the experiment is done.


School

59

Experimental Design for Biologists - Establishing the System

The Framework Question or Hypothesis Governs the Choice of System 1) The framework operates most effectively if it can govern the entire project - not just a single experiment

Once you establish what you want to answer, you can figure out what tools you need to answer that

question.

Note - some people pick the system first: Then you are limited to particular questions/hypotheses

appropriate to that system.


School

60

Experimental Design for Biologists - Establishing the system by accessing the “inductive space”

The choice of the system is often governed by the “inductive space” established by the framework question or hypothesis 1) The framework operates most effectively if it can govern the entire project - not just a single experiment

2) The framework helps guide the scientist as to the knowledge which might impinge on the experiment - this is referred here as “accessing the inductive space”

The “Inductive Space” is the knowledge gained from past experience, which the scientist might profit from applying to the current issue.

Challenge of knowing what to access: 1.  What is relevant about the thing you are studying?

For a protein it may be a domain, or an activity, or the cellular localization, or some other thing that it perturbs, which is your readout

2.  This is another reason why the framework issues are so important

- what are you actually interested in finding out - how protein X perturbs Y, or what are the things which cause Y to do Z? (you need to make sure you are studying the right thing)


School

62

Establishing the system

MISTAKE: MOST PEOPLE JUMP TO THIS:

Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure. (Exercise: What is the framework question?)


School

63


Correct way: Understand your framework question, so you know full range of what you’ll need to measure. Then prove you can measure this thing, and in what context (what

animal, under what conditions): that is your “System”

Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) (Broader framework: “what is the effect of caffeine?” would alert thescientist to the possibility of other effects that might perturb the experimental issue, even if not measured in a particular experiment - “independent variables”, side effects) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.


School

64

Framework: What is the effect of caffeine on blood pressure? (have to measure both an increase or a decrease, or a lack of change, vs only an increase) Experimental question: Does caffeine cause an increase in blood pressure? Experimental hypothesis: Caffeine causes an increase in blood pressure.

Systems: Measuring changes in blood pressure Human, Dog, Rodent? How are you going to perturb blood pressure? (pharmacologic agent?

how are you going to validate that drug?) How are you going to measure/establish what the blood pressure is?



School

65


Systems: Measuring changes in blood pressure

A) Equipment



School

66


System: Measuring changes in blood pressure

A) Equipment B) Equipment verification



School

67



A) Equipment B) Equipment verification C) Perturbation

Establishing the framework establishes the system


School

68



A) Equipment B) Equipment verification C) Perturbation D) Perturbation verification



School

69



A) Equipment B) Equipment verification C) Perturbation D) Perturbation verification

The way one addresses these issues is with “system controls”



School

70

Question: Does caffeine cause an increase in blood pressure? Equipment Verification Controls

A) Validation vs a standard example: blood pressure monitor: compare to “validated” standard 2nd example: pH meter: use solution of “known” pH 3rd example: antibody: test on peptide used to raise ab

B) Validation by different pieces of equipment

Do two different blood pressure monitors agree? Do two antibodies raised against the same peptide agree?

C) Reproducibility of measurement

Does the equipment give the same answer when the same condition is surveyed multiple times? What is the degree of difference?

D) Different type of equipment or different method of addressing the issue

E) Subject population can experience changes in blood pressure (what if they’re all on medication?)

Establishing the system - system controls


School

71

Does caffeine cause an increase in blood pressure? Perturbation Verification Controls

A) Positive Control Reagent known to cause an increase in blood pressure

B) Negative control

Reagent known to have no effect on blood pressure

C) “All but X” negative control

Decaffeinated coffee vs coffee

C) Equipment control - sensitivity of changes

What is the degree of change that can be assessed in a statistically significant manner?

Establishing the system - system controls


School

72

Additional system controls - eliminating independent variables

A)  Limiting Additional Variables that could effect the measurement

example: “What is the effect of caffeine on blood pressure?”

control for: hypertension, other sources of caffeine besides those in the experiment, other chemical perturbations of blood pressure, stress

(What is the effect of caffeine on blood pressure, in the absence of other known pressors?) (Is that what you meant to ask?)

B)  What is the effect of Insulin on Akt phoshorylation?

control for: other stimulators of Akt (starve down the cells)

(What is the effect of Insulin in comparison to no growth factors stimulation on Akt phosphorylation) (Is that what you meant to ask?)


School

73

Definitions; it is all about language

Definitions are a “big” issue...

What do you call a significant increase in blood pressure?

What do you chacterize as Akt phosphorylation?

Issue of positive controls to set ranges.

Issue of physiologic responses to establish “relevant changes”


School

74

Establishing the system in advance of the experiment definitions, time courses and

experimental repeats Does caffeine cause an increase in blood pressure?

A)  Definition:

i.What is meant by “caffeine”? (does coffee = caffeine? if not, then one would need to control for the non-caffeine elements of coffe)

ii. What is meant by an “increase” in blood pressure? (degree of difference accepted) iii. What is meant by “blood pressure”? (systolic or diastolic or both)

EXERCISE: note how different an experimental outcome might be, depending on these definitions


School

75

Establishing the system in advance of the experiment definitions, time courses and experimental repeats

Does caffeine cause an increase in blood pressure?

B) Time Course

i. Establishes the “usual case”

(example: What color is the sky? What is the effect of caffeine on blood pressure? What is the effect of IGF1 on Akt signaling?

ii. Increases the likelihood of finding more than one answer to the question.

iii. Causes built-in repeats of measurements, but does not count as a “repeat”


School

76

Does caffeine cause an increase in blood pressure?

B) Experimental repeats

i. Establishes the stability of the system

ii. Allows for verification or falsification of models

iii. Increases the likelihood to find multiple answers to a question?

(example: What time does the sun rise? What is the effect of caffeine on blood pressure? What is the effect of IGF1 on Akt signaling?

ii. Increases the likelihood of finding more than one answer to the question.

iii. Counts as a repeat!

Establishing the system in advance of the experiment definitions, time courses and experimental repeats


School

77

Criteria for Validation - Being able to predict the future

Basis for all clinical testing - “success paramaters” in advance of trials. Meeting success criteria.

P value; probability of random occurence - verification via repetition

Statistics of causal inference

Establishing the system in advance of the experiment


School

78

Criteria for Validation - Examples Relevance of Yeast to Humans? (particular examples) Relevance of tissue culture to perturbations in organisms Relevance of subpopulation of humans to broad population of humans

(gender, race)

Validating the System as Being Relevant to Your Issue


School

79

Framework: What genes mediate aging? What genes when perturbed, can modulate aging?

Definitional issues: Aging? Positive or negative perturbation?

System choice issues:

1) Short lived organism - bacteria vs yeast vs worms vs mice?

2) Can you do genetics? Can you measure aging? 3) Markers vs phenotypes

4) Validation in other organisms

Examples of going from Framework Question to System, setting up the Experiment


School

80

Framework: How is the cell cycle regulated?

Definitional issues: Entire cycle? Particular parts? Positive or negative perturbation? Type of cell?

New Framework: How is G0 regulated in NIH 3T3 cells?

System choice issues:

1) Tissue culture vs general case

2) Genetics & biochemistry? 3) Markers vs phenotypes

4) Validation in whole organisms



School

81

Framework: Does IGF-1 cause Akt phosphorylation?

Definitional issues: What Akt site? For how long? How much IGF-1? Under what what conditions (Starved? differentiated cells?)

System issues:

1) Tissue culture ? What cell? Rationale for that choice?

2) What antibody? Capability vs Specificity vs Sensitivity Negative & Positive controls for each

3) How do you quantify?

4) Akt phos/Akt total

5) Akt1 vs Akt2 vs Akt3



School

82

Framework: How does PTEN knockdown cause glioblastoma? What are the genetic changes induced by PTEN knockdown in glial cells, and which are sufficient to induce glioblastoma?

Definitional issues: Degree of genetic changes? Single vs multiple? Time of analysis?

System issues:

1) Tissue culture vs brain? 2) How to knockdown the PTEN? 3) Measuring RNA differences 4) Validating RNA differences 5) Setting up the follow-up systems…



School

83

Rethinking System Validation – Understanding each perturbation relevant for your hypothesis/question

1. Equipment Validation - Tubes, Heaters, others 2. Reagent Validation - Enzymes, Antibodies,

Mice (knockout vs wt) (Strain Issue) 3. Approach Validation - Biochemical vs Genetic 4. Reductionism Validation - Test tube vs Animal Model vs Human

Expansive View of Systems – Each perturbation


School

84

Boot Camp

Introduction to Experimental Design

Class # 3 – Negative and Positive Controls


School

85

Designing the Experiment - Negative & Positive Controls

Experimental Program: Includes Distinct Experiments, performed using particular “Controls”

Project B

Experiment 2

FRAMEWORK

Experiment 1

Experiment 2

Project A

Experiment 2

FRAMEWORK

Sy

Experiment 1

System

Experiment 1

System


School

87

The Experiment - The Measurement (establishing the measurement is part of establishing the system)

What is the effect of X on Y?

Need to be able to measure “Y” and “not Y”

Need to establish that effect X actually happened.

Therefore if an experiment is a measurement of changes in Y caused by X, you need to be able to prove that:

1) the change in Y can be measured 2) that X happened 3) that any measured change is actually do to X


School

88

The Experiment - The Definition of the Negative Control

The negative control can be defined as “The unperturbed by X” control, where X is the thing who’s effect you seek to measure.

The negative control demonstrates that in the experiment “What is effect of X on Y” you have sufficient data to measure the unique effect of “X”. Thus the negative controls are the measurements you need to prove you can measure “X” vs the unperturbed case, AND against everything else perturbed in your system besides “X”; thus the negative control is more than simply “unperturbed”; it’s controlling for everything perturbed in your system perturbed in addition to X, where X is the particular thing you’re interested in.

The Measurement in the setting of an experiment usually means a comparison: seeing a difference between a perturbation and a series of “negative controls.”


School

89

The Experiment - The Definition of the Positive Control

The positive control is an internal system validation control.

The positive control demonstrates that in the experiment framed by “What is effect of X on Y?” you have sufficient data to determine that a measurement of “Y” can be determined, and that X is operational.

The positive control is an internal system control – it proves the system is operation in that experiment.


School

90

The combination of negative and positive controls allows these determinations:

1.  Sensitivity - If X activates Y to degree n, can the system measure degree n?

2.  potency - If X activates Y, how much X is necessary to see Y?

3.  Degree pf activation - For how long does Y stay activated?

4.  relative degree - if X activates Y , but much less well than the positive control , does that alter the interpretation?

5.  type - Is there more than one way to activate Y?

6.  context - Does X activate Y in a particular cell? Does X activate Y only under a particular condition?


School

91

Issues in Measurement - Negative and Positive Controls need to be in effect for these issues

1.  Sensitivity - Signal to Noise; can you measure an effect over the baseline signal

2.  Potency - Can you determine the maximum effect?

3.  Duration - Does the negative stay negative over this period of time? Is the system operational for the required duration?

4.  Context - You need distinct negative and positive controls for different ways to activate Y, and for X existing

in different contexts (e.g. cell types)

1/17/13 Experimental Design Harvard Medical

School

92

Negative and Positive Controls for this vector, assuing You are running v-Ras off the CMV?


School

93

The Experiment- The Negative Control; not just the unperturbed case, but also the “all but X” control

The negative control as more than the unperturbed case

Unperturbed by What?

The scientist must isolate distinct relevant parts of the Perturbation. You want to determine the effect of v-Ras transformation on NIH3T3 cells

Plasmid expressing CMV-v-Ras-IRES-EGFP + Neo run off a distinct promoter

Therefore you need to keep doing additional negative controls until you have tested all the variables except “X”, where “X” is the thing you are querying in your experiment.


School

94

The Experiment- The Positive Control; the internal system control

Experiment 1)  WT mouse on a running wheel (negative control for deleting X) 2)  X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes

to understand mechanism. how do you know changes are caused by deletion of X vs running vs

both? 3)  X-/- mouse without running (negative control for running) 4)  WT mouse without running

how do you know the system is working? that the animal is capable of being perturbed? this is why you need positive controls

5) WT mouse + Known Exercise Enhancer (Testosterone or

Clenbuterol) on a running wheel (need to establish this FIRST)


School

95

The Experiment- The Negative Control; from the unperturbed case To the “all but X” case

The negative control as the all but X control

The scientist must isolate distinct relevant parts of the Perturbation. You want to determine the effect of v-Ras transformation on NIH3T3 cells

Plasmid expressing CMV-v-Ras-IRES-EGFP + Neo run off a distinct promoter

Negative controls?: In Red (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV c-RAS-IRES EGFP CMV-v-RAS-IRES EGFP


School

96

The Experiment- The Negative Control; The All-but-X Controls

Negative controls? (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV-c-RAS-IRES EGFP Measurement negative controls: Depends on what you measure (for

example, do you have to starve the cells to get a “negative” for a particular readout?)

In other words: you need to have a setting where the cells would be both ‘usually not transformed’ but also ‘transformed if there is an

oncogene’ ... thus the question or hypothesis again governs what your control is and what you measure


School

97

The Experiment- The Negative Control: refers to every level of the experiment

There are Negative Controls at multiple levels of the experiment The plasmid controls in the previous example are negative controls for the transfection. But if then one wants to measure something, there may be negative

controls to other aspects of the experiment:. Cell types (-) Neo plasmid + Neomycin Empty plasmid CMV IRES EGFP plasmid CMV-c-RAS-IRES EGFP Transformation paradigm: +/- p53 deletion; Metastasis paradigm: +/- VEGF inhibition, etc...


School

98

The Experiment- Figuring out what your negative and positive controls are depends on your question or hypothesis, and what system you are using

to address that question or hypothesis...

Consider this question: What is the effect of deleting gene X on exercise capacity?

Consider this hypothesis: Deletion of gene X is sufficient to increase exercise capacity.

System: wt vs X-/- mouse Perturbation: running wheel You want to measure the amount the animal runs, and determine how

deletion of gene X effects running Readouts: 1)  Amount the animals run 2)  Changes in gene expression caused by deletion of gene X 3)  Biochemical changes caused by deletion of gene X 4)  Expression and biochemical changes caused by gene X in the

context of running


School

99

The Experiment- The Negative Control; the unperturbed by X



System: Free running wheel Experiment 1)  WT mouse on a running wheel (negative control for deleting X) 2)  X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes




School

100

The Experiment- The Negative Control; the unperturbed by X



System: Free running wheel Experiment 1)  WT mouse on a running wheel (negative control for deleting X) 2)  X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes


both? 3)  X-/- mouse without running (negative control for running) 4)  WT mouse without running how do you know changes are caused by deletion of X in a specific

setting? 5) conditional knockouts – tissue specific; time/condition specific


School

101


Experiment 1)  WT mouse on a running wheel (negative control for deleting X) 2)  X-/- mouse on a running wheel measure exercise capacity and coincident gene and biochemical changes



how do you know the system is working? that the animal is capable of being perturbed? this is why you need positive controls

5) WT mouse + Known Exercise Enhancer (Testosterone or

Clenbuterol) on a running wheel (need to establish this FIRST)


School

102


WT mouse + Known Exercise Enhancer (Testosterone or Clenbuterol) on

a running wheel (need to establish this FIRST)

It’s only by mapping out this experiment that you see the need to first determine the conditions by which a “known” enhancer of exerciser functions. There are risks to this: you need to make sure your positive control is truly validated... the choice of a positive control and how it is used is critical to the success or failure of an experiment. Using a positive control, you learn how long the experiment needs to run,

when you need to make measurements... what degree of perturbation(s) are necessary...


School

103

The Experiment- The Negative Control for a drug trial in the setting of a Genetic mutation

The negative control as the unperturbed case

Unperturbed by What?

The scientist must isolate distinct relevant parts of the Perturbation. Consider this project: Give mice a particular drug - test resistance to a

particular tumor Question: Does tamoxifen decrease tumor load caused by BRCA1 (+)

breast cancer Hypothesis: Tamoxifen decreases tumor load caused by BRCA1 (+)

breast cancer (can you suggest the unperturbed by X controls for this experiment?)


School

104

The Experiment- The Negative Control; the unperturbed case

Consider this project: Give mice a particular drug - test resistance to a

particular tumor Question: Does tamoxifen decrease tumor load caused by BRCA1 (+)

breast cancer Hypothesis: Tamoxifen decreases tumor load caused by BRCA1 (+)

breast cancer System: need to induce tumor with BRCA1 (+). Need to see tumor

progression over some time course. A positive control for inhibition would be quite helpful (?deletion of BRCA1?)

Negative controls:


School

105

The Experiment- The Negative Control; the unperturbed case

Consider this project: Give mice a particular drug - test efficacy on a

particular tumor Question: Does tamoxifen increase survival in mice with a BRCA1 (+)

breast cancer Hypothesis: Tamoxifen increases survival in mice with a BRCA1 (+)

breast cancer System: need to induce tumor with BRCA1 (+). Need to see tumor

progression over some time course. A positive control for BRCA1 requirement would be quite helpful (deletion of BRCA1?); need to have markers for cell death in the tumor ( why?)


School

106

The Experiment- The Negative Control: the all but X controls The Positive Control: the internal system control

Hypothesis: Tamoxifen increases survival in mice with BRCA1 (+) breast

cancer Readout: survival; markers of cell death in the tumor; gene/biochemical

changes induced by tumor vs those perturbed by tamoxifen Experiment Mice (number used determined by prior experiments – need enough to

give statistical significance for tamoxifen effect) MIce, unperturbed (negative control) MIce+ BRCA1 (+) tumor MIce+ Tamoxifen (negative control for tumor) MIce+ BRCA1 tumor + Tamoxifen (test case) positive controls:


School

107


Mouse, unperturbed Mouse + BRCA1 Mouse + Tamoxifen Mouse + BRCA1 + Tamoxifen are you done? How do you know you gave enough tamoxifen? System design! System controls for sufficient tamoxifen.. You need to know how tamoxifen works, and therefore have a “marker” of

sufficient tamoxifen treatment - you need to give enough tamoxifen so as to perturb that marker (as a positive control to prove the tamoxifen is working)


School

108


Mouse, unperturbed Mouse + BRCA1 Mouse + Tamoxifen dose 100X Mouse + BRCA1 + Tamoxifen dose X etc (using markers to prove tamoxifen is operative, you establish concentration X... sufficient to

perturb the markers) Mouse + BRCA1 + Tamoxifen dose 2X Mouse + BRCA1 + Tamoxifen dose 10X Mouse + BRCA1 + Tamoxifen dose 100X (why do you do the dose response? establishes the range of efficacy for tamoxifen... the generalizability of claims made as to utility of

the drug)


School

109

Next example - caffeine

The negative control as the unperturbed case

What is the effect of caffeine on blood pressure? Experiment: Give 50 people caffeine; measure their blood pressure

at different time points...

This measurement is meaningless unless there is a point of comparison.

That point is the negative control, the unperturbed case.

Experiment: Compare 50 people on nothing to

50 people on caffeine vs One group of 50 people, make sure they don’t have caffeine

for 24 hours, measure their blood pressure, Then give them caffeine, then measure their blood pressure... (the same group can be their own negative control)

The Negative control first establishes a point of reference.


School

110


Negative control as all-but-X controls:

What is the effect of caffeine on blood pressure? Experiment:

One group of 50 people, make sure they don’t have caffeine for 24 hours, measure their blood pressure, Then give them caffeine, then measure their blood pressure...

objection: is the unperturbed case enough?


School

111

2. The negative control as the “all-but X” treatment

What is the effect of caffeine on blood pressure?

A) Give 50 people caffeine in pill form B) Give 50 different people pills that are identical to A) except without caffeine (controls for the caffeine) C) Give 50 different people nothing (controls for the pill) - also controls for the placebo effect in B… determines if there is a placebo effect D) Give 50 people known perturber of blood pressure (positive control)

(alternate design; this design controls for differences between people) A) Give 50 people nothing (establish their baseline) B) Give same 50 people placebo (negative control for caffeine) C) Give same 50 people caffeine (how much? need positive control for caffeine – some marker demonstrating it is in the system) D) Give same 50 people known perturber of blood pressure (positive control for being able to alter blood pressure)

The Experiment- Caffeine experiment The Negative Control; “all but X”; the positive control as an internal system control


School

112

Control for surrogates of your desired perturbation

Experimental question: What is the effect of caffeine on blood pressure? Experimental hypothesis: Caffeine injestion causes an increase in blood

pressure Experiment restatement, to allow for the use of coffee as caffeine treatment:

Does caffeinated coffee affect blood pressure, and if so, is the caffeine responsible?

A.  No treatment (establishes baseline) B. Water, matched to the amount of coffee - (control for volume) C.  Decaffeinated coffee – (all but X control for caffeine_ D. Caffeinated coffee - treatment E. Furosemide – positive control for altering blood pressure

* Eliminate other known blood pressure stimulants or depressants. (to establish baseline... unperturbed by other blood pressure modulators)


School

113

Example – genetic pertubation

Experimental question: What is the effect of Ras on proliferation? Experimental hypothesis: Activation of Ras induces cell proliferation. Experiment: Transfect cells with plasmid expressing tamoxifen-inducible

c.a.Ras-ERt2, determine if activation perturbs cell proliferation rates.

Definitions, System controls need to be established...(measuring

transformation) Experiment A.  Untransfected cells B. Untransfected cells (+) tamoxifen C. Cells transfected with plasmid containing c.a.Ras-ERt2, (-) tamoxifen D. Cells transfected with plasmid containing c.a.Ras-ERt2, (+) tamoxifen E. Cells transfected with plasmid alone, (-) tamoxifen F.  Cells transfected with plasmid alone, (+) tamoxifen G.  Cells transfected with gene known to perturb cell proliferation


School

114

Additional Controls

Negative Controls: All-but-X controls Positive Controls: Internal System Controls Experimentalist Controls: Part of the system; change who does the

experiment

Blinded analysis!

Pre-existing criteria for evaluation Method Controls: Do the experiment a different way Meta-system controls: Use an entirely different system to address the

framework question Assumption controls: If X leads to Y, and then implies Z, test for Z

Question 1: Is IGF-IR and different isoforms of IRS proteins degraded and ubiquitinated in response to IGF-I, DEX and a cytokine cocktail treatment?

cells + IGF-I

immunoprecipitate IGF-IR, do western blot to check whether IGF-IR protein level decreased (1) and ubiquitin content increased (2)

Repeat the treatment along with proteosomal inhibitor MG132 or lactacystin If (1) yes

and (2) no

Repeat the treatment with cells transiently transfected with Myc-Ub cDNA

IGF-IR is not ubiquitinated and degraded by proteosome after IGF-I treatment

do time- couse assay and use different dosage of IGF-I

both are no both are yes

If (1) yes and (2) no

If (1) yes

(2) no

IGF-IR is ubiquitinated and degraded after IGF-I treatment

IGF-IR is not degraded after IGF-I treatment in muscle cells

Add lysosomal inhibitor(chloroquine) with IGF-I. If IGF-IR protein level remains constant, then IGF-IR is degraded by lysosome.

Repeat the flow chart for + Dex and other atrophy stimulus such as + TNFα, IL-1, inteferon- γ cocktail;Repeat the flow chart for Insulin Receptor and the four isoforms of IRS

Question 2: Which ligases are responsible for the ubiquitination of IGF-IR and IRS cells + IGF-I

+ siRNA to Nedd4 Immunoprecipitate IGF-IR and check whether IGF-IR ubiquitin content decreased

Nedd4 can ubiquitinate IGF-IR

If yes

+ IGF-I + siRNA to Mdm2

Mdm2 can ubiquitinate IGF-IR

If yes

+ IGF-I + siRNA to SOCS1, 2, 3, 4, 5, 6, 7, CIS

SOCS can ubiquitinate IRS

+ IGF-I + siRNA to Mdm2

Mdm2 can ubiquitinate IRS

+ IGF-I + siRNA to c-Cbl and Cbl-b

Cbl can ubiquitinate IRS

Repeat the flow chart for + DEX and + TNFα, IL-1, inteferon- γ cocktail If all are no Transfect cells with siRNA library towards

ubiquitin ligase from Dharmacon Find new E3 ligases responsible for the ubiquitination of IGF-IR and IRS in response to different signals

Incubate cells with crosslinker Immunoprecipitate IRS and do mass-spectrometry to find new IRS interacting proteins

Immunoprecipitate different isoforms of IRS and immunoblot wit hanti-ubiquitin antibody to check whether their ubiquitin content decreased

If yes

If yes

If yes

Fractionate cell homogenate and test the ligase activity in every fraction

Summary 1. Framework. What is the question that you want to answer? 2. Inductive Space. Do you know what you need to know about the subject? Decide what aspects of prior

knowledge relate to your question. Read the literature. 3. System. What tools are you going to use to answer your question? 4. System controls. How do you know your system works? How do you know your system can provide the type

of data you require? Is the system best matched to your question, or might a different system be better? 5. Experiment. What are you going to do to answer the question? Make sure measurements are made multiple

times. Make sure you are measuring the effect in a representative fashion. Study the effect over time, and over a range of the experimental agent. Consult a statistician and discuss the mode of analysis for your data, and how many data points are required.

6. Establish Criteria for the effect, in advance of the experiment. 7. Negative Controls. What negative controls are required? Is there an “all but X” control available? 8. System Positive Controls. How do you know that the system is still operational? What positive controls are

necessary to prove that the thing you want to measure was actually measurable within the context of the experiment?

9. Effect Positive Controls. How do you know that the effect you want to measure can be produced in your system? What positive controls are necessary to produce the effect?

10. Assumption Controls. If X is being measured, can something else be measured that “should” occur when X occurs? If you think Y has happened as a result of X, is there something else you can measure which should also happen when Y happens?

11. Do the experiment 12. Experimentalist Controls. Analyze the data in a blinded fashion 13. Repetition. Repeat the experiment using the same criteria and methodology 14. Model building. What is the answer to the question? 15. Model check. Is the answer responsive to the question? 16. Does the model predict what will happen again? Repeat the experiment. 17. Extension. Does the model hold in different circumstances? 18. Change the system. Approach the question in another way. 19. Change the scientist. See if others can reproduce the effect.

Documents

Experimental Design for Biologists Lecture Topicsstaging.postdoc.hms.harvard.edu/slides/ExpDesign-Slides.pdf · Newton: Second edition of Principia: Hypotheses non fingo: “Hypotheses