Experimental Design. Anecdotal Evidence (and worse!)… What’s wrong with this? Anecdotal...

Experimental Design

Anecdotal Evidence (and worse!)…

What’s wrong with this?What’s wrong with this?

Anecdotal Evidence: based on haphazardly selected individual cases. No attention given to outliers, lurking variables etc. Works from a fallacy of making an un-warranted generalization from a small group of individuals to a larger population. One of the most common “misconceptions” about statistical evidence.

Observation vs Experimentation

An observational study looks for patterns, correlations etc in a population without interfering with the population.

An experimental study deliberately introduces a treatment to elicit a response and to observe this in a population.

Group Discussion…

Let’s look at 3 questions from your text and discuss them.

Spend 3 minutes discussing each of these in turn (we will discuss as a whole between each one)

Look for problems with the statistical claims or analyze what kind of statistical argument is being made in each case.

The questions: 3.1, 3.5, 3.8

Placebo Effect…

Doctor: I’ve got to give you a shot. Do you have private health insurance?Patient: No.Doctor: Nurse, fetch the placebo.

Doc Martin’s Amazing Brain Pills…

Suppose you were foolish enough to take on of doc Martins amazing brain pills before a stats exam – and you ACED it!

You said it was the pills but shortly after doc M was arrested it was discovered that the pills were just Smarties!

How do you explain your performance?

The Placebo Effect (and variants)

Often the very act of carrying out statistical trial (interview, treatment etc) affects the results

A placebo is a “fake” or facsimile treatment given in place of the real treatment

Good experimental design…

Some terms to learn… Subjects/units are the individuals being studied Treatment is a specific experimental condition being

administered Factors are the explanatory variables in the study Comparative experiment follows a simple

Treatment Observe response structure Bias is a systematic trend to favour a specific outcome Control group is a group that receive a placebo treatment

Randomized Experiment

Control

Principles of Good Experimental Design…

Control lurking variables – compare two or more treatments

Randomize choice between subject/unit and treatment received

Replicate on as many subjects/units as possible

Trying to eliminate bias – Double Blind Experiments… Researcher bias can creep into an experiment in subtle

ways Suppose you wanted to test a potentially life saving drug

on a group of very sick children. You were so convinced about the drug’s effectiveness that you really didn’t want to give a placebo to some so…

When you gave the real drug you were up-beat and encouraging but when giving the placebo you were much more reserved…

You get the idea! To avoid this a double-blind experiment would randomize the patients and the drugs so that neither you or the patient would not know ahead of time which drug you were giving.

Comparison and looking for Statistical Significance

The purpose of setting up controlled experiments with controls is to test for effected change

A result is considered statistically significant if the observed effect is so large that it is very unlikely to have occurred by chance.

To be continued…

Block Design Experiments…

A block design experiment divides the subjects/units into identifiable groups prior to random sorting.

For example, you may wish to divide the group by gender, or religion or political affiliation

Sample Block Design: 3 treatments/2 groups

Group Work…

Discuss the following questions from your text: 3.10, 3.12, 3.21

If helpful, draw a block diagram Table B refers to the table of random digits at

the back. To see how to use this consult example 3.7 on page 235

Sampling and Experiment Design

A call-in late night radio show runs a caller poll concerning funding for private, religious colleges and universities. Callers were asked to vote Yes – they should be funded or No – they should not be funded by the government. The results were 23% Yes and 77% No.

Should policy be based on this survey?

Simple Random Sample

If any one subject or unit in a population is as likely as another to be sampled then random selection from this set produces a simple random sample.

Question: does the random sample represent the entire population?

Go to excel example…

Stratified Random Samples…

Sometimes (often) a population consists of numerous identifiable sub-groups or strata.

A random sample should include selections from each of the strata.

This is called a stratified random sample.

In conclusion…

Make sure you familiarize yourself with the many terms and definitions introduced in 3.1-3.3

Understand what anecdotal evidence is and why it cannot be used as a strong statistical argument

Understand what a double blind experiment is

Sample questions: 3.33, 3.34,3.39, 3.47