4.2 ExperimentsObjectives

SWBAT:• DISTINGUISH between an observational study and an experiment.• EXPLAIN the concept of confounding.• IDENTIFY the experimental units, explanatory and response variables,

and treatments in an experiment.• EXPLAIN the purpose of comparison, random assignment, control, and

replication in an experiment.• DESCRIBE a completely randomized design for an experiment.• DESCRIBE the placebo effect and the purpose of blinding in an

experiment.• INTERPRET the meaning of statistically significant in the context of an

experiment.• EXPLAIN the purpose of blocking in an experiment. DESCRIBE a

randomized block design or a matched pairs design for an experiment

• A mother’s smoking during pregnancy and exposure to lead significantly increases her child’s risk for developing attention deficit hyperactivity disorder (ADHD), say researchers. In fact, as many as one third of cases of ADHD in children are linked to exposure to tobacco smoke and lead before birth, giving moms yet another reason to quit smoking during pregnancy.

• For the study, researchers from Cincinnati Children’s Hospital Medical Center surveyed over 4,700 children between the ages of 4 and 15 and their parents. Over 4 percent of the children included had ADHD. The researchers found that those children whose mother smoked during pregnancy were over twice as likely to develop ADHD than a child whose mother had not smoked.

• Based on this study, should we conclude that smoking during pregnancy causes an increase in the likelihood that a child develops ADHD? Explain.

• Is there any way for us to prove that smoking is the cause of ADHD?

• Is it possible that other things could have been occurring that would have lead to ADHD?

• Another item that is associated with poor health is potatoes. Is it possible that some of these mothers, while pregnant, were eating potatoes that were not prepared in the healthiest of ways, and perhaps were served with other unhealthy foods, and maybe the potatoes lead to the ADHD? Or perhaps both potatoes and smoking lead to ADHD?

• Confounding occurs when two variables are associated in such a way that their effects on a response variable cannot be distinguished from each other.

• Can it be distinguished that smoking causes ADHD? Can the potatoes cause ADHD? Can the effects of both smoking and potatoes cause ADHD? Can there be other causes?

• Is there any way to prove that smoking causes ADHD?How can we go about proving it? What if we took a sample of 100 pregnant women. We confined these women to observed compounds and fed them the exact same diets consisting of healthy meals, so we can monitor their intake. We split the group into 50 women that remained smoke-free, and 50 women that were forced to smoke two packs of cigarettes a day. If we did this, could be prove smoking causes ADHD?

• Up until now, we have only examined observational studies.

• We want to differentiate between observational studies and experiments.

An observational study observes individuals and measures variables of interest but does not attempt to influence the responses.

An experiment deliberately imposes some treatment on individuals to measure their responses.

• When our goal is to understand cause and effect, experiments are the only source of fully convincing data.

• The distinction between observational study and experiment is one of the most important in statistics.

What are some differences between an observational study and an experiment?• The goal of an observational study can be to

describe some group or situation, to compare groups, or to examine relationships between variables.– Observational studies are poor ways to gauge the effect

that changes in one variable have on another variable.• The goal of an experiment is to determine whether

the treatment causes a change in the response.– Experiments are the only source of fully convincing data

when we are trying to understand cause and effect.

What's the difference between an explanatory and response variable?• An explanatory variable is a variable that may

help explain or predict changes in a response variable.

• A response variable the variable that measures the outcome of a study.

• In our ADHD example, the explanatory variable is whether or not the mother was smoking, and the response variable is whether or not the child had ADHD.

Prestige Worldwide wants to design an experiment to see if caffeine affects pulse rate.Here is the initial plan:

– measure initial pulse rate– give each student some caffeine– wait for a specified time– measure final pulse rate– compare final and initial rates

What are some problems with this plan? What other variables are most likely to be sources of variability in pulse rates?• Even if pulse rates go up, we couldn’t attribute the increase to

caffeine. Perhaps the excitement of being in an experiment made their pulse rates increase. Perhaps it was the sugar in the beverage and not the caffeine. Perhaps they were hanging out with Carucci and he told them a funny joke during the waiting period and made everyone laugh! Maybe they saw a cat and got excited. In other words, there are many other variables that are potentially confounded with taking caffeine.

To solve these problems:1) Include a that does not receive caffeine so we have something to compare to. Otherwise, any pulse-raising (or lowering) event that occurs during the experiment would be confounded with the caffeine. For example, an amazing stats lecture during the waiting period would CERTAINLY raise pulse rates, making it hard to know how much of the pulse increase was due to caffeine.control groupA control group is an experimental group whose primary purpose is to provide a baseline for comparing the effects of the other treatments.

– For example, in a study to determine the effects of a new pill, a control group would take a placebo.

• A specific condition applied to the individuals in an experiment is called a treatment. If an experiment has several explanatory variables, a treatment is a combination of specific values of these variables.

• The experimental units are the smallest collection of individuals to which treatments are applied. When the units are human beings, they often are called subjects.

• Factor: explanatory variables in an experiment• Level: specific value of an explanatory

variable(s) in an experiment

2) The second step is to make sure that the two groups (caffeine and non-caffeine) are as similar as possible and are treated in exactly the same way, with the exception of the treatments. To make this happen, we use randomization, replication, and control.2a) We subjects to treatments to create groups that are roughly equivalent at the beginning of the experiment. Random assignment ensures that the effects of uncontrolled variables are balanced among the treatment groups. We must ALWAYS randomize since there will always be other variables we cannot control or that we do not consider. Randomizing guards against what we don’t know and prevents people from asking “But what about this variable?”Randomly assign

How do we randomize? What is a completely randomized design?• Randomization can occur using any of the

techniques already examined:– Random number generator– Table of random digits– “Hat trick”

• In a completely randomized design, the treatments are assigned to all the experimental units completely by chance.

2b) means ensuring that there are an adequate number of observations in each treatment group so that the two groups are as equivalent as possible. Then, differences in the effects of the treatments can be distinguished from chance differences between the groups.Replication Note: Replication (repeatability) can also refer to repeating the experiment with different subjects. This can help us feel more confident applying the results of our experiment to a

.comparison

2c) Control means holding other variables constant for each member of both treatment groups. This prevents these other variables from becoming confounded with caffeine and from adding additional variability to the distribution of the response variable.-Control prevents confounding: For example, sugar is an important variable to consider because it may affect pulse rates. If one treatment group was given regular Coke (which has sugar) and the other treatment group was given caffeine free Diet Coke (which has no sugar), then sugar and caffeine would be confounded. If there was a difference in the average change in pulse rates of the two groups after receiving the treatments, we wouldn’t know which variable caused the change, and to what extent. To prevent sugar from becoming confounded with caffeine, we need to make sure that members of both treatment groups get the same amount of sugar.

Control also reduces variability. For example, the amount of soda consumed is important to consider because it may affect pulse rates. If we let subjects in both groups drink any amount of soda they want, the changes in pulse rates will be more variable than if we made sure each subject drank the same amount of soda. This will make it harder to identify the effect of the caffeine (our study will have less power). For example, the first set of dotplots on the next slide show the results of a well-done experiment. The second set of dotplots show the results of an experiment where students were allowed to drink as much (or as little) soda as they pleased. The additional variability in pulse rate changes makes the evidence for caffeine less convincing.

Question: Will an individual’s weight be a confounding variable in this experiment?

No! Treatments were randomly assigned!!!It is also important that all subjects in both groups are blind so that the expectations are the same for the subjects in both groups. Otherwise, members of the caffeine group might suffer from the placebo effect . If the people measuring the response are also blind, the experiment is double-blind.

Note: Not all experiments have a control group or use a placebo as long as there is comparison. For example, if you are testing a new drug, it is usually compared to the currently used drug, not a placebo. Also, you can do an experiment to compare four brands of paint without using a placebo.SUMMARY: With randomization, replication, and control, each treatment group should be nearly identical, and the effects of other variables should be about the same in each group. Now, if changes in the explanatory variable are associated with changes in the response variable, we can attribute the changes to the explanatory variable or the chance variation in the random assignment.

Principles of Experimental Design

The basic principles for designing experiments are as follows:

1. Comparison. Use a design that compares two or more treatments.

2. Random assignment. Use chance to assign experimental units to treatments. Doing so helps create roughly equivalent groups of experimental units by balancing the effects of other variables among the treatment groups.

3. Control. Keep other variables that might affect the response the same for all groups.

4. Replication. Use enough experimental units in each group so that any differences in the effects of the treatments can be distinguished from chance differences between the groups.

Principles of Experimental Design

The results of an experiment are called statistically significant if they are unlikely to occur by random chance. That is, if it is unlikely that the results are due to the possible imbalances created from the random assignment.For example, if caffeine really has no effect on pulse rates, then the average change in pulse rate of the two groups should be exactly the same. However, because the results will vary depending on which subjects are assigned to which group, the average change in the two groups will probably differ slightly. Thus, whenever we do an experiment and find a difference between two groups, we need to determine if this difference could be attributed to the chance variation in random assignment or because there really is a difference in effect of the treatments.

How can we determine if the results of our experiment are statistically significant?Let’s look at the driving example on page 249.

48 playing cards. Remove jokers, the ace of spades, and three 2’s2 through 10 represent drivers that stopJacks, queens, kings, and aces represent drivers that do not stop.Shuffle and deal into two piles, the first represents the cell phone group and the second the passenger group.Record the number of drivers that fail to stop in the cell phone group.How surprising is it to get a result at least as extreme as 12 of 24 drivers simply due to chance? Is the result statistically significant? What conclusion can we draw about whether cell phones are more distracting than passengers?

• Here is a dotplot of 100 simulations. Only once in the 100 simulations did 12 or more drivers in the cell-phone group miss the rest area, by chance.

• Would you say these results are statistically significant?

• Trig seems to think so.

SAT SchoolsMany students enroll in prep courses to improve their SAT scores. Twenty students who have taken the SAT once volunteered to participate in an experiment comparing online and classroom prep courses.1) Describe how we can use a completely randomized

design to compare online and classroom SAT prep courses?

Maybe we decide to take our 20 students, assign them each a number 1-20, and use randIntNoRep. We can take the first 10 unique numbers and have those students do an online prep course and take the remaining 10 numbers and have those students do a classroom prep course.

2) Among the 20 volunteers, 10 are in Precalculus, 6 are in Algebra 2, and 4 are in Geometry. What problem does this cause? How can we address this problem?A student’s current math class might be a source of variability in their improvement. This might make it hard to detect a difference in the effectiveness of the two SAT prep courses. We can address the problem by separating the students into groups (blocks) based on their current math class and randomly divide the members of each block into the online and classroom SAT courses. This randomized block design will help us account for the variation in improvement due to one’s current math class.

Here are the results using math level as a blocking variable.

There certainly appears to be a lot of overlap between classroom and online. This is because we actually ignored the fact that we blocked by math level. We need to use different symbols to tell the difference between the math levels.

Here is the dotplot again, using different symbols for students in each math level. Notice that within each math level, the online students clearly did better. We couldn’t see this difference when we ignored the blocks. The average improvement for students in Precalculus was 86 points, for Algebra 2 it was 40 points, and for Geometry it was 20 points. How can we use this information to account for the variability created by differences in class level?Let’s level the playing field by subtracting 66 points from each student in Precalculus and 20 from each student in Algebra 2. Here are the adjusted results.Now that variability due to current math class has been accounted for, it is much more obvious that the online course is better than the classroom course.

• Blocking in experiments is similar to stratification in sampling.– Blocking always accounts for a source of variability,

just like stratifying. This means that blocking is a good way to increase the strength of your results.

– Blocks should be chosen like strata: the units within the block should be similar, but different than the units in the other blocks. You should only block when you expect that the blocking variable is associated with the response variable.

– Blocks, like strata, are not formed at random!

What are some variables that we can block for in the caffeine experiment? In general, how can we determine which variables might be best for blocking? What about a matched pairs design?• It would be important to take a look at your subjects

for the experiment. Maybe you would want to block by sex, or age range if you have varying age ranges.

• If you think sex might help predict pulse from caffeine, then that would be a good variable to block for.

• The best variables to block for are the ones that best predict the response variable.

• When blocks only include 2 experimental units, it is called matched pairs.

Matched Pairs Design

A common type of randomized block design for comparing two treatments is a matched pairs design. The idea is to create blocks by matching pairs of similar experimental units.

A matched pairs design is a randomized blocked experiment in which each block consists of a matching pair of similar experimental units.

Chance is used to determine which unit in each pair gets each treatment.

Sometimes, a “pair” in a matched-pairs design consists of a single unit that receives both treatments. Since the order of the treatments can influence the response, chance is used to determine with treatment is applied first for each unit.

Matched pairs exampleLet’s say you want to test if a new polyurethane swim suit can make you swim faster than an older suit. A matched pair study would take each individual swimmer in a study and give them both types of suits to test. A completely randomized design might not be good here because there are many factors other than a type of swimsuit that can cause certain individuals to swim faster than others, and those can’t be controlled without using matched pairs.

Microwave PopcornA popcorn lover wants to know if it is better to use the “popcorn button” on her microwave over or use the amount of time recommended on the bag of popcorn. To measure how well each method works, she will count the number of unpopped kernels remaining after popping. She goes to the store and buys 10 bags each of 4 different varieties of microwave popcorn (movie butter, light butter, natural, and kettle corn), for a total of 40 bags.Explain why a randomized block design might be preferable to a completely randomized design for this experiment.A completely randomized design ignores the difference between the four types of popcorn, which will probably result in a great deal of variability in the number of unpopped kernels for both treatments. For example, if there are many more unpopped kernels in bags of light butter popcorn and kettle corn than the other two varieties, it will be harder to tell if there is a difference in the methods of popping. A randomized block design considers each variety of popcorn separately, which allows us to account for the variability in the number of unpopped kernels created by the difference in varieties.

Outline a randomized block design for this experiment.• We will randomly assign 5 bags of each variety to

each of the two treatments. To perform the random assignment, place all 10 bags of a particular variety in a large grocery bag. Shake the bag and then randomly select 5 bags to be popped using the “popcorn button.” The remaining 5 bags will be popped using the instructions on the bags. Repeat this process for the remaining 3 varieties. After popping each of the 40 bags in random order, count the number of unpopped kernels in each bag and compare the results within each variety. Then combine the results from the 4 varieties after accounting for the difference in average response for each variety.