Lecture 4: Data collection, observational studies and … · Lecture 4: Data collection, observational studies and experiments Statistics 101 Mine C˘etinkaya-Rundel September 8,

Lecture 4: Data collection, observational studies andexperiments

Statistics 101

Mine Cetinkaya-Rundel

September 8, 2011

Recap

More on mosaic plots

Below is a mosaic plot displaying the relationship between car andbike ownership.

Statistics 101 (Mine Cetinkaya-Rundel) L4: Data coll., obs. studies and experiments September 8, 2011 1 / 45

Recap

Anatomy of a mosaic plot

BikeNo Yes Total

No 29 4 33Car

Yes 11 2 13Total 40 6 46


Recap

Another mosaic plot

Newsmap (http:// newsmap.jp/ ): stories are arranged by topic area(news, sports, etc.) and sized by the number of mentions.


http://newsmap.jp/

Recap

Clicker question (graded)

Which of the following statements is (are) correct?

I. Majority of the students in this class are females.

II. Males are more likely than females to be confident about changing acar tire on their own.

III. Males and females are equally likely to be confident about changing acar tire on their own.

IV. Majority of the students in this class are confident that they can changea tire on their own.

V. These data suggest that gender and confidence about being able tochange a car tire alone may be independent.

(a) I and II(b) Only III

(c) I and IV(d) I, III, V

(e) II, IV, V


Recap


Overview of data collection principles Populations and samples

Populations and samples

Consider the following three research questions:

1 What is the average mercury content in swordfish in the AtlanticOcean?

2 Over the last 5 years, what is the average time to degree forUCLA undergraduate students?

3 Does the drug sulphinpyrazone reduce the number of deaths inheart attack patients?


Overview of data collection principles Anecdotal evidence

Anecdotal evidence

Let’s consider the following possible responses to our threeresearch questions:

1 A man on the news got mercury poisoning from eating swordfish,so the average mercury concentration in swordfish must bedangerously high.

2 I met two students who took more than 10 years to graduate fromUCLA, so it must take longer to graduate at UCLA than at manyother colleges.

3 My friend’s dad had a heart attack and died after they gave himsulphinpyrazone. The drug must not work.

These are all anecdotal evidence, based on a limited sample sizethat might not be representative of the population.

Anecdotal evidence is typically composed of unusual cases thatwe recall based on their striking characteristics.


Overview of data collection principles Anecdotal evidence

Smoking research in the 1930s and 1940s ∗

Anti-smoking research started in the 1930s and 1940s whencigarette smoking became increasingly popular.

While some smokers seemed to be sensitive to cigarette smoke,others were completely unaffected.

Anti-smoking research was faced with resistance based onanecdotal evidence such as “My uncle smokes three packs a dayand he’s in perfectly good health”.

Back then it was concluded that “smoking is a complex humanbehavior, by its nature difficult to study, confounded by humanvariability.”

In time researchers were able to examine larger samples of cases(smokers) and trends showing that smoking has negative healthimpacts became much clearer.

∗The Cigarette Century (Brandt)Statistics 101 (Mine Cetinkaya-Rundel) L4: Data coll., obs. studies and experiments September 8, 2011 8 / 45

Overview of data collection principles Sampling from a population

Census

Wouldn’t it be better to just include everyone and “sample” theentire population?

Such a special sample is called a census.

There are problems with taking a census:

It can be difficult to complete a census: there always seem to besome individuals who are hard to locate or hard to measure. Andthere may be a certain characteristic about those individuals whoare hard to locate.Populations rarely stand still. Even if you could take a census, thepopulation changes while you work, so it’s never possible to get aperfect measure.Taking a census may be more complex than sampling.





Sampling is natural

Sampling is a natural thing to do.

Think about sampling something you are cooking - you taste(examine) a small part of what you’re cooking to get an ideaabout the dish as a whole.

If you walk into a store that you’re not familiar with, in order todecide if the store is affordable you wouldn’t check the tag ofevery single item in the store. You would instead try to checkout the price of a variety of items (a representative sample) andbased on what you see decide if you think the store overall isoverpriced or not.



Exploratory analysis to inference

When you taste a spoonful of soup and decide it doesn’t tastesalty enough, that’s exploratory analysis.

If you generalize and conclude that your soup needs salt, that’san inference.

For your inference to be valid the spoonful you tasted (thesample) needs to be representative of the entire pot (thepopulation).

If your spoonful comes only from the surface and the salt iscollected at the bottom of the pot, what you tasted is probablynot representative of the whole pot.If you first stir the soup thoroughly before you taste, yourspoonful will more likely be representative of the whole pot.



Random sampling

The cars data set was a random sample from the population ofall cars from 1993.

In simple random sampling all cases in the population are equallylikely to be selected, hence the resulting sample is representativeof the population.

all cars

sample



Biased sampling

If we were to ask a muscle car enthusiast to suggest cars to beselected, the sample could be biased towards that persons interests.

all cars

sample

high mileagecars



Other sources of bias

In surveys, if only a small fraction of the randomly sampledpeople choose to respond non-response bias may be introducedto the data since the sample may no longer be representative ofthe population.

Voluntary response bias occurs when those who respond havestrong opinions on the issue since such a sample will also not berepresentative of the population.

Another common downfall is a convenience sample, whereindividuals who are easily accessible are more likely to beincluded in the sample.



Clicker question

We can easily access ratings for products, sellers, and companiesthrough websites. These ratings are based only on those people whogo out of their way to provide a rating. What type of bias might thissample have?

(a) Non-response

(b) Voluntary response

(c) Convenience sample

(d) All of the above

If a seller has a rating of 95% on Amazon, do you think this numbermight be artificially low or high? Why?



Landon vs. FDR

A historical example of a biased sample yielding misleading results:

In 1936, Landonsought theRepublicanpresidentialnomination opposingthe re-election ofFDR.



The Literary Digest Poll

The Literary Digest polled about 10 millionAmericans, and got responses from about2.4 million.

The poll showed that Landon would likelybe the overwhelming winner and FDRwould get only 43% of the votes.

Election result: FDR won, with 62% of thevotes.

The magazine was completely discredited because of the poll,and was soon discontinued.



The Literary Digest Poll - what went wrong?

The magazine had surveyed

its own readers,registered automobile owners, andregistered telephone users.

These groups had incomes well above the national average of theday (remember, this is Great Depression era) which resulted inlists of voters far more likely to support Republicans than a trulytypical voter of the time, i.e. the sample was not representativeof the American population at the time.



Large samples are preferable, but...

The Literary Digest election poll was based on a sample size of2.4 million, which is huge, but since the sample was biased, thesample did not yield an accurate prediction.

Back to the soup analogy: If the soup is not well stirred, itdoesn’t matter how large a spoon you have, it will still not tasteright. If the soup is well stirred, it doesn’t matter whether youhave a large or small spoon, it will taste fine either way.


Overview of data collection principles Explanatory and response variables

Explanatory and response variables

To identify the explanatory variable and the response variable ina pair of variables, identify which of the two is suspected ofaffecting the other.

might affectexplanatoryvariable

responsevariable

Labeling variables as explanatory and response does notguarantee the relationship between the two is actually causal,even if there is an association identified between the twovariables. We use these labels only to keep track of whichvariable we suspect affects the other.


Overview of data collection principles Explanatory and response variables

Clicker question

A study is designed to test the effect of type of light on exam per-formance of students. 180 students are randomly assigned to threeclassrooms: one that is dimly lit, another with yellow lighting, and athird with white fluorescent lighting and given the same exam. Whichis correct?

(a) explanatory: dimly lit, yellow, white fluorescent; response: examperformance

(b) explanatory: exam performance, response: dimly lit, yellow, whitefluorescent

(c) explanatory: type of light (categorical with 3 levels); response:exam performance

(d) explanatory: exam performance; response: type of light(categorical with 3 levels)


Overview of data collection principles Introducing observational studies and experiments

Observational studies and experiments

Researchers perform an observational study when they collectdata in a way that does not directly interfere with how the dataarise.

When researchers want to establish a causal connection, theyconduct an experiment. Usually there will be both anexplanatory and a response variable.

To check for a causal connection between the explanatory andthe response variables experimental units (subjects) are randomlyassigned various treatments. This is called a randomizedexperiment.

In general, association does not imply causation, and causationcan only be inferred from a randomized experiment.


Observational studies and sampling strategies Observational studies

Observational studies

Suppose an observational study tracked sunscreen use and skin cancer,and it was found that the more sunscreen someone used, the more likelythe person was to have skin cancer. Does this mean sunscreen causesskin cancer?

sun exposure

use sunscreen skin cancerX



Observational studies (cont.)

It just so happens that if someone is exposed to the sun theyalso usually use sunscreen. Exposure to the sun is unaccountedfor in the investigation, giving the incorrect impression thatsunscreen causes skin cancer.

Sun exposure is what is called a lurking variable (or aconfounding variable).

Since observational studies do not control for lurking variablesthey cannot be used for establishing causality but they are usefulfor discovering trends and possible relationships.





3 possible explanations:

1 Eating breakfast causes girls to be thinner.

2 Being thin causes girls to eat breakfast.

3 A lurking variable is responsible for both.



Types of observational studies

A prospective study identifies individuals and collects informationas events unfold.

Medical researchers may identify and follow a group of similarindividuals over many years to assess the possible influences ofbehavior on cancer risk.

Retrospective studies collect data after events have taken place.

Researchers may review past events in medical records.


Observational studies and sampling strategies Three sampling methods

Three sampling methods

1 Simple random sampling: Each case in the population has equalchance of being selected.

2 Stratified sampling: Population is first divided into homogenousstrata and then cases are sampled randomly from within eachstrata.

3 Cluster sampling: Population is divided into clusters. Firstclusters are randomly sampled and then cases from within thoseclusters are also randomly selected. Unlike strata, clusters do notneed to be homogenous. Usually clusters are formed in a waythat makes sampling easier (more economical), such as based ongeography.

Describe a situation where cluster sampling would be more efficientthan simple random or stratified sampling.


Experiments Principles of experimental design

Principles of experimental design

1 Controlling: Compare treatment of interest to a control group.

2 Randomization: Randomly assign subjects to treatments.

3 Replication: Within a study, replicate by collecting a sufficientlylarge sample. Or replicate the entire study.

4 Blocking: If there are variables that are known or suspected toaffect the results, first group subjects intro blocks and thenrandomize cases within each block to treatment groups.



Clicker question

A study is designed to test the effect of light level and noise level onexam performance of students. The researcher also believes that lightand noise levels might have different effects on males and females, sowants to make sure both genders are represented equally under differentconditions. Which of the below is correct?

(a) There are 3 explanatory variables (light, noise, gender) and 1response variable (exam performance)

(b) There are 2 explanatory variables (light and noise), 1 blockingvariable (gender), and 1 response variable (exam performance)

(c) There is 1 explanatory variable (gender) and 3 response variables(light, noise, exam performance)

(d) There are 2 blocking variables (light and noise), 1 explanatoryvariable (gender), and 1 response variable (exam performance)



Clicker question

What is the main difference between observational studies and experi-ments?

(a) Experiments take place in a lab while observational studies do notneed to.

(b) In an observational study we only look at what happened in thepast.

(c) Most experiments use random assignment while observationalstudies do not.

(d) Observational studies are completely useless since no causalinference can be made based on their findings.



Random assignment vs. random sampling


Experiments Reducing bias in human experiments

Reducing bias in human experiments

Randomized experiments are the gold standard for datacollection, but they do not ensure an unbiased perspective intothe cause and effect relationships in all cases. Especially inhuman studies bias can unintentionally arise.

Using a control for the treatment is one way to reduce this bias.

Keeping the patients uninformed about their treatment, blinding,can also help reduce bias. This is usually done through the use ofa placebo.

If both the patients and the researchers are uninformed aboutwhich treatment which patient is getting, this is called adouble-blind study. This helps reduce

Note: If you have printed your slides earlier in the day you might be missing this slide.

Statistics 101 (Mine Cetinkaya-Rundel) L4: Data coll., obs. studies and experiments September 8, 2011

Case Study

Clicker question

Do you think yawning is conta-gious?

(a) Yes

(b) No

(c) Don’t know


Case Study

Is yawning contagious?

An experiment conducted by the MythBusters tested if a person canbe subconsciously influenced into yawning if another person nearthem yawns.

http:// www.yourdiscovery.com/ video/mythbusters-top-10-is-yawning-contagious


Case Study

Study description

In this study 50 people were randomly assigned to two groups: 34 toa group where a person near them yawned (seeded) and 16 to acontrol group where there wasn’t a yawn seed (control).

The results are as follows:

Seeded Control Total

Yawn 10 4 14Not Yawn 24 12 36

Total 34 16 50


Case Study Variability within data

Proportion of yawners


Yawn 10 4 14Not Yawn 24 12 36

Total 34 16 50

Proportion of yawners in the seeded group: 1034 = 0.29

Proportion of yawners in the control group: 416 = 0.25

Note: Our results match the ones calculated on the MythBusters episode.

Based on the proportions we calculated, do you think yawning is reallycontagious, i.e. are yawning and seeding dependent? Or is it possiblethat the difference in the proportion of yawners might be due to chance?



Dependence, or another possible explanation?

We saw that the proportion of yawners in the seeded group washigher, which may suggest that yawning is contagious (yawningand seeding are dependent).

But another possible explanation is that the difference in the twoproportions is simply due to chance. Perhaps if we were torepeat the experiment, we would see slightly different results.

So we will repeat the experiment many times - well, somewhat -and see what happens.

Instead of actually conducting the experiment many times, wewill simulate results.


http://www.yourdiscovery.com/video/mythbusters-top-10-is-yawning-contagious

http://www.yourdiscovery.com/video/mythbusters-top-10-is-yawning-contagious


But before we do that...


Yawn 10 4 14Not Yawn 24 12 36

Total 34 16 50

Now consider the idea that the seeding has absolutely no effect. Whatis the overall proportion of yawners?

Overall proportion of yawners: 1450 = 0.28

If seeding has no effect, what is the expected number of yawners in theseeded group?

Expected number of yawners in seeded group: 34 ∗ 0.28 = 9.52


Case Study Simulating the study

Simulating the experiment...

... under the assumption of independence, i.e. leave things up tochance.

If results from the simulations based on the chance model look likethe data, then we can determine that the difference between theproportions of yawners in the seeded and control groups was simplydue to chance (yawning is not contagious).

If the results from the simulations based on the chance model do notlook like the data, then we can determine that the difference betweenthe proportions of yawners in the seeded and control groups was notdue to chance, but due to an actual effect of seeding (yawning iscontagious).



Simulation setup

A regular deck of cards is comprised of 52 cards: 4 aces, 4 ofnumbers 2-10, 4 jacks, 4 queens, and 4 kings.

Take out two aces from the deck of cards and set them aside.

The remaining 50 playing cards to represent each participant inthe study:

14 face cards (including the 2 aces) represent the people whoyawn.36 non-face cards represent the people who don’t yawn.



Running the simulation

1 Shuffle the 50 cards at least 7 times ∗ to ensure that the cardscounted out are from a random process.

2 Count out the top 16 cards and set them aside. These cardsrepresent the people in the control group.

3 Out of the remaining 34 cards (seeded group) count the numberof face cards (the number of people who yawned in the seededgroup).

4 Record this number.

5 Repeat steps (1) - (4) many times.

∗http://www.dartmouth.edu/∼chance/course/topics/winning number.html


http://www.dartmouth.edu/~chance/course/topics/winning_number.html


Simulation results - Clicker

Click the corresponding choice for the number of face cards in theseeded group in your simulation.

(a) ≤ 6

(b) 7 - 8

(c) 9 - 10

(d) 11- 12

(e) ≥ 13


Case Study Checking for independence

Clicker question

Do the simulation results suggest that yawning is contagious, i.e. doesyawning and seeding appear to be dependent? (Hint: In the actualdata there were 10 yawners in the seeded group, does this appear tobe an unusual observation for the chance model?)

(a) Yes (b) No


Case Study Checking for independence

Simulations using software

http:// www.rossmanchance.com/ applets/ Yawning/ Yawning.html


http://www.rossmanchance.com/applets/Yawning/Yawning.html

Documents

Lecture 4: Data collection, observational studies and … · Lecture 4: Data collection, observational studies and experiments Statistics 101 Mine C˘etinkaya-Rundel September 8,