Common Statistical Mistakes

Mistake #1Failing to investigate data for data entry or recording errors.Failing to graph data and calculate basic descriptive statistics before analyzing data.

Example: Wrong Decision Due to Error

Example:Wrong Decision Due to ErrorTest of mu = 26.000 vs mu not = 26.000

Variable N Mean StDev SE Mean T PWith 16 25.625 3.964 0.991 -0.38 0.71Without 15 24.733 1.792 0.463 -2.74 0.016Variable N Mean StDev SE Mean 95.0 % CIWith 16 25.625 3.964 0.991 (23.513, 27.737)Without 15 24.733 1.792 0.463 (23.741, 25.725)

Mistake #2Using the wrong statistical procedure in analyzing your data.Includes failing to check that necessary assumptions are met.

Example: Wrong Decision Due to Wrong AnalysisPaired Data Design, so analyze with Paired t-test.

Example: Wrong Decision Due to Wrong AnalysisPaired T for AFTER - BEFORE

N Mean StDev SE MeanAFTER 4 82.00 12.96 6.48BEFORE 4 71.00 15.87 7.94Difference 4 11.00 5.03 2.52

95% CI for mean difference: (2.99, 19.01)T-Test of mean difference = 0 (vs not = 0): T-Value = 4.37 P-Value = 0.02Conclude mean pulse rate after is greater than mean pulse rate before.

Example: Wrong Decision Due to Wrong AnalysisTwo sample T for AFTER vs BEFORE

N Mean StDev SE MeanAFTER 4 82.0 13.0 6.5BEFORE 4 71.0 15.9 7.9

95% CI for mu AFTER - mu BEFORE: ( -15.3, 37.3)T-Test mu AFTER = mu BEFORE (vs not =): T = 1.07 P = 0.33 DF = 5Conclude no difference in mean pulse rates before and after marching.

Mistake #3Failing to design your study so that it has high enough power to call meaningful differences significantly different.Includes concluding that the null hypothesis is true. Should be not enough evidence to say the null is false.

Example: Low PowerSuccess = Yes, I recycle.

Gender X N Sample pMale 33 59 0.559322Female 54 79 0.683544

Estimate for p(1) - p(2): -0.12422295% CI for p(1) - p(2): (-0.287215, 0.0387704)Test for p(1) - p(2) = 0 (vs not = 0): Z = -1.49 P-Value = 0.135A number of students said that they were surprised that the hypothesis test said no difference in percentages.

Example: Low PowerPower and Sample SizeTest for Two Proportions

Testing proportion 1 = proportion 2 (versus not =)Calculating power for: proportion 1 = 0.55 and proportion 2 = 0.70Alpha = 0.05 Difference = -0.15

Sample Size Power 60 0.4366 70 0.4911 80 0.5421 *Sample size = # in EACH group

Mistake #4Failing to report a confidence interval as well as the P-value.P-value tells you if statistically significant.Confidence interval tells you what the population value might be.

Example: A Significant, but Potentially Meaningless DifferenceTwo sample T for Phone

Gender N Mean StDev SE MeanMale 59 79 162 21Female 80 153 247 28

95% CI for mu (1) - mu (2): ( -142, -5)T-Test mu (1) = mu (2) (vs not =): T = -2.11 P = 0.036 DF = 135P-value tells us significant difference, but confidence interval tells us that the difference in the averages could be as small as 5 minutes.

Incidentally.Outliers

Removing Outliers Two sample T for Phone

Gender N Mean StDev SE MeanMale 58 59.9 66.5 8.7Female 79 129 133 15

95% CI for mu (1) - mu (2): ( -103.7, -35)T-Test mu (1) = mu (2) (vs not =): T = -4.02 P = 0.0001 DF = 121The difference in male and female phone usage becomes even more significant. We are 95% confident that the difference in the averages is now more than 35 minutes.

Mistake #5Fishing for significant results. That is, performing several hypothesis tests on a data set, and reporting only those results that are significant.If = P(Type I) = 0.05, and we perform 20 tests on the same data set, we can expect to make 1 Type I error. (0.05 20 = 1).

Example: Results Obtained from Fishing Primary driver of $10,000 vehicle and going away for Spring Break are related (P=0.01).Virginity and supporting self through school are related (P = 0.045).Virginity and graduating in four years are related (P = 0.041).Virginity and attending non-football PSU sports events are related (P = 0.016).

Mistake #6Overstating the results of an observational study. That is, suggesting that one variable caused the differences in the other variable.As opposed to correctly saying that the two variables are associated or correlated.Dont forget that a significant result may be spurious.

Example: Misleading HeadlinesVirgins dont support themselves through school.Non-virgins too busy to go to non-football PSU sporting events.Non-virgins also too busy to graduate in four years.

Mistake #7Using a non-random or unrepresentative sample.Includes extending the results of an unrepresentative sample to the population.

Example: Unrepresentative sampleShere Hite wrote a book in 1987 called Women in Love100,000 questionnaires about love, sex, and relationships sent to womens groups. Only 4,500 questionnaires returned.Entire book devoted to results of survey.Examples: 91% of divorcees initiated the divorce; 70% of women married 5 years committed adultery.

Mistake #8Failing to use all of the basic principles of experiments, including randomization, blinding, and controlling.