Significance, Importance, and Undetected DifferencesReal Importance versus Statistical Significance

A statistically significant relationship or difference does not necessarily mean an important one.

•Whether results are statistically significant or not, it is helpful to examine a confidence interval so that you can determine the magnitude of the effect.

•From width of the confidence interval, also learn how much uncertainty there was in sample results.

Example : Is Aspirin Worth the Effort?

Relationship between taking aspirin and incidence of heart attack. Null (no relationship) vs alternative (yes relationship), chi-squared (test) statistic over 25 with p-value < 0.00001.

The Magnitude of the Effect

The test statistic and p-value do not provide information about the magnitude of the effect.

Representing the Size of the EffectRates of heart attack: 9.4 per 1000 for aspirin group and 17.1 per 1000 for placebo group, difference < 8 people per 1000, about 1 less heart attack for every 125 who took aspirin.

Relative risk: Aspirin group had half as many heart attacks; so could cut risk almost in half. Estimated relative risk as 0.53, with a 95% confidence interval extending from 0.42 to 0.67.

Significance, Importance, and Undetected DifferencesRole of Sample Size in Width of Confidence Intervals

Precision = variation/square root of the sample size

• In a study looking at a continuous variable such as age, height or blood pressure, that variation isexpressed in terms of standard deviation:

a large sample standard deviation of say, age, means that the individuals in your study are all sorts of ages from young to old (e.g., a study of voters);

a small sample standard deviation means that most of those in your study are a similar age (e.g., a study of professional football players).

•So if you want to make your results more precise, you need to increase your sample size: put more patients on a clinical trial for example

•But precision is related to the square root of sample size, so if you want to double your precision you need to quadruple your sample size.

Significance, Importance, and Undetected DifferencesRole of Sample Size in Statistical

SignificanceThere is almost always a slight relationship between two variables, or a difference between two groups, and if you collect enough data, you will find it.

If the sample size is too small, an important relationship or difference can go undetected.

In that case, we would say that the power of the test is too low.

Significance, Importance, and Undetected DifferencesType I, Type II errors and Power of a Test

Significance, Importance, and Undetected Differences

The Power of a Test - Calculating Sample SizeThe power of a test is the probability of making the correct decision when the alternative hypothesis is true.

Medical Testing Example: Detecting that a person has the disease when they truly have it, a true positive.

•The typical question that scientists ask isn't, "OK, I've got 100 people here. What are my chances of a positive result?" but, "I want to test a hypothesis. How many people will I need?"

•This sort of question is particularly important in medical research.

•In a trial of a new drug, for example, you want to have a good chance of a statistically significant result if the drug is effective because it would be great to have another way to help sick people.

•But you can't have too large a sample size-drugs often have side effects, so you don't want togive a new drug to lots of people if it doesn't work.

Significance, Importance, and Undetected Differences

The Power of a Test - Calculating Sample SizeA typical question asked of statisticians in medical research might be something like:

• About 20% of patients recover from a typical cold within 36 hours. We think that our new cold treatment might increase this to 30%.

• How big a sample do we need to avoid making a type II error – failing to reject the null hypothesis when it is false – detecting a true difference.

•As a general rule, we use the usual alpha of 5% and a power of 80%, that is, if the drug is effective, we want a 80% chance of showing that it does indeed work.

How many patients do we need?

Significance, Importance, and Undetected Differences

If the Outcome is:

And we want to detect a Difference of:

Sample Size Required*

5%

1% 6,391

2% 1,725

3% 821

10%

1.5% 5,236

3% 1,388

5% 534

15%

3% 1,882

6% 502

9% 236

20%

2% 5,109

5% 857

10% 229

25%

25% 3,818

5% 980

12.5% 168

30%

3% 2,951

9% 342

15% 127

35%

3.5% 2,337

9% 362

17.5% 98

40%

4% 1,873

10% 304

20% 76

45%

4.5% 1,514

10% 307

22.5% 59

SOME BENEFITS OF META-ANALYSISMETA-ANALYSIS

1. Detecting small or moderate relationships

2. Obtaining a more precise estimate of a relationship

3. Determining future research

4. Finding patterns across studies

CRITICISMS OF META-ANALYSIS1. Simpson’s Paradox

2. Confounding variables

3. Subtle differences in treatments of the same name

4. The file drawer problem

5. Biased or flawed original studies

6. Statistical significance versus practical importance