Sample Size and Power Steven R. Cummings, MD Director, S.F. Coordinating Center

SFSFCoordinating CenterCoordinating Center

Sample Size and Power

Steven R. Cummings, MDSteven R. Cummings, MD

Director, S.F. Coordinating CenterDirector, S.F. Coordinating Center


The Secret of Long Life

• ResveratrolResveratrol

• In the skin of red grapesIn the skin of red grapes

• Makes mice Makes mice Run fasterRun faster Live longerLive longer


What I want to show

• Consuming reservatrol Consuming reservatrol prolongs healthy lifeprolongs healthy life


Sample Size Ingredients

• Testable hypothesisTestable hypothesis

• Type of studyType of study

• Statistical testStatistical test Type of variablesType of variables

• Effect size (and its variance)Effect size (and its variance)

• Power and alphaPower and alpha


Sample Size Ingredients

• Testable hypothesisTestable hypothesis






My research question

• I need to plan the studyI need to plan the study

• My question isMy question is

Does consuming reservatrol Does consuming reservatrol

lead to a long and healthy life?lead to a long and healthy life?


What’s wrong with the question?

• I need to plan the studyI need to plan the study

• My question isMy question is

Does consuming reservatrol Does consuming reservatrol

lead to a long and healthy life?lead to a long and healthy life?


What’s wrong with the question?

Does consuming resveratrol lead to a Does consuming resveratrol lead to a long and healthy life?long and healthy life?

• VagueVague

• Must be measurableMust be measurable


“Consuming resveratrol”

• Most rigorous design: Most rigorous design: randomized placebo-randomized placebo-controlled trial controlled trial

• Comparing red wine to Comparing red wine to placebo would be difficultplacebo would be difficult

• But resveratrol But resveratrol supplements are widely supplements are widely availableavailable


Measurable (specific) outcome

• ““Consuming resevertrol” =Consuming resevertrol” =taking resveratrol supplements taking resveratrol supplements vs. taking placebovs. taking placebo

• ““Prolong healthy life” = Prolong healthy life” =


Measurable (specific) outcome

• ““Consuming resevertrol” =Consuming resevertrol” =taking resveratrol supplements taking resveratrol supplements vs. taking placebovs. taking placebo

• ““Prolong healthy life” = Prolong healthy life” = reduces all-cause mortalityreduces all-cause mortality

Do Do peoplepeople randomized to get a resveratrol randomized to get a resveratrol supplement have a lower mortality rate than supplement have a lower mortality rate than those who get a placebo?those who get a placebo?


In whom?

• Elderly men and women (≥70 years)Elderly men and women (≥70 years)


The research hypothesis

Men and women > age 70 yearsMen and women > age 70 years randomized to get a resveratrol randomized to get a resveratrol supplement have a lower mortality rate supplement have a lower mortality rate than those who get a placebo.than those who get a placebo.


The research hypothesisThe ‘alternative’ hypothesis

Men and women > age 70 years Men and women > age 70 years randomized to get a resveratrol randomized to get a resveratrol supplement have a lower mortality rate supplement have a lower mortality rate than those who get a placebo.than those who get a placebo.

• Cannot be tested statisticallyCannot be tested statistically

• Statistical tests only reject null hypothesis Statistical tests only reject null hypothesis - that there is no effect- that there is no effect


The Null Hypothesis

Men and women > age 70 years Men and women > age 70 years randomized to receive a resveratrol randomized to receive a resveratrol supplement supplement do notdo not have lower mortality rate have lower mortality rate than those who receive placebo.than those who receive placebo.

• Can be rejected by statistical testsCan be rejected by statistical tests


Ingredients for Sample Size

Testable hypothesisTestable hypothesis






Type of study

• DescriptiveDescriptive Only one variable / measurementsOnly one variable / measurements

• What proportion of centenarians take What proportion of centenarians take resveratrol supplements?resveratrol supplements? Confidence interval for proportionsConfidence interval for proportions

• What is the mean red wine intake of What is the mean red wine intake of centenarians?centenarians? Confidence interval for the meanConfidence interval for the mean


Sample size for a descriptive study

For example:For example:

• ““What proportion of centenarians (>100 years old) What proportion of centenarians (>100 years old) take resveratrol supplements?”take resveratrol supplements?”

• How much precision do you want?How much precision do you want? Sample size is based on the width of the confidence Sample size is based on the width of the confidence

interval (Table 6D and 6E)interval (Table 6D and 6E)

• I assume that 20% of centenarians take resveratrolI assume that 20% of centenarians take resveratrol Conventional 95% C.I.Conventional 95% C.I. I want to be confident that the truth is within ±10%I want to be confident that the truth is within ±10% Total width of the C.I. = 0.20Total width of the C.I. = 0.20



Analytical studies

• Analytical means a comparison Analytical means a comparison Cross-sectionalCross-sectional

Mean red wine intake in centenarians Mean red wine intake in centenarians vsvs.. 60-80 year olds 60-80 year olds

Randomized trialRandomized trial Elders who get resveratrol have lower Elders who get resveratrol have lower

mortality mortality thanthan those who get placebo those who get placebo




Type of study: analytical (RCT)Type of study: analytical (RCT)





Type of statistical testsDepends on the types of variables

Predictor,outcome

Dichotomous Continuous

Dichotomous Chi-squared T test

Continuous T test Correlation

This works for This works for most study planningmost study planning


The types of variables?

Men and women > age 70 yearsMen and women > age 70 years randomized to receive a resveratrol randomized to receive a resveratrol supplement do not have lower mortality rate supplement do not have lower mortality rate than those who receive placebothan those who receive placebo

• Dichotomous: resveratrol or placeboDichotomous: resveratrol or placebo

• Continuous: mortality Continuous: mortality raterate

What’s wrong?What’s wrong?


The types of variables?


• Dichotomous: reseveratrol or placeboDichotomous: reseveratrol or placebo

• Continuous: mortality Continuous: mortality raterate It is a It is a proportionproportion at certain times at certain times For example, 3% at 1 yearFor example, 3% at 1 year


The appropriate test for this randomized trial for mortality

Predictor,outcome

Dichotomous Continuous

Dichotomous Chi-squared T test

Continuous T test Correlation





Statistical testStatistical test

Type of variablesType of variables




Estimating the effect size

For randomized trials,For randomized trials,

• Start with the expected rate in the placeboStart with the expected rate in the placebo

• Usually available from population or Usually available from population or cohort studiescohort studies

• In this case, we know the mortality rates In this case, we know the mortality rates by age:by age:

• 3-4% per year*; for a 3 year study: 10%3-4% per year*; for a 3 year study: 10%

* ~ * ~ mean annual female/males @ 78 yrs mean annual female/males @ 78 yrs


Effect sizethe hardest part

What should I assume for the effect of What should I assume for the effect of resveratrol on mortality?resveratrol on mortality?


Effect sizethe hardest part

Ways to choose an effect size:Ways to choose an effect size:

• What is likely, based on other data?What is likely, based on other data?

• Do a pilot studyDo a pilot study

• Estimate based on effect on biomarkersEstimate based on effect on biomarkers

• What difference is important to detect?What difference is important to detect? ““We don’t want to miss a __We don’t want to miss a __%%_ difference”_ difference”

• What can we afford?What can we afford?


The effect of resveratrol on mortality rate?



• Estimate based on effect on biomarkersEstimate based on effect on biomarkers

• What difference is important to detect?What difference is important to detect? ““We don’t want to miss a __We don’t want to miss a __%%_ difference”_ difference”

• What can we afford?What can we afford?


Resveratrol pronged survival of mice fed high calorie diet

Baur, Nature 2006Baur, Nature 2006

~ 25%~ 25%




• Pilot study? What endpoint?Pilot study? What endpoint?

• No reliable markers for the effect on deathNo reliable markers for the effect on death

• What difference is important to detect?What difference is important to detect? ““We don’t want to miss a ____ difference”We don’t want to miss a ____ difference”

• What can we afford to find?What can we afford to find?





• Estimate based on biomarkersEstimate based on biomarkers

• What difference is important to detect?What difference is important to detect? ““We don’t want to miss a We don’t want to miss a _1%_1%_ difference”_ difference”

• What can we afford?What can we afford? 1%: too big & expensive1%: too big & expensive 5%: small and cheap5%: small and cheap



• Finding a smaller effect is important to Finding a smaller effect is important to healthhealth

• Allowing a larger effect is important for your Allowing a larger effect is important for your budgetbudget


Effect size


• It would be important to find (I don’t want to It would be important to find (I don’t want to miss) a 20% decreasemiss) a 20% decrease

• Placebo rate: 10%Placebo rate: 10%

• Resveratrol rate: 8%Resveratrol rate: 8%





Statistical testStatistical test

Type of variablesType of variables

Effect size (and its variance)Effect size (and its variance)



(alpha)

The probability of finding a The probability of finding a ‘significant’ result if ‘significant’ result if nothing is going onnothing is going on


I will need to convince people

• Customarily, a result is ‘statistically Customarily, a result is ‘statistically significant’ if P<0.05significant’ if P<0.05

In other words,In other words,

• Probability of a type I error = 5% Probability of a type I error = 5%

• (alpha) = 0.05(alpha) = 0.05


I will need to convince skeptics

• VeryVery small chance that a positive result is small chance that a positive result is an erroran error

(alpha) = 0.01 (alpha) = 0.01

P<0.01P<0.01

• A smaller A smaller means larger sample sizemeans larger sample size


Two-sided vs. one-sided

• A 2-sided A 2-sided assumes that the result could go either assumes that the result could go either wayway Recognizes that you have two chances of finding Recognizes that you have two chances of finding

something that isn’t really theresomething that isn’t really there Resveratrol decreases mortalityResveratrol decreases mortality Resveratrol increases mortalityResveratrol increases mortality

• A 1-sided hypothesis reduces sample size A 1-sided hypothesis reduces sample size (somewhat)(somewhat) A one-sided A one-sided of 0.05 corresponds to a two-sided of 0.05 corresponds to a two-sided of of

0.100.10

• It assumes that the result could, plausibly, go only It assumes that the result could, plausibly, go only one wayone way


Two-sided vs. one-sided

• You may believe that your effect could only go You may believe that your effect could only go one way!one way! Resveratrol is ‘natural.’ It could not increase Resveratrol is ‘natural.’ It could not increase

mortality!mortality!

• Be humble. Be humble. The history of research is filled with results that The history of research is filled with results that

contradicted expectationscontradicted expectations Vitamin D trial (JAMA 2010): Vitamin D trial (JAMA 2010):

To everyone’s surprise, ~1500 IU of vitamin To everyone’s surprise, ~1500 IU of vitamin D/d D/d increasedincreased the risk of falls and fractures in the risk of falls and fractures in elderly women and menelderly women and men

• A 1-sided test is almost never the best choiceA 1-sided test is almost never the best choice


(beta)

The probability of The probability of missingmissing this this effect size in this sample, if it is effect size in this sample, if it is really true in the populations really true in the populations


Power (1-)

The probability of The probability of findingfinding this this effect size in this sample, if it is effect size in this sample, if it is

really true in the population really true in the population


If it’s true, I don’t want to miss it

• The chance of missing the effect (The chance of missing the effect ())

is “customarily” 20%is “customarily” 20%

In other wordsIn other words

• Probability of a type II error = 0.20Probability of a type II error = 0.20

• (beta) = 0.20(beta) = 0.20

• PowerPower = 1- = 1- 0.800.80


I really don’t want to miss it

• = .10= .10

• Power (1- Power (1- ) = 0.90) = 0.90

• Greater power requires a larger sample Greater power requires a larger sample sizesize


We have all of the ingredients



Statistical test: Chi-squaredStatistical test: Chi-squared

Effect size 10% vs 8%Effect size 10% vs 8%

Power: 0.90; alpha: 0.20Power: 0.90; alpha: 0.20


From Table 6B.2Comparing two proportions




From Table 6B.2

• Sample size: 4,401Sample size: 4,401

• Per groupPer group

• Total: 8,802Total: 8,802

• Does not include drop-outsDoes not include drop-outs 20% drop-out: 11,002 total sample size!20% drop-out: 11,002 total sample size!


Alternatives

• Tweak Tweak one-sided one-sided Almost never appropriate Almost never appropriate

• Tweak the power: 0.80Tweak the power: 0.80




Alternatives

• Tweak Tweak one-sided one-sided Almost never appropriate Almost never appropriate

• Tweak the power: 0.80Tweak the power: 0.80

• Modest effect: 3,308 (6,616 total)Modest effect: 3,308 (6,616 total)


Alternatives

• Increase the effect sizeIncrease the effect size 10% vs. 6% 10% vs. 6%




Increasing the effect size

• 10% vs. 6% 10% vs. 6%

• Makes a Makes a bigbig difference! difference!

• 769 / group; 1,538 total 769 / group; 1,538 total (no dropouts)(no dropouts)

• However, still large (and not affordable)However, still large (and not affordable)

• Not believableNot believable


Alternatives: a new hypothesis

• Change the outcome measureChange the outcome measure Continuous measurementContinuous measurement A precise measurementA precise measurement

• A ‘surrogate’ for mortality rateA ‘surrogate’ for mortality rate Strongly associated with mortality rateStrongly associated with mortality rate Likely to be influenced by resveratrolLikely to be influenced by resveratrol

• Walking speedWalking speed


Mice on resveratrol

• Mice fed resveratrolMice fed resveratrol Live 25% longerLive 25% longer Are significantly Are significantly

fasterfaster Have greater Have greater

enduranceendurance


Increased gait speed (0.1 m/s) in 1 year and survival over 8 years

Faster by ≥0.1 m/s Faster by ≥0.1 m/s

Slower Slower

~20% decreased ~20% decreased mortality ratemortality rate


The new ingredients

New testable hypothesis New testable hypothesis

Type of study: RCTType of study: RCT

• Statistical test: Statistical test: t-testt-test Continuous variable (walking speed)Continuous variable (walking speed) Difference between means (pbo vs. tx)Difference between means (pbo vs. tx)

> Effect size and variance: E/SEffect size and variance: E/S



E/S

• For t-scores (dichotomous predictor, For t-scores (dichotomous predictor, continuous outcome)continuous outcome)

• Sample size depends on the ratio of E/SSample size depends on the ratio of E/S E: Effect size (difference in means)E: Effect size (difference in means) S: Standard deviation for measurementS: Standard deviation for measurement

• You need smaller sample size ifYou need smaller sample size if Greater effect (E)Greater effect (E) More precise measurement (lower SD)More precise measurement (lower SD)


What we need to determine E/S for our RCT

• Effect size (E) for change in walking speedEffect size (E) for change in walking speed Mean baseline value = 1.0 m/sMean baseline value = 1.0 m/s Change in the placebo group = 0 Change in the placebo group = 0 Change with resveratrol = +0.1 m/sChange with resveratrol = +0.1 m/s

• Standard deviation (S) Standard deviation (S) Standard deviation of the changeStandard deviation of the change


S

• Standard deviation for the measurementStandard deviation for the measurement Cross-sectional data: 0.25 m / secCross-sectional data: 0.25 m / sec

• However, we are interested in However, we are interested in changechange

• Standard deviation of Standard deviation of changechange in speed? in speed? Often more difficult to find because Often more difficult to find because

cross-sectional surveys are more cross-sectional surveys are more common than longitudinal studiescommon than longitudinal studies


What if you don’t know the SD?

• Standard deviation of Standard deviation of changechange in speed? in speed?

• If you cannot find data from other studiesIf you cannot find data from other studies

• AlternativesAlternatives Pilot study: measure change in 3 or 6 Pilot study: measure change in 3 or 6

mo.smo.s


What if you don’t know the SD?

• Standard deviation of Standard deviation of changechange in speed? in speed?

• If you cannot find data from other studiesIf you cannot find data from other studies

• AlternativesAlternatives Pilot study: measure change in 3 or 6 Pilot study: measure change in 3 or 6

mo.smo.s Or, a well educated guess Or, a well educated guess


Estimating an S.D.the 1/4 rule

~ 4 S.D.s across a ‘usual’ range of values~ 4 S.D.s across a ‘usual’ range of values

So, 1 S.D. will = 1/4 of the rangeSo, 1 S.D. will = 1/4 of the range


Estimating S.D. for change in walking speed

the 1/4 rule

• Range of changes over 1 year*Range of changes over 1 year*

• +0.2 m/sec to -0.6 m/sec+0.2 m/sec to -0.6 m/sec

• Range = 0.8 m/secRange = 0.8 m/sec

• 1/4 of the range = 0.2 m/sec 1/4 of the range = 0.2 m/sec

* Single short, 6 meter walk* Single short, 6 meter walk


E/S

• Effect size: 0.1 m/sec difference in Effect size: 0.1 m/sec difference in change change

• Standard deviation: 0.2 m/secStandard deviation: 0.2 m/sec

• E/S = 0.5 E/S = 0.5


The new ingredients



Statistical test: t-testStatistical test: t-test Continuous variable Continuous variable Difference between meansDifference between means

Effect size 1.0 vs. 1.1 m/sec; E/S = 0.5Effect size 1.0 vs. 1.1 m/sec; E/S = 0.5

Power: Power: 0.800.80; alpha: 0.20; alpha: 0.20



The new ingredients



T-testT-test

Effect size 1.0 vs.1.1 m/sec; E/S: 0.5Effect size 1.0 vs.1.1 m/sec; E/S: 0.5

Power: Power: 0.800.80; alpha: 0.20; alpha: 0.20

Sample size: 64 per groupSample size: 64 per group; ; 128 total128 total

With 20% drop out: With 20% drop out: 160 total160 total


Improving precision of the outcome measurement

• Increased precision (decreased SD) will Increased precision (decreased SD) will decrease the sample size decrease the sample size

• For exampleFor example Mean of repeated walksMean of repeated walks Longer, 400 m walkLonger, 400 m walk Standard deviation may improve from Standard deviation may improve from

0.2 m/sec to 0.15 m/sec0.2 m/sec to 0.15 m/sec

• E/S improves from 0.5 to 0.1÷ 0.15 = 0.7E/S improves from 0.5 to 0.1÷ 0.15 = 0.7


A modest improvement in precisionA modest improvement in precisionreduced sample size by 1/2reduced sample size by 1/2


Summary

• Estimate sample size earlyEstimate sample size early

• Systematically collect the ingredientsSystematically collect the ingredients

• Effect size is the most difficult - and Effect size is the most difficult - and important - judgementimportant - judgement

• Alternatives that reduce sample sizeAlternatives that reduce sample size Compromise powerCompromise power Increase effect sizeIncrease effect size Precise continuous outcomesPrecise continuous outcomes

Documents

Sample Size and Power Steven R. Cummings, MD Director, S.F. Coordinating Center