Statistics for clinicians

Statistics for cliniciansStatistics for clinicians Biostatistics course by Kevin E. Kip, Ph.D., FAHA

Professor and Executive Director, Research CenterUniversity of South Florida, College of NursingProfessor, College of Public HealthDepartment of Epidemiology and BiostatisticsAssociate Member, Byrd Alzheimer’s InstituteMorsani College of MedicineTampa, FL, USA

1

SECTION 7.1SECTION 7.1Introduction to Introduction to Meta-AnalysisMeta-Analysis

Meta analysis and measures of public

health impact

Learning Outcome:

Recognize the methodological basis for conducting meta-analyses.

Question 1: Why is it difficult to appropriately evaluate/synthesize findings from multiple sources of research?

Question 2:Why do we often arrive at different or incorrect conclusions in groups of studies,and among individual studies?

Key QuestionsKey Questions

Traditional DefinitionsTraditional Definitions

Meta-Analysis: A quantitative approach for systematically combining the RESULTS of previous research in order to arrive at conclusions about the body of research.

(e.g. Determination of risk ratio and confidence interval across studies – Is Intervention A better than Intervention B?)

Traditional DefinitionsTraditional Definitions

Pooled-Analysis: Pooling of PRIMARY DATA from multiple studies for the purpose of conducting an analysis of the enlarged data set.

(Often referred to as “meta-analysis using individual patient data”).

National Health and Medical Research Council (NHMRC) levels of evidence and corresponding National Heart, Lung, and Blood Institute categories

NHLBI category

NHMRC level

Basis of Evidence

A I Evidence obtained from a systematic review of all

relevant randomised controlled trials

B II Evidence obtained from at least one properly designed randomised controlled trial

C III - 1 Evidence obtained from well-designed pseudorandomised controlled trials (alternate allocation or some other method)

C III - 2 Evidence obtained from comparative studies (including systematic reviews of such studies) with concurrent controls and allocation not randomised, cohort studies, case-control studies, or interrupted time series with a control group

C III - 3 Evidence obtained from comparative studies with historical control, two or more single arm studies, or interrupted time series without a parallel group

C IV Evidence obtained from case series, either post-test or pretest/ post-test

The Need for Meta-AnalysisThe Need for Meta-Analysis1940: 2,300 Biomedical journals

2011: 39,529 journals listed in PubMed

2011: More than 10,000 randomized clinical trials conducted each year

Similar proliferation of journal articles for social science disciplines

Range of Reaction to Meta-AnalysisRange of Reaction to Meta-AnalysisSupportive:

Cook, Guyatt (1994)The professional meta-analyst: an evolutionary

advantage

Rosendaal (1994)The emergence of a new species: the professional meta-analyst

Feinstein (1995)Meta-analysis: statistical alchemy for the 21st

century

Range of Reaction to Meta-AnalysisRange of Reaction to Meta-Analysis

Neutral:

Meinart (1989)Meta-analysis: science or religion?

Spector, Thompson (1991)The potential and limitations of meta-analysis.

Bailar (1997) The promise and problems of meta-analysis.

Range of Reaction to Meta-AnalysisRange of Reaction to Meta-AnalysisCritical:Eysenck (1978)

An exercise in mega-silliness.

Chalmers (1991)Problems induced by meta-analysis

Thompson, Pocock (1991)Can meta-analysis be trusted?

Greenland (1994)Can meta-analysis be salvaged?

Shapiro (1994)Meta-analysis/schmeta-analysis

SECTION 7.2SECTION 7.2Steps InvolvedSteps Involved

In Meta AnalysisIn Meta Analysis

Learning Outcome:

Demonstrate the primary steps involved in conducting a meta-analysis

Steps in a Meta-AnalysisSteps in a Meta-Analysis

Step 1: Identify studies with relevant data

Step 2: Define inclusion and exclusion criteria for component studies

Step 3: Abstract the data

Step 4: Analyze the data


Step 1: Identify studies with relevant data

• Completeness of information (e.g. published and unpublished reports)

• Specificity of hypothesis (e.g. similarity of treatments and/or exposure)

• Choice among multiple publications

Steps in a Meta-AnalysisSteps in a Meta-AnalysisStep 2: Define inclusion and exclusion

criteria for component studies• Study designs (experimental, observational, etc.)

• Years of publication of study conduct

• Languages (e.g. English language only)

• Incomplete data (e.g. loss to follow-up)

• Quality (e.g. subject selection, blinding, treatment compliance, statistical methods, etc.)

Step 2: Define inclusion and exclusion criteria for studies

Steps in a Meta-AnalysisSteps in a Meta-AnalysisStep 3: Abstract the data

• Select the desired measure of effect and reported estimate (e.g. odds ratio, standardized mean difference, correlation coefficient)

--- unadjusted--- adjusted for age only--- adjusted for multiple confounders, etc.

• Are the data available for subgroup analyses?

Concern: Publication BiasStatistically significant results 3x more likely to be published than papers affirming a null result.(1) Most common reason for non-publication is investigator declining to submit results(2) e.g.:

--- loss of interest in topic--- expectation that others will not be

interested in null resultsAlso known as “file drawer” bias.

1. Dickersin K, Chan S, Chalmers TC, et al. Publication bias and clinical trials. Controlled Clin Trials 1987; 8: 343-53

2. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867 72.

Selection of Component StudiesSelection of Component Studies

http://clinicaltrials.gov/ct2/home

ClinicalTrials.gov is a registry and results database of federally and privately supported clinical trials conducted in the United States and around the world. ClinicalTrials.gov gives you information about a trial's purpose, who may participate, locations, and phone numbers for more details. This information should be used in conjunction with advice from health care professionals.

Who is responsible for registering the trial?In most cases, the Sponsor of the trial as defined by FDA regulations[21 CFR 50.3(e)] has the obligation to register the clinical trial with ClinicalTrials.gov.

http://clinicaltrials.gov/ct2/home


Step 4: Analyze the data (statistical methods)

• Fixed effects

• Random effects

• Bayesian approach (not discussed)

SECTION 7.3SECTION 7.3Statistical Methods Statistical Methods

in Meta-Analysisin Meta-Analysis

Learning Outcomes:

Recognize analytical considerations in conducting a meta-analysis

Calculate and interpret a summaryodds ratio using the Mantel-

Haenszel method

Step 4: Analyze the data (statistical methods)

Fixed Effect Method: Meta-AnalysisFixed Effect Method: Meta-Analysis

--- The “within-study” variance is used as the weighting factor for each study

Example: MH method to estimate the summary odds ratio:

sum(weighti x ORi)ORMH = ----------------------------

sum(weighti)

where weighti = 1 / varianceNote – variance is for log(OR)

sum(weighti x ORi) 821.24ORMH = ---------------------------- OR = ---------------------- = 0.90

sum(weighti) 910.56

CASES CONTROLSExposed Not exposed Exposed Not exposed OR Variance Weight W x OR

Study 1 49 67 566 557 0.72 0.0389 25.71 18.50Study 2 44 64 714 707 0.68 0.0412 24.29 16.54Study 3 102 126 730 724 0.80 0.0205 48.80 39.18Study 4 32 38 285 271 0.80 0.0648 15.44 12.36Study 5 85 52 725 354 0.80 0.0352 28.41 22.67Study 6 246 219 2021 2038 1.13 0.0096 103.99 117.79Study 7 1570 1720 7017 6880 0.89 0.0015 663.92 594.19

Totals 2128 2286 12058 11531 910.56 821.24

Example Calculations for Study #1

OR = (Odds of exposure among cases / Odds of exposure among controls)OR = (49 / 67) / (566 / 557) = 0.72

Variance - (log)OR = ((1/N1) + (1/N2) + (1/N3) + (1/N4))Variance – log(OR) = ((1/49) + (1/67) + (1/566) + (1/557) = 0.0389

Weight = 1 / variance = 1 / 0.0389 = 25.71Weight x OR = 25.71 x 0.72 = 18.50

Example: Mantel Haenszel Summary Odds Ratio

sum(weighti x ORi)ORMH = ---------------------------- OR = ---------------------- =

sum(weighti)

CASES CONTROLS Exposed Not exposed Exposed Not exposed OR Variance Weight W x ORStudy 1 32 72 111 445Study 2 19 39 54 132Study 3 12 20 16 44Study 4 62 112 48 110Study 5 43 72 51 74Study 6 90 192 74 202Totals 258 507 354 1007

Practice: Calculate the Mantel-Haenszel Summary Odds Ratio for the Studies Listed Below

OR = (Odds of exposure among cases / Odds of exposure among controls)Variance – log(OR) = ((1/N1) + (1/N2) + (1/N3) + (1/N4))Weight (W) = 1 / variance

sum(weighti x ORi) 122.75ORMH = ---------------------------- OR = ---------------------- = 1.32

sum(weighti) 93.06

CASES CONTROLS Exposed Not exposed Exposed Not exposed OR Variance Weight W x ORStudy 1 32 72 111 445 1.78 0.0564 17.73 31.59Study 2 19 39 54 132 1.19 0.1044 9.58 11.41Study 3 12 20 16 44 1.65 0.2186 4.58 7.55Study 4 62 112 48 110 1.27 0.0550 18.19 23.07Study 5 43 72 51 74 0.87 0.0703 14.23 12.33Study 6 90 192 74 202 1.28 0.0348 28.75 36.79Totals 258 507 354 1007 93.06 122.75

Practice: Calculate the Mantel Haenszel Summary Odds Ratio for the Studies Listed Below

OR = (Odds of exposure among cases / Odds of exposure among controls)Variance – log(OR) = ((1/N1) + (1/N2) + (1/N3) + (1/N4))Weight (W) = 1 / variance

Across the 6 studies, the summary odds ratio is 1.32

Fixed Effect Method: Meta-AnalysisFixed Effect Method: Meta-Analysis

Since larger studies are more precise the smaller studies (hence smaller confidence intervals), the fixed effect approach gives more weight in the analysis to the larger studies.

The inference applies to only the studies included in the meta-analysis.

Random Effects: Meta-AnalysisRandom Effects: Meta-Analysis

--- The “within-study” and “between-study” variances are used as the weighting factor for each study.

Example: Dersimonian and Laird method:

sum(w*i x ln ORi)ln ORDL = ----------------------------

sum w*i

where w*i = 1 / [D + (1 / wi)]

Random Effects: Meta-AnalysisRandom Effects: Meta-Analysis

For the term in w*i

wi = 1 / variance (within-study variance)

[Q – (S – 1)] x sum wi (between-D = ------------------------------- study

[(sum wi)2 – sum (wi2)] variance)

Q = sum wi(ln ORi – ln ORMH)2

Random Effects: Meta-AnalysisRandom Effects: Meta-AnalysisThe addition of the between-variance term usually results in a more conservative estimate (larger confidence interval) than the fixed effects method.

Larger studies AND those with disparate results are given more weight in the analysis.

The inference applies to the research question at large, not just the studies included in the meta-analysis.

Fixed Effect

Random Effects

Note wider CI compared to fixed effect

Note differences in Relative Weights

Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Research Synthesis Methods 2010; 1:97-111.

SECTION 7.4SECTION 7.4Heterogeneity among Heterogeneity among

studies in meta-studies in meta-analysesanalyses

Learning Outcome:

Explain the importance of investigating heterogeneity among studies included

in a meta-analysis.

Heterogeneity Among StudiesHeterogeneity Among Studies

Statistical heterogeneity: Disparate results across studies comprising a common research question.

1) Statistical question is whether there is greater variation between the results of the studies than is compatible with the play of chance.

2) Tests of heterogeneity typically lack statistical power – hence they are often inconclusive.

More Recent Issues - HeterogeneityMore Recent Issues - HeterogeneityHeterogeneity:Petiti (2001)

Approaches to heterogeneity in meta-analysis

Higgins (2002)Quantifying heterogeneity in a meta-analysis

Huedo-Medina (2006)Assessing heterogeneity in meta-analysis

Lau (1998)Summing up evidence: one answer is not always enough

Investigating HeterogeneityInvestigating Heterogeneity

1) Qualitative: Compare effect estimates of component studies by respective design and methodological characteristics:

• Year of publication• Type/dose of intervention• Outcome assessment• Length and number of follow-up intervals• Study inclusion/exclusion criteria• Study size

Investigating HeterogeneityInvestigating Heterogeneity2) Quantitative: Investigate variation in effect

estimates by statistical means:

• “Meta” regression analysis

--- variation in parameter of interest serves as the dependent variable

--- treatment intervention or factor of interest is included in the model with other study variables suspected to contributing of heterogeneity

Scenario:Scenario: Assume among 10 randomizedcontrolled trials, there is evidence of statistical heterogeneity. The trials have the following features:

1) All address the same basic research question (e.g. is treatment A better than treatment B?)2) They all have sufficiently large sample size.3) They are all of reasonably good quality (e.g. no obvious problems of validity).

What are some of the possible sources of Heterogeneity seen between the 10 trials?

1. Patient Population Heterogeneity:

• Age, gender, race, etc.• Baseline medical history• Baseline inclusion/exclusion criteria

2. Intervention Heterogeneity:

• Therapeutic improvements over time• Individual versus group therapy

3. Outcome Measurement Heterogeneity:

• Study endpoints ascertained using different criteria/measures

4. Study Protocol Heterogeneity:

• Requirements for patient follow-up(e.g. length, number of protocol specified visits)

RightCoronary

Artery

Left Main

LAD LCx

SOURCE OF STATISTICAL HETEROGENEITY - EXAMPLE

TRIALS COMPARING ANGIOPLASTY VS. BYPASS SURGERY -- SELECTED CATEGORIES OF BASELINE EXCLUSION CRITERIA

TRIAL

Vessel Disease

Ejection Fraction

Prior MI

Age

Unstable Angina

CABRI

Single

Left Main Severe Triple

< 35%

Yes - w/i 10 days/overt

cardiac failure

> 76

-----

RITA

Left Main

-----

-----

-----

-----

EAST

Single

Left Main w/ >30% stenosis

< 25%

Yes

w/i 5 days

-----

-----

GABI

Single

Left Main w/ >30% stenosis

-----

Yes

w/i 4 weeks

> 75

-----

MASS

All except single w/at least 80% proximal

LAD stenosis

LV

dysfunct- ion

Yes

-----

Yes

Lausanne

All except single w/at least 50% proximal

LAD stenosis

< 50%

Yes

(Q-wave anterior)

-----

Yes

ERACI

Single

Severe left main Severe triple

< 35%

Yes

evolving acute MI

-----

-----

BARI

Single

Left Main w/>50% stenosis

-----

Yes - acute req. emerg.

revasc.

> 80

Yes

Testing for Statistical Heterogeneity

Q Test (Cochran, 1954)•Sum squared deviations of each study effect estimate from overall

effect estimate, weighting contribution of each study by inverse of variance

•H0: Homogeneity of effect sizes; Q statistic follows χ2

distribution with k-1 (#studies) degrees of freedom•Q statistic poor power to detect true heterogeneity among studies when meta-analysis includes small number of studies.•Non-significant result is often inconclusive

Q = wi (Ti – T)2

Testing for Statistical HeterogeneityQ = wi (Ti – T)2

For Odds ratio, the effect size (T), is in the log scale – ln(OR)

CASES CONTROLS

Exposed Not exposed Exposed Not exposed OR ln(OR) Variance Weight W x OR (Ti – T)2 Qi

Study 1 191 295 141 312 1.43 0.3595 0.0189 52.85 75.71 0.0606 3.2003Study 2 21 97 19 92 1.05 0.0472 0.1214 8.24 8.63 0.0044 0.0362Study 3 75 165 68 139 0.93 -0.0735 0.0413 24.22 22.50 0.0349 0.8463Study 4 99 132 87 134 1.16 0.1442 0.0366 27.30 31.53 0.0009 0.0259Study 5 74 607 111 735 0.81 -0.2141 0.0255 39.17 31.62 0.1073 4.2031

Totals 460 1296 426 1412 151.76 170.00 8.31 Summary OR 1.12 0.1135

OR = (Odds of exposure among cases / Odds of exposure among controls)Ln(OR) = natural logarithm of odds ratioVariance – log(OR) = ((1/N1) + (1/N2) + (1/N3) + (1/N4))Weight (W) = 1 / variance

sum(weighti x ORi)Summary OR(T) = ----------------------------

sum(weighti)

Critical value for χ2 at p=0.05= 9.49 (5-1 = 4 d.f. = see table 3 of textbook)Even though χ2 of 8.31 < 9.49, individual OR’s look different…….

Practice: Testing for Statistical HeterogeneityQ = wi (Ti – T)2 For Odds ratio, the effect size (T), is in the log scale – ln(OR)


sum(weighti x ORi)Summary OR(T) = -------------------------- = ------- =

sum(weighti)

Critical value for χ2 at p=0.05= _________

Conclusion: ________________________________

CASES CONTROLS


Study 1 38 62 44 55Study 2 18 54 24 48Study 3 50 60 30 62Study 4 29 48 32 42

Totals 135 224 130 207 Summary OR

Q = _______

Practice: Testing for Statistical HeterogeneityQ = wi (Ti – T)2 For Odds ratio, the effect size (T), is in the log scale – ln(OR)


sum(weighti x ORi) 41.25Summary OR(T) = -------------------------- = ------- = 1.03

sum(weighti) 39.99

Critical value for χ2 at p=0.05= 7.81 (4-1 = 4 d.f. = see table 3 of textbook)χ2 of 6.13 < 7.81Conclusion: Do not conclude the studies differ in effect size.

CASES CONTROLS


Study 1 38 62 44 55 0.77 -0.2664 0.0834 12.00 9.19 0.0885 1.0618Study 2 18 54 24 48 0.67 -0.4055 0.1366 7.32 4.88 0.1906 1.3955Study 3 50 60 30 62 1.72 0.5436 0.0861 11.61 20.00 0.2627 3.0498Study 4 29 48 32 42 0.79 -0.2320 0.1104 9.06 7.18 0.0692 0.6270


Q = 6.13

Testing for Statistical HeterogeneityI2 Index (Higgins and Thompson, 2002)•Percentage of total variability in a set of effect sizes due to true heterogeneity (i.e. between-study variability)•Example: I2 = 50 means half of total variability in effect sizes is not due to sampling error, but by true heterogeneity between studies.•classification of I2 values (Higgins and Thompson, 2002)

I2 ~ 25%: “Low” heterogeneityI2 ~ 50%: “Medium” heterogeneityI2 ~ 75%: “High” heterogeneity

I2 = ((Q - (n-1)) / Q) x 100where n = number of studies (i.e. d.f for Q)

Testing for Statistical HeterogeneityI2 Index (Higgins and Thompson, 2002)

I2 = ((Q - (n-1)) / Q) x 100where n = number of studies (i.e. d.f for Q)

CASES CONTROLS


Study 1 38 62 44 55 0.77 -0.2664 0.0834 12.00 9.19 0.0885 1.0618Study 2 18 54 24 48 0.67 -0.4055 0.1366 7.32 4.88 0.1906 1.3955Study 3 50 60 30 62 1.72 0.5436 0.0861 11.61 20.00 0.2627 3.0498Study 4 29 48 32 42 0.79 -0.2320 0.1104 9.06 7.18 0.0692 0.6270


I2 = ((6.13 - (4-1)) / 6.13) x 100 = 51.09“medium heterogeneity”

SECTION 7.5SECTION 7.5Example of Example of

Meta-AnalysisMeta-Analysisand Conclusionsand Conclusions

Learning Outcome:

Recognize methodological considerations in interpreting results of meta-analyses

Example: Meta-analysis of St. John’s Example: Meta-analysis of St. John’s Wort for Depression (Kim et al. – 1999)Wort for Depression (Kim et al. – 1999)Criteria for Trial Inclusions:

1. Any language 2. Blinded controlled studies of St. John’s Wort versus

standard anti-depressant medications

3. Subjects from similar SES backgrounds with depressive disorders defined by either ICD 10, DSM-IIIR or DSM-IV criteria

4. Clinical outcomes measured with the Hamilton

Depression Scale

5. Varying dosages of St. John’s Wort

Example: Meta-analysis of St. John’s Wort Example: Meta-analysis of St. John’s Wort for Depression (Kim et al. – 1999)for Depression (Kim et al. – 1999)

Study N RR 95% C.I.1 165 0.79 0.63 – 1.002 102 0.92 0.67 – 1.263 139 1.15 0.87 – 1.534 80 1.14 0.90 – 1.45

MA – Fixed 486 1.11 0.92 – 1.29MA - Random 486 0.98 0.67 – 1.28

Meta-analysis of St. John’s WortMeta-analysis of St. John’s WortFour controlled trials versus anti-depressant treatment:

Study

N Response Rate RR (95% CI)

Dropout Rate RR (95% CI)

Side Effect Rate RR (95% CI)

1 165 0.79 0.78 0.57 (0.63, 1.00) (0.48, 1.29) (0.42, 0.79) 2 102 0.92 0.78 0.72 (0.67, 1.26) (0.31, 1.93) (0.40, 1.32) 3 139 1.15 0.26 0.51 (0.87, 1.53) (0.03, 2.31) (0.27, 0.96) 4 80 1.14 1.00 0.41 (0.90, 1.45) (0.15, 6.76) (0.21, 0.78) Fixed Rand. Fixed Rand. Fixed Rand.

All 486 1.11 0.98 0.65 --- 0.58 --- (0.92, (0.67 (0.39, --- (0.47, --- 1.29) 1.28) 0.94) --- 0.77) ---

Example: Meta-analysis of St. John’s Wort Example: Meta-analysis of St. John’s Wort for Depression (Kim et al. – 1999)for Depression (Kim et al. – 1999)

In this example, both the fixed effect result and random effects result suggest that St. John’s Wort provides similar treatment effect to conventional anti-depressant medication.

However, the meta-analysis also found that St. John’s Wort was associated with a lower dropout rate (RR = 0.65, 95% CI: 0.39 – 1.94) and a lower side-effect rate (RR = 0.58, 95% CI: 0.47 – 0.77).

CTT meta-analysis: Effects of MORE vs. LESS STATIN on MAJOR VASCULAR EVENTS

0.5 0.75 1 1.25 1.5

OutcomeEvents (%)

Treatment Control RR (CI)

Control betterTreatment better

Non fatal MICHD deathAny major coronary event

CABGPTCAUnspecifiedAny coronary revascularisation

Haemorrhagic strokePresumed ischaemic strokeAny stroke

Any major vascular event

1175 (5.9%) 730 (3.7%)

1804 (9.1%)

637 (3.2%)1167 (5.9%) 322 (1.6%)

2126 (10.7%)

63 (0.3%) 509 (2.6%) 572 (2.9%)

3777 (19.0%)

1380 (7.0%) 804 (4.1%)

2072 (10.5%)

731 (3.7%)1508 (7.6%) 382 (1.9%)

2621 (13.2%)

57 (0.3%) 606 (3.1%) 663 (3.4%)

4376 (22.1%)

0.85 (0.76 - 0.94)0.91 (0.80 - 1.03)0.86 (0.81 - 0.92)

0.86 (0.75 - 0.99)0.76 (0.69 - 0.84)0.82 (0.68 - 1.00)0.80 (0.75 - 0.85)

1.10 (0.69 - 1.77)0.84 (0.72 - 0.98)0.86 (0.77 - 0.96)

0.85 (0.81 - 0.89)

99% or 95% CI

“Beneficial” Effects of High Dose Statins(likely overestimated for general population)

1. RCTs often enroll “vanilla” patients without high comorbidities or other factors related to safety and efficacy.

2. Compliance with statin therapy in the general population is notoriously low.

3. Metabolic syndrome is key target today – statins have little to no impact of HDL and triglycerides.

4. Life insurance companies would basically “care less” about total or LDL cholesterol.

Risk ratio

0.5 1 2

Combined

Lee, 2011

Currie, 2009

Libby, 2009

Chung, 2008

Yang, 2004

Standard figure used in meta-analyses:(to depict size and relative risk

of individual studies)

Analytical RecommendationsAnalytical Recommendations

1) Consider multiple methods of analysis and compare results across methods.

2) Conduct a sensitivity analysis to assess the influence of each individual study on the overall results.

3) Explore sources of heterogeneity both qualitatively and quantitatively.

Meta-Analysis – Concluding RemarksMeta-Analysis – Concluding Remarks

4) There is a lack of consensus on the appropriate statistical techniques to use, particularly when evidence of heterogeneity exists between studies.

5) When heterogeneity is present, calculation of an “average” effect may be of dubious validity (to what study population do the results apply?)

Documents

Statistics for clinicians