張偉豪

三星統計服務有限公司 執行長

SEM 亞洲一哥

統計黑傑克

版次:20151225

If you can't explain it simply, you don't understand it well enough.

Albert Einstein

What is Meta-Analysis

Software of Meta-Analysis

How to plan a Meta-Analysis

RCT, Cohort study or Case study

Effect size

Risk ratio(RR) vs. Odds ratio(OR)

Fix effect vs. Random effect

Heterogeneity test

Publication bias

Reporting the results of a Meta-Analysis

Outline

Books

Meta-analysis is a quantitativeapproach for systematically combining results of previous research to arrive at conclusions about the body of research.

What is Meta-Analysis

Quantitative : numbers

Systematic : methodical

Combining: putting together (mean and variance)

Previous research: what's already done

Conclusions: new knowledge

What is Meta-Analysis

When individual trials or studies’ sample sizes

are too small to give reliable answers.

When large trials or studies are impractical or impossible

Potentially lead to more timely introduction of effective treatment

When there have been many trials or studies showing small effects may be important.

Avoid institutional review board (IRB) censor.

7

Advantages of Use of Meta-Analysis

Hierarchy of evidence

Meta-Analysis

Systematic Review

Randomized Controlled Trial

Cohort studies

Case Control studies

Case Series/Case Reports

Animal research

Individual studies

Collecting similarity studies from previous research.

Effect sizes (ES)

Transform data (analysis results) into effect size to reflect the magnitude of treatment effect or the strength of a relationship between two variables.

Precision

The effect size for each study is bounded by as confidence interval (CI), reflect the precision of effect size.

How a Meta-Analysis work

Study weight

Ideal studies (sample size are larger) are assigned relatively high weight.

P-value

A p-value for a test of the null hypothesis.

If p<0.05 reject null hypothesis.

The summary effect

Summary the effect size from all studies, including mean ES (fix effect), CI, weight, p-value, ES heterogeneity, random effect, publication bias etc..

How a Meta-Analysis work

CMA is able to accept data in more

than 100 formats and allows the user to mix and match formats in the same analysis.

CMA is able to perform fixed-effect and random-effects analyses. They all report the key statistics, such as the summary effect and confidence intervals, measures of heterogeneity (T2, Q, I2)

CMA allow the researcher to automate the process, performing the analysis repeatedly and removing a different study on each pass.

Why use Comprehensive Meta-Analysis (CMA)

Why use Comprehensive Meta-Analysis (CMA)

CMA allows the user to define a hierarchical structure and then offers the user a set of options including the option to create a synthetic variable based on some (or all) the outcomes, or to work with each outcome separately.

CMA offer a full set of tools to assess publication bias.

CMA support 50 formats for data entry, all of the basic computational options, and high-resolution forest plots.

Define the Research Question

Perform the literature

search

Determine eligibility of

studies

Extract the data from

studies

Analyze the data in the

study statistically

Examine heterogeneity

Assess publication

bias

Interpret and Report the

results

Steps in a meta-analysis

Eight Steps of Meta Analysis

1. Define the Research Question

2. Perform the literature search

3. Determine eligibility of studies

Inclusion: which ones to keep

Exclusion: which ones to throw out

4. Extract the data from studies

5. Analyze the data in the study statistically

6. Examine heterogeneity

7. Assess publication bias

8. Interpret and Report the results

How to plan a Meta-Analysis

In patients with coronary artery disease (CAD)

does vitamin E supplementation decrease the risk of death?

Patients digest Carotenoids will decrease the chance of lung cancer happen.

Define the Research Question

Define the Research Question

Potentially relevant references identified after liberal screening of the electronic search (n=#)

Excluded by Title/Abstract (n=#) List the reasons

Articles retrieved for more detailed evaluation (n=#)

Articles excluded after evaluation of full text (n=#) List the reasons

Relevant studies included in the meta-analysis (n=#)

Flow Diagram of Study Selection Process

Be methodical: plan first

List of popular databases to search

Pubmed/Medline/Embase

List every possible database you may search.

Other strategies you may adopt

Hand search (go to the library...)

Personal references, and emails

web, eg. Google scholar (http://scholar.google.com)

Identify your studies

Perform the literature

search

Let's say we want to know that passive smoking really cause lung cancer.

How should we set up a search strategy?

What is the key words?

“Smoking” or/and “lung cancer”

Passive/Second hand smoking

Active smoking

Air pollution

Lung disease

Search key word

“passive smoking” OR “second hand smoking”[text word] OR lung cancer produces ALL articles that contain EITHERsmoking OR lung cancer to get a lot of articles.

“Passive smoking” AND “lung cancer” will capture only those subsets that have BOTH smoking AND lung cancer reduce the articles.

The Search

Cannot include all studies

Keep the ones with

high levels of evidence

good quality

Usually, MA done with RCTs

Case series, and case reports definitely out

Selection problems are major problems

Keep some, throw out others

Determine eligibility of

studies

Are the studies similar enough to combine?

Can I combine studies with different designs?

Experiential VS. Observational

Studies that used independent groups, paired groups, clustered groups

Can I combine studies that report results in different ways?

How many studies are enough to carry out a meta-analysis?

When Does it Make Sense to Perform a Meta-Analysis?

Randomized Controlled Trials (RCTs)

• The cases who was random select from population

• Belong to experimental study

• Exposure didn’t naturally

• Blind randomized trial

RCT, Cohort study or Case study

Cohort Study is any group of people who are

linked in some way and followed over time.

Belong to observational study

Expose naturally in nature world

Prospective Cohort study

Retrospective Cohort study

Time Series Study

Case Control

examine associations between disease/disorder/health issue and one or more risk factors

RCT, Cohort study or Case study

Question: Will smoke behavior cause lung cancer?

Prospective Cohort study

Causality research

Find multiple consequence

Retrospective Cohort study

Find multiple causes may cause diseases

Outcome is determined before exposure status

No need huge sample size

Cohort study

Researchers use existing records to identify

people with a certain health problem (“cases”) and a similar group without the problem (“controls”).

Similar retrospective Cohort study

Example: To learn whether a certain drug causes birth defects, one might collect data about children with defects (cases) and about those without defects (controls).

The data are compared to see whether cases are more likely than controls to have mothers who took the drug during pregnancy.

Case control study

Create a spreadsheet (Excel, or OpenOffice Calc)

For each study, create the following columns:

name of the study

name of the author, year published

number of participants who received intervention

number of participants who were in control

number who developed outcomes in intervention

number who developed outcomes in control

How to Abstract Data

Extract the data from

studies

Spreadsheet Data for Strepto Study

We created seven columns

trial: trial identity code

trialname: name of trial

year: year of the study

pop1: study population

deaths1: deaths in study

pop0: control population

deaths0: deaths in control

There are 22 studies to do our meta analysis

Data entry

The properties of effect size in a

meta-analysis

be comparable across studies (standardization)

represent magnitude & direction of the relationship

be independent of sample size

Effect size

The ES makes meta-analysis possible

The ES encodes the selected research findings on a numeric scale

There are many different types of ES measures, each suited to different research situations

Each ES type may also have multiple methods of computation

Effect size (ES)

Standardized mean difference

Group contrast research

Cohen’s d = 02, 0.5, and 0.8 as a small, medium, and large effect size

Output is continuous.

Odds-ratio

Group contrast research

OR = 1.68, 3.47, and 6.71 as a small, medium, and large effect size

Output is dichotomous.

Correlation coefficient

Association between variables research31

Different Types of Effect Sizes

Odds definition The probability of event divided by the probability of

the alternative.

Odds = p/1-p

𝑶𝑹 =𝑶𝒅𝒅𝒔 𝒐𝒇𝒆𝒙𝒑𝒐𝒔𝒖𝒓𝒆 𝒊𝒏 𝒕𝒉𝒐𝒔𝒆 𝒘𝒊𝒕𝒉 𝒅𝒊𝒔𝒆𝒂𝒔𝒆

𝑶𝒅𝒅𝒔 𝒐𝒇𝒆𝒙𝒑𝒐𝒔𝒖𝒓𝒆 𝒊𝒏 𝒕𝒉𝒐𝒔𝒆𝒘𝒊𝒕𝒉𝒐𝒖𝒕 𝒅𝒊𝒔𝒆𝒂𝒔𝒆

Interpretation

OR>1 Increase frequency of exposure among cases

OR=1 No change in frequency of exposure

OR<1 Decrease frequency of exposure

An OR about 2 is usually important

Odds ratio(OR)

Definition of RR

The proportion experiencing the event in one group divided by the proportion experiencing it in the other.

RR = p1/p2

𝑹𝑹 =𝑰𝒏𝒄𝒊𝒅 𝒐𝒇 𝒐𝒖𝒕𝒄𝒐𝒎𝒆 𝒘𝒊𝒕𝒉 𝒆𝒙𝒑𝒐𝒔𝒖𝒓𝒆

𝑰𝒏𝒄𝒊𝒅 𝒐𝒇 𝒐𝒖𝒕𝒄𝒐𝒎𝒘𝒊𝒕𝒉𝒐𝒖𝒕 𝒆𝒙𝒑𝒐𝒔𝒖𝒓𝒆

RR is suitable Cohort studies

Interpretation

RR>1 Increase risk of outcome

RR=1 No risk of outcome

RR<1 Reduce risk of outcome

Risk ratio(RR)

Fix effect

Assumes that all studies are estimating the same true effect

Variability only from sampling of people within each study

Precision depends mainly on study size

Fix vs. Random effect

Random effect

Studies allowed to have different underlying or true effects

Allows variation between studies as well as within studies

Fix vs. Random effect

Random effects generally yield larger variances

and CI

Why? Incorporate

If heterogeneity between studies is large between variance, will dominate the weights and all studies will be weighted more equally

Model weight for large studies less in random vs. fixed effects model

Fix vs. Random effect

Statistical test for heterogeneity

Visual inspection/Graphical approach

Forest plot

Meta-regression

Unit of regression: study

Dependent variable: study-specific effect estimate

Independent variables: study-specific characteristics (e.g., study design, geographic location, length of follow-up)

37

Examining Heterogeneity

Examine heterogeneity

Different study designs

Different incidence rates among unexposed

Different length of follow-up

Different distributions of effect modifiers

Different statistical methods/models used

Different sources of bias

Study quality

Sources of Between Study Heterogeneity

39

Examining Forest Plot for Heterogeneity

The I2 statistic describes the percentage of variation across studies that is due to heterogeneity rather than chance. .

I2 statistic value is a standardized value.I2 statistic (between variance/total variance)

1. 0% ~ 40%: heterogeneity might not be important;

2. 30% ~ 60%: may represent moderate heterogeneity;

3. 50% ~ 90%: may represent substantial heterogeneity;

4. 75% ~ 100%: considerable heterogeneity.

Heterogeneity test

In traditional (fixed-effects) meta-analysis heterogeneity test using the Q statistic.

The test has low power, so you use p<0.10 rather than p<0.05.

If p<0.10, you exclude "outlier" studies and re-test, until p>0.10.

When p>0.10, you declare the effect homogeneous.

Heterogeneity test

Strategies for addressing heterogeneity

Check again that the data are correct

Do not do a meta-analysis

Explore heterogeneity (subgroup analysis, meta-regression)

Ignore heterogeneity (there is no an intervention effect but a distribution of intervention effects)

Perform a random-effects meta-analysis (when heterogeneity cannot be explained)

Change the effect measure (different scales in different studies)

Exclude studies (outlying studies)

Sensitivity analysis

Sensitivity analysis have been used to examine the effects of studies identified as being aberrant concerning conduct or result, or being highly influential in the analysis.

One study removed meta-analysis

Cumulative analysis

how the results would change if one study (or a

set of studies) was removed from the analysis.

One study removed meta-analysis

A cumulative meta-analysis is performed first

with one study, then with two studies, and so on, until all relevant studies have been included in the analysis.

A cumulative analysis entering the larger studies at the top and adding the smaller studies at the bottom, sorted by sample size or precision.

A benefit of the cumulative analysis is that it displays not only if there is a shift in effect size, but also the magnitude of the shift.

Cumulative analysis

What is Meta-Analysis bias?

Can bias the results of a meta-analysis toward a positive finding

Can evaluate publication bias graphically (funnel plot) or through statistical analysis

Test of Publication Bias

Assess publication bias

Outcome reporting bias

Significant outcomes are more likely to be reported than non-significant outcomes.

Should unpublished data be included in systemic review?

Pre-specified inclusion (quality) criteria are recommended.

Database Bias

No single database is likely to contain all published

studies on a given subject.”

Where Can Publication Bias Occur?

Publication Bias

selective publication of articles that show positive

treatment of effects and statistical significance.

English-language (duplication) bias

Studies with statistically significant results are more likely

to be published in English

Citation bias

occurs when studies with significant or positive results are

referenced in other publications, compared with studies

with inconclusive or negative findings

Meta-Analysis bias

Funnel plot

Rosenthal’s Fail-safe N

Orwin’s Fail-safe N

Duval and Tweedie’s Trim & Fill

rank correlation (P>0.05)

Regression

Methods for addressing publication bias

Funnel plot has several caveats:

1. funnel plot may yield a very different picture depending on the index used in the analysis (risk difference versus risk ratio).

2. Funnel plot makes sense only if there is a reasonable amount of dispersion in the sample sizes and a reasonable number of studies.

3. even when these criteria are met, the tests tend to have lower power.

Funnel plot

The absence of a significant correlation or

regression cannot be taken as evidence of symmetry.

To solve these problems, we use

Rosenthal’s Fail-safe N

Orwin’s Fail-safe N

Duval and Tweedie’s Trim and Fill

Funnel plot

What is our best estimate of the unbiased effect

size?

Trim and fill procedure will tell you the answer, the method separate into trim and fill two steps.

Trim & fill

Trim first

remove the most extreme small studies from the positive side of the funnel plot, re-computing the effect size at each iteration until the funnel plot is symmetric about the (new) effect size.

yields the adjusted effect size(unbiased summate ES).

Fill follow

adds the original studies back into the analysis, and imputes a mirror image for each.

to correct the ES variance.

Trim and Fill procedure

The fail-safe N (Rosenthal, 1991) determines the

number of studies with an effect size of zero needed to lower the observed effect size to a specified (criterion) level.

The fail-safe N actually compute how many missing studies we would need to retrieve and incorporate in the analysis before the p-value became nonsignificant..

Rosenthal’s Fail-safe N (File drawer analysis)

the Fail-safe N is 38, suggesting that there would

need to be nearly 40 studies with a mean risk ratio of 1.0 added to the analysis, the research will become statistically nonsignificant.

Rosenthal’s Fail-safe N

Orwin’s method allows the researcher to

determine how many missing studies would bring the overall effect to a specified level other than zero.

it allows the researcher to specify the mean effect in the missing studies as some value other than zero.

Orwin’s Fail-safe N

Begg and Mazumdarrank correlation

Is there evidence of bias?

Egger’s regression

Combine data to arrive at a summary, 3 measures

Effect Size (Odds Ratio or Risk Ratio or Correlations)

Variance with 95% Confidence Interval

Test of heterogeneity

Two Graphs

Forest Plot

Funnel Plot

Examine why the studies are heterogeneous

Examine publication bias.

Reporting the results

Interpret and Report the

results

Meta-Analysis check list

Are the studies similar enough to combine?

There is no restriction on the similarity of studies Based on the types of participants, interventions, or exposures.

Can I combine studies with different designs?

Randomized trials versus observational studies

Studies that used independent groups, paired groups, clustered groups

Can I combine studies that report results in different ways?

When Does it Make Sense to Performa Meta-Analysis?

How many studies are enough to carry out a

meta-analysis?

Fix effect model

At least two studies, since a summary based on two or more studies yields a more precise estimate of the true effect than either study alone.

Random effect model

When Does it Make Sense to Performa Meta-Analysis?

One number cannot summarize a research field

The file drawer problem invalidates meta-analysis

Mixing apples and oranges

Garbage in, garbage out

Important studies are ignored

Meta-analysis can disagree with randomized trials

Meta-analyses are performed poorly

Criticisms of Meta-Analysis