Meta-Analysis_ Its Srengths and Limitations. 2013

CLEVELAND CL IN IC JOURNAL OF MEDICINE VOLUME 75 NUMBER 6 JUNE 2008 431

ESTEBAN WALKER, PhDDepartment of Quantitative HealthSciences, Cleveland Clinic

Meta-analysis:Its strengths and limitations

REVIEW

ABSTRACTNowadays, doctors face an overwhelming amount ofinformation, even in narrow areas of interest. In response,reviews designed to summarize the large volumes ofinformation are frequently published. When a review isdone systematically, following certain criteria, and theresults are pooled and analyzed quantitatively, it is calleda meta-analysis. A well-designed meta-analysis canprovide valuable information for researchers, policy-makers, and clinicians. However, there are many criticalcaveats in performing and interpreting them, and thusmany ways in which meta-analyses can yield misleadinginformation.

KEY POINTSMeta-analysis is an analytical technique designed tosummarize the results of multiple studies.

By combining studies, a meta-analysis increases thesample size and thus the power to study effects ofinterest.

There are many caveats in performing a valid meta-analysis, and in some cases a meta-analysis is notappropriate and the results can be misleading.

HE AMOUNT OF INFORMATION generatedin medical research is becoming over-

whelming, even for experienced researchers.New studies are constantly being published,and clinicians are finding it nearly impossibleto stay current, even in their own area of spe-cialty.

To help make sense of the information, weare seeing more and more review articles thatpool the results of multiple studies. When cer-tain principles are followed and the data arequantitatively analyzed, these reviews arecalled meta-analyses. A PubMed search of theword meta-analysis in the title yielded 1,473articles in the year 2007.

Combining available information to gen-erate an integrated result seems reasonableand can save a considerable amount ofresources. Nowadays, meta-analyses are beingused to design future research, to provide evi-dence in the regulatory process,1 and even tomodify clinical practice.

Meta-analysis is powerful but also contro-versialcontroversial because several condi-tions are critical to a sound meta-analysis, andsmall violations of those conditions can leadto misleading results. Summarizing largeamounts of varied information using a singlenumber is another controversial aspect ofmeta-analysis. Under scrutiny, some meta-analyses have been inappropriate, and theirconclusions not fully warranted.2,3

This article introduces the basic conceptsof meta-analysis and discusses its caveats, withthe aim of helping clinicians assess the meritsof the results. We will use several recent meta-analyses to illustrate the issues, including acontroversial one4 with potentially far-reach-ing consequences.

T

ADRIAN V. HERNANDEZ, MD, PhDDepartment of Quantitative Health Sciences,Cleveland Clinic

MICHAEL W. KATTAN, PhDDepartment of Quantitative Health Sciences,Cleveland ClinicCREDIT

CME

on November 11, 2013. For personal use only. All other uses require permission.www.ccjm.orgDownloaded from

432 CLEVELAND CL IN IC JOURNAL OF MEDICINE VOLUME 75 NUMBER 6 JUNE 2008

OBJECTIVES OF META-ANALYSIS

The main objectives of a meta-analysis are to: Summarize and integrate results from a

number of individual studies Analyze differences in the results among

studies Overcome small sample sizes of individual

studies to detect effects of interest, andanalyze end points that require larger sam-ple sizes

Increase precision in estimating effects Evaluate effects in subsets of patients Determine if new studies are needed to

further investigate an issue Generate new hypotheses for future studies.

These lofty objectives can only beachieved when the meta-analysis satisfactorilyaddresses certain critical issues, which we willdiscuss next.

CRITICAL ISSUESIN META-ANALYSIS DESIGN

Four critical issues need to be addressed in ameta-analysis: Identification and selection of studies Heterogeneity of results Availability of information Analysis of the data.

IDENTIFICATIONAND SELECTION OF STUDIES

The outcome of a meta-analysis depends onthe studies included. The critical aspect ofselecting studies to be included in a meta-analysis consists of two phases. The first is theidentification phase or literature search, inwhich potential studies are identified. In thesecond phase, further criteria are used to createa list of studies for inclusion. Three insidiousproblems plague this aspect of meta-analysis:publication bias and search bias in the identifi-cation phase, and selection bias in the selec-tion phase. These biases are discussed below.

Publication bias: Positive studiesare more likely to be printedSearches of databases such as PubMed orEmbase can yield long lists of studies. However,these databases include only studies that have

been published. Such searches are unlikely toyield a representative sample because studiesthat show a positive result (usually in favor ofa new treatment or against a well-establishedone) are more likely to be published than thosethat do not. This selective publication of stud-ies is called publication bias.

In a recent article, Turner et al5 analyzedthe publication status of studies of antidepres-sants. Based on studies registered with the USFood and Drug Administration (FDA), theyfound that 97% of the positive studies werepublished vs only 12% of the negative ones.Furthermore, when the nonpublished studieswere not included in the analysis, the positiveeffects of individual drugs increased between11% and 69%.

One reason for publication bias is thatdrug manufacturers are not generally interest-ed in publishing negative studies. Anothermay be that editors favor positive studiesbecause these are the ones that make theheadlines and give the publication visibility.In some medical areas, the exclusion of studiesconducted in non-English-speaking countriescan increase publication bias.6

To ameliorate the effect of publicationbias on the results of a meta-analysis, a seriouseffort should be made to identify unpublishedstudies. Identifying unpublished studies is eas-ier now, thanks to improved communicationbetween researchers worldwide, and thanks toregistries in which all the studies of a certaindisease or treatment are reported regardless ofthe result.

The National Institutes of Health main-tains a registry of all the studies it supports,and the FDA keeps a registry and database inwhich drug companies must register all trialsthey intend to use in applying for marketingapproval or a change in labeling. Banks ofpublished and unpublished trials supported bypharmaceutical companies are also available(eg, http://ctr.gsk.co.uk/welcome.asp). TheCochrane collaboration (www.cochrane.org/)keeps records of systematic reviews and meta-analyses of many diseases and procedures.

Search bias: Identifying relevant studiesEven in the ideal case that all relevant studieswere available (ie, no publication bias), afaulty search can miss some of them. In

META-ANALYSIS WALKER AND COLLEAGUES

Even smallviolations ofthe rules ofmeta-analysiscan lead tomisleadingresults


searching databases, much care should betaken to assure that the set of key words usedfor searching is as complete as possible. Thisstep is so critical that most recent meta-analy-ses include the list of key words used. Thesearch engine (eg, PubMed, Google) is alsocritical, affecting the type and number of stud-ies that are found.7 Small differences in searchstrategies can produce large differences in theset of studies found.8

Selection bias:Choosing the studies to be includedThe identification phase usually yields a longlist of potential studies, many of which are notdirectly relevant to the topic of the meta-analysis. This list is then subject to additionalcriteria to select the studies to be included.This critical step is also designed to reduce dif-ferences among studies, eliminate replicationof data or studies, and improve data quality,and thus enhance the validity of the results.

To reduce the possibility of selection biasin this phase, it is crucial for the criteria to beclearly defined and for the studies to be scoredby more than one researcher, with the finallist chosen by consensus.9,10 Frequently usedcriteria in this phase are in the areas of: Objectives Populations studied Study design (eg, experimental vs obser-

vational) Sample size Treatment (eg, type and dosage) Criteria for selection of controls Outcomes measured Quality of the data Analysis and reporting of results Accounting and reporting of attrition

rates Length of follow-up When the study was conducted.

The objective in this phase is to selectstudies that are as similar as possible withrespect to these criteria. It is a fact that evenwith careful selection, differences among stud-ies will remain. But when the dissimilaritiesare large it becomes hard to justify pooling theresults to obtain a unified conclusion.

In some cases, it is particularly difficult tofind similar studies,10,11 and sometimes thediscrepancies and low quality of the studies

can prevent a reasonable integration ofresults. In a systematic review of advancedlung cancer, Nicolucci et al12 decided not topool the results, in view of systematic quali-tative inadequacy of almost all trials and lackof consistency in the studies and their meth-ods. Marsoni et al13 came to a similar conclu-sion in attempting to summarize results inadvanced ovarian cancer.

Stratification is an effective way to dealwith inherent differences among studies andto improve the quality and usefulness of theconclusions. An added advantage to stratifica-tion is that insight can be gained by investi-gating discrepancies among strata.

There are many ways to create coherentsubgroups of studies. For example, studies canbe stratified according to their quality,assigned by certain scoring systems. Commonlyused systems award points on the basis of howpatients were selected and randomized, thetype of blinding, the dropout rate, the outcomemeasurement, and the type of analysis (eg,intention-to-treat). However, these criteria,and therefore the scores, are somewhat subjec-tive. Moher et al14 expand on this issue.

Large differences in sample sizes amongstudies are not uncommon and can causeproblems in the analysis. Depending on thetype of model used (see below), meta-analysescombine results based on the size of eachstudy, but when the studies vary significantlyin size, the large studies can still have anunduly large influence on the results.Stratifying by sample size is done sometimes toverify the stability of the results.4

On the other hand, the presence of dis-similarities among studies can have advan-tages by increasing the generalizability of theconclusions. Berlin and Colditz1 point outthat we gain strength in inference when therange of patient characteristics has beenbroadened by replicating findings in studieswith populations that vary in age range, geo-graphic region, severity of underlying illness,and the like.

Funnel plot: Detecting biases in theidentification and selection of studiesThe funnel plot is a technique used to inves-tigate the possibility of biases in the identifi-cation and selection phases. In a funnel plot


Exclusion ofnonpublishedstudiesincreasesselection bias



the size of the effect (defined as a measure ofthe difference between treatment and control)in each study is plotted on the horizontal axisagainst standard error15 or sample size9 on thevertical axis. If there are no biases, the graph

will tend to have a symmetrical funnel shapecentered in the average effect of the studies.When negative studies are missing, the graphshows lack of symmetry.

Funnel plots are appealing because theyare simple, but their objective is to detect acomplex effect, and they can be misleading.For example, lack of symmetry in a funnel plotcan also be caused by heterogeneity in thestudies.16 Another problem with funnel plotsis that they are difficult to interpret when thenumber of studies is small. In some cases, how-ever, the researcher may not have any optionbut to perform the analysis and report thepresence of bias.11

Dentali et al17 conducted a meta-analysisto study the effect of anticoagulant treatmentto prevent symptomatic venous thromboem-bolism in hospitalized patients. The conclu-sion was that the treatment was effective toprevent symptomatic pulmonary thromboem-bolism, with no significant increase in majorbleeding. FIGURE 1 shows the funnel plots for thetwo outcomes. Dentali et al17 concluded thatthe lack of symmetry in the top plot suggests alack of inclusion of small studies showing anincrease in the risk of pulmonary thromboem-bolism, and thus, bias. The bottom plot showsthe symmetry of the funnel plot for majorbleeding, suggesting absence of bias.

HETEROGENEITY OF RESULTS

In meta-analysis, heterogeneity refers to thedegree of dissimilarity in the results of individualstudies. In some cases, the dissimilarities inresults can be traced back to inherent differ-ences in the individual studies. In other situa-tions, however, causes for the dissimilaritiesmight not be easy to elucidate. In any case, asthe level of heterogeneity increases, the justifi-cation for an integrated result becomes more dif-ficult. A tool that is very effective to display thelevel of heterogeneity is the forest plot. In a for-est plot, the estimated effect of each study alongwith a line representing a confidence interval isdrawn. When the effects are similar, the confi-dence intervals overlap, and heterogeneity islow. The forest plot includes a reference line atthe point of no effect (eg, one for relative risksand odds ratios). When some effects lie onopposite sides of the reference line, it means


0.01

0.0

0.4

0.8

1.2

1.6

0.1 1

Stan

dar

d e

rro

rSt

and

ard

err

or

10 100

0.01

0.0

0.4

0.8

1.2

1.6

0.1 1 10 100

Relative risk (fixed)

Relative risk (fixed)

The funnel plot, a way to detectpossible selection bias

FIGURE 1. Top, a funnel plot of studies of anticoagu-lant prophylaxis that measured the outcome of symp-tomatic pulmonary embolism. The plot is asymmetri-cal, suggesting that small studies in which prophylaxiswas associated with an increased risk are missing.Bottom, a funnel plot of studies with the outcome ofmajor bleeding is symmetrical, suggesting absence ofselection bias.

DENTALI F, DOUKELIS D, GIANNI M, ET AL. META-ANALYSIS: ANTICOAGULANT PROPHYLAXIS TOPREVENT SYMPTOMATIC VENOUS THROMBOEMBOLISM IN HOSPITALIZED MEDICAL PATIENTS.

ANN INTERN MED 2007; 146:278288.


that the studies are contradictory and hetero-geneity is high. In such cases, the conclusions ofa meta-analysis are compromised.

The previously mentioned study byDentali et al17 presented several forest plotsthat display the level of heterogeneity of vari-ous outcomes. FIGURE 2 shows the forest plot forthe outcome of pulmonary embolism. Exceptfor one, the estimated effects are on the sameside of the unit line and the confidence inter-vals overlap to a large extent. This plot showsa low level of heterogeneity. FIGURE 3 shows theforest plot for major bleeding. Here the effectsare on both sides of the unit line, implying ahigh level of heterogeneity. Cochrans Q testis a statistical test used in conjunction withthe forest plot to determine the significance ofheterogeneity among studies.18

Gebski et al19 performed a meta-analysisof randomized controlled trials comparing thesurvival of patients with esophageal carcino-ma who received neoadjuvant chemotherapyvs those who underwent surgery alone. In onlyone of the eight studies included was neoadju-vant chemotherapy significantly beneficial.Three of the studies suggested that it washarmful, although the effects were not statisti-cally significant. The pooled result was mar-ginally significant in favor of the treatment (P= .05). This positive result was due largely tothe fact that the only study with a significant-ly positive result study also was, by far, thelargest (with 400 patients in each treatmentgroup, vs an average of 68 per treatment groupfor the rest). Even though the test for hetero-geneity was not significant, the marginal Pvalue and the differences in study size makethe results of this meta-analysis suspect.

AVAILABILITY OF INFORMATION

Most reports of individual studies includeonly summary results, such as means, stan-dard deviations, proportions, odds ratios, andrelative risks. Other than the possibility oferrors in reporting, the lack of informationcan severely limit the type of analyses andconclusions that can be reached in a meta-analysis. For example, lack of informationfrom individual studies can preclude thecomparison of effects in predetermined sub-groups of patients.


Study, Year

Belch et al, 1981

Dahan et al, 1986

Samama et al, 1999

Leizorovic et al, 2004

Cohen et al, 2006

Lederle et al, 2006

Total (95% CI)

Total events

Prophylaxis,n/n

Placebo,n/n

10.10.01

Favors treatment Favors control

0.001 10 100 1000

Mahe et al, 2005

0/50

1/132

0/291

5/1759

0/429

1/140

10/1230

0.20 (0.01 - 4.06)

0.33 (0.03 - 3.14)

0.14 (0.01 - 2.73)

1.24 (0.33 - 4.60)

0.59 (0.27 - 1.29)

0.33 (0.04 - 3.17)

0.43 (0.26 - 0.71)

0.09 (0.00 - 1.60)

2/50

3/131

3/288

Gardlund et al, 1996 3/5776 0.26 (0.07 - 0.91)12/5917

4/1740

5/420

3/140

20 49

17/1244

RR (fixed) (95% Cl)

RR (fixed) (95% Cl)

A low level of heterogeneity:Anticoagulation prevents pulmonaryembolism

FIGURE 2. A forest plot of studies of anticoagulantprophylaxis with the outcome of pulmonary embolism.All except one of the studies show a better outcomewith treatment than with placebo, indicating a lowlevel of heterogeneity among the studies.

DENTALI F, DOUKELIS D, GIANNI M, ET AL. META-ANALYSIS: ANTICOAGULANT PROPHYLAXIS TOPREVENT SYMPTOMATIC VENOUS THROMBOEMBOLISM IN HOSPITALIZED MEDICAL PATIENTS.

ANN INTERN MED 2007; 146:278288.

Study, Year

Dahan et al, 1986

Samama et al, 1999

Fraisse et al, 2000

Leizorovic et al, 2004

Cohen et al, 2006

Lederle et al, 2006

Total (95% CI)

Total events

Prophylaxis,n/n

Placebo,n/n

10.10.01

Favors treatment Favors control

0.001 10 100 1000

Mahe et al, 2005

1/132

6/360

6/108

8/1856

1/425

2/140

1/1230

0.33 (0.03 - 3.14)

1.51 (0.43 - 5.30)

2.09 (0.54 - 8.16)

16.95 (0.98 - 293.36)

0.34 (0.04 - 3.24)

0.40 (0.08 - 2.03)

1.32 (0.73 - 2.37)

0.97 (0.06 - 15.52)

3/131

4/362

3/113

0/1850

1/414

5/140

25 19

3/1244

RR (fixed) (95% Cl)

RR (fixed) (95% Cl)

A high level of heterogeneity:Does anticoagulation increasethe risk of major bleeding?

FIGURE 3. Risk of major bleeding in studies of anticoagu-lant prophylaxis. Some of the studies favor the controland others the treatment. This represents a high level ofheterogeneity

DENTALI F, DOUKELIS D, GIANNI M, ET AL. META-ANALYSIS: ANTICOAGULANT PROPHYLAXIS TOPREVENT SYMPTOMATIC VENOUS THROMBOEMBOLISM IN HOSPITALIZED MEDICAL PATIENTS. ANN

INTERN MED 2007; 146:278288.




The best scenario is when data at thepatient level are available. In such cases, theresearcher has great flexibility in the analysisof the information. Trivella et al20 performed ameta-analysis of the value of microvessel den-sity in predicting survival in non-small-celllung cancer. They obtained information onindividual patients by contacting researchcenters directly. The data allowed them tovary the cutoff point to classify microvesseldensity as high or low and to use statisticalmethods to ameliorate heterogeneity.

A frequent problem in meta-analysis isthe lack of uniformity in how outcomes aremeasured. In the study by Trivella et al,20 themicrovessel density was measured by twomethods. The microvessel density was a signif-icant prognostic factor when measured by oneof the methods, but not the other.

RANDOMIZED CONTROLLED TRIALSVS OBSERVATIONAL STUDIES

Some researchers believe that meta-analysesshould be conducted only on randomized con-trolled trials.3,21 Their reasoning is that meta-analyses should include only reasonably well-conducted studies to reduce the risk of a mis-leading conclusion. However, many importantdiseases can only be studied observationally. Ifthese studies have a certain level of quality,there is no technical reason not to includethem in a meta-analysis.

Gillum et al22 performed a meta-analysispublished in 2000 on the risk of ischemicstroke in users of oral contraceptives, based onobservational studies (since no randomizedtrials were available). Studies were identifiedand selected by multiple researchers usingstrict criteria to make sure that only studiesfulfilling certain standards were included. Of804 potentially relevant studies identified,only 16 were included in the final analysis. Afunnel plot showed no evidence of bias andthe level of heterogeneity was fairly low. Themeta-analysis result confirmed the results ofindividual studies, but the precision withwhich the effect was estimated was muchhigher. The overall relative risk of stroke inwomen taking oral contraceptives was 2.75,with a 95% confidence interval of 2.24 to3.38.

A more recent meta-analysis23 (publishedin 2004) on the same issue found no signifi-cant increase in the risk of ischemic strokewith the use of oral contraceptives. Gillumand Johnston24 suggest that the main reasonfor the discrepancy is the lower amount ofestrogen in newer oral contraceptives. Theyalso point out differences in the control groupsand study outcomes as reasons for the discrep-ancies between the two studies.

Bhutta et al25 performed a meta-analysisof case-control (observational) studies of theeffect of preterm birth on cognitive andbehavioral outcomes. Studies were includedonly if the children were evaluated after theirfifth birthday and the attrition rate was lessthan 30%. The studies were grouped accord-ing to criteria of quality devised specifically forcase-control studies. The high-quality studiestended to show a larger effect than the low-quality studies, but the difference was not sig-nificant. Seventeen studies were included, andall of them found that children born pretermhad lower cognitive scores; the difference wasstatistically significant in 15 of the studies. Asexpected, the meta-analysis confirmed thesefindings (95% confidence interval for the dif-ference 9.212.5). The number of patients(1,556 cases and 1,720 controls) in the meta-analysis allowed the researchers to concludefurther that the mean cognitive scores weredirectly proportional to the mean birth weight(R2 = 0.51, P < .001) and gestational age (R2= 0.49, P < .001).

ANALYSIS OF DATA

There are specific statistical techniques thatare used in meta-analysis to analyze and inte-grate the information. The data from the indi-vidual studies can be analyzed using either oftwo models: fixed effects or random effects.

The fixed-effects model assumes that thetreatment effect is the same across studies.This common effect is unknown, and the pur-pose of the analysis is to estimate it with moreprecision than in the individual studies.

The random-effects model, on the otherhand, assumes that the treatment effect is notthe same across studies. The goal is to estimatethe average effect in the studies.

In the fixed-effects model, the results of

Data-mininggreatlyincreases therisk of false-positive results


individual studies are pooled using weightsthat depend on the sample size of the study,whereas in the random-effects model eachstudy is weighted equally. Due to the hetero-geneity among studies, the random-effectsmodel yields wider confidence intervals.

Both models have pros and cons. In manycases, the assumption that the treatmenteffect is the same in all the studies is not ten-able, and the random-effects model is prefer-able. When the effect of interest is large, theresults of both models tend to agree, particu-larly when the studies are balanced (ie, theyhave a similar number of patients in the treat-ment group as in the control group) and thestudy sizes are similar. But when the effect issmall or when the level of heterogeneity ofthe studies is high, the result of the meta-analysis is likely to depend on the model used.In those cases, the analysis should be doneand presented using both models.

It is highly desirable for a meta-analysis toinclude a sensitivity analysis to determine therobustness of the results. Two common waysto perform sensitivity analysis are to analyzethe data using various methods and to presentthe results when some studies are removedfrom the analysis.26 If these actions cause seri-ous changes in the overall results, the credi-bility of the results is compromised.

The strength of meta-analysis is that, bypooling many studies, the effective sample sizeis greatly increased, and consequently morevariables and outcomes can be examined. Forexample, analysis in subsets of patients andregression analyses9 that could not be done inindividual trials can be performed in a meta-analysis.

A word of caution should be given withrespect to larger samples and the possibility ofmultiple analyses of the data in meta-analysis.Much care must be exercised when examin-ing the significance of effects that are notconsidered prior to the meta-analysis. Thetesting of effects suggested by the data andnot planned a priori (sometimes called data-mining) increases considerably the risk offalse-positive results. One common problemwith large samples is the temptation to per-form many so-called subgroup analyses inwhich subgroups of patients formed accordingto multiple baseline characteristics are com-

pared.27 The best way to minimize the possi-bility of false-positive results is to determinethe effects to be tested before the data are col-lected and analyzed. Another method is toadjust the P value according to the number ofanalyses performed. In general, post hocanalyses should be deemed exploratory, andthe reader should be made aware of this factin order to judge the validity of the conclu-sion.

META-ANALYSIS OF RARE EVENTS

Lately, meta-analysis has been used to analyzeoutcomes that are rare and that individualstudies were not designed to test. In general,the sample size of individual studies providesinadequate power to test rare outcomes.Adverse events are prime examples of impor-tant rare outcomes that are not always formal-ly analyzed statistically. The problem in theanalysis of adverse events is their low inci-dence. Paucity of events causes serious prob-lems in any statistical analysis (see Shuster etal28). The reason is that, with rare events,small changes in the data can cause dramaticchanges in the results. This problem can per-sist even after pooling data from many studies.Instability of results is also exacerbated by theuse of relative measures (eg, relative risk andodds ratio) instead of absolute measures of risk(eg, risk difference).

In a controversial meta-analysis, Nissenand Wolski4 combined 42 studies to examinethe effect of rosiglitazone (Avandia) on therisk of myocardial infarction and death fromcardiovascular causes. The overall estimatedincidence of myocardial infarction in thetreatment groups was 0.006 (86/14,376), or 6in 1,000. Furthermore, 4 studies did not haveany occurrences in either group, and 2 of the42 studies accounted for 28.4% of the patientsin the study.

Using a fixed-effect model, the odds ratiowas 1.42, ie, the odds of myocardial infarctionwas 42% higher in patients using rosiglita-zone, and the difference was statistically sig-nificant (95% confidence interval 1.031.98).Given the low frequency of myocardial infarc-tion, this translates into an increase of only1.78 myocardial infarctions per 1,000 patients(from 4.22 to 6 per 1,000). Furthermore,


With rareeffects, evena smalldifferencecan seem large


REFERENCES1. Berlin JA, Colditz GA. The role of meta-analysis in the regulatory process

for food, drugs, and devices. JAMA 1999; 281:830834.2. Bailar JC. The promise and problems of meta-analysis. N Engl J Med

1997; 337:559561.3. Simon R. Meta-analysis of clinical trials: opportunities and limitations. In:

Stangl DK, Berry DA, editors. Meta-Analysis in Medicine and HealthPolicy. New York: Marcel Dekker, 2000.

4. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardialinfarction and death from cardiovascular causes. N Engl J Med 2007;356:24572471.

5. Turner EH, Matthews AM, Linardatos E, et al. Selective publication ofantidepressant trials and its influence on apparent efficacy. N Engl J Med2008; 358:252260.

6. LeLorier J, Grgoire G, Benhaddad A, Lapierre J, Derderian F.Discrepancies between meta-analyses and subsequent large randomized,


when the data were analyzed using othermethods or if the two large studies wereremoved, the effect became nonsignificant.29

Nissen and Wolskis study4 is valuable andraises an important issue. However, the med-ical community would have been better servedif a sensitivity analysis had been presented tohighlight the fragility of the conclusions.

META-ANALYSIS VS LARGE RANDOMIZEDCONTROLLED TRIALS

There is debate about how meta-analysescompare with large randomized controlled tri-als. In situations where a meta-analysis and asubsequent large randomized controlled trialare available, discrepancies are not uncom-mon.

LeLorier et al6 compared the results of19 meta-analyses and 12 subsequent largerandomized controlled trials on the sametopics. In 5 (12%) of the 40 outcomes stud-ied, the results of the trials were significant-ly different than those of the meta-analysis.The authors mentioned publication bias,study heterogeneity, and differences in pop-ulations as plausible explanations for thedisagreements. However, they correctlycommented: this does not appear to be alarge percentage, since a divergence in 5percent of cases would be expected on thebasis of chance alone.6

A key reason for discrepancies is thatmeta-analyses are based on heterogeneous,often small studies. The results of a meta-analysis can be generalized to a target popula-tion similar to the target population in each ofthe studies. The patients in the individualstudies can be substantially different withrespect to diagnostic criteria, comorbidities,severity of disease, geographic region, and thetime when the trial was conducted, amongother factors. On the other hand, even in alarge randomized controlled trial, the targetpopulation is necessarily more limited. These

differences can explain many of the disagree-ments in the results.

A large, well-designed, randomized con-trolled trial is considered the gold standard inthe sense that it provides the most reliableinformation on the specific target populationfrom which the sample was drawn. Withinthat population the results of a randomizedcontrolled trial supersede those of a meta-analysis. However, a well conducted meta-analysis can provide complementary informa-tion that is valuable to a researcher, clinician,or policy-maker.

CONCLUSION

Like many other statistical techniques,meta-analysis is a powerful tool when usedjudiciously; however, there are manycaveats in its application. Clearly, meta-analysis has an important role in medicalresearch, public policy, and clinical prac-tice. Its use and value will likely increase,given the amount of new knowledge, thespeed at which it is being created, and theavailability of specialized software for per-forming it.30

A meta-analysis needs to fulfill several keyrequirements to ensure the validity of itsresults: Well-defined objectives, including precise

definitions of clinical variables and out-comes

An appropriate and well-documentedstudy identification and selection strategy

Evaluation of bias in the identificationand selection of studies

Description and evaluation of hetero-geneity

Justification of data analytic techniques Use of sensitivity analysis.

It is imperative that researchers, policy-makers, and clinicians be able to criticallyassess the value and reliability of the conclu-sions of meta-analyses.


Randomizedtrials are thegold standard,but meta-analysesprovidevaluablecomplementaryinformation



controlled trials. N Engl J Med 1997; 337:536542.7. Steinbrook R. Searching for the right searchreaching the medical liter-

ature. N Engl J Med 2006; 354:47.8. Dickersin K, Scherer R, Lefebvre C. Systematic reviews: identifying rele-

vant studies for systematic reviews. BMJ 1994; 309:12861291.9. De Luca G, Suryapranta H, Stone GW, et al. Coronary stenting versus bal-

loon angioplasty for acute myocardial infarction: a meta-regressionanalysis of randomized trials. Int J Cardiol 2007;doi:10.1016/j.ijcard.2007.03.112

10. Ng TT, McGory ML, Ko CY, et al. Meta-analysis in surgery. Arch Surg 2006;141:11251130.

11. Ray CE, Prochazka A. The need for anticoagulation following inferiorvena cava filter placement: systematic review. Cardiovasc InterventRadiol 2007; 31:316324.

12. Nicolucci A, Grilli R, Alexanian AA, Apolone G, Torri V, Liberati A. Qualityevolution and clinical implications of randomized controlled trials on thetreatment of lung cancer. A lost opportunity for meta-analysis. JAMA1989; 262:21012107.

13. Marsoni S, Torri V, Taiana A. Critical review of the quality and develop-ment of randomized clinical trials and their influence on the treatmentof advanced epithelial ovarian cancer. Ann Oncol 1990; 1:343350.

14. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improvingthe quality of reports of meta-analyses of randomised controlled trials:the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet1999; 354:18961900.

15. Gami AS, Witt BJ, Howard DE, et al. Metabolic syndrome and risk of inci-dent cardiovascular events and death. J Am Coll Cardiol 2007; 49:403414.

16. Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in thepresence of heterogeneity. Stat Med 2003; 22:21132126.

17. Dentali F, Doukelis D, Gianni M, et al. Meta-analysis: anticoagulant pro-phylaxis to prevent symptomatic venous thromboembolism in hospital-ized medical patients. Ann Intern Med 2007; 146:278288.

18. Whitehead A. Meta-Analysis of Controlled Clinical Trials. New York:Wiley, 2002.

19. Gebski V, Burmeister B, Smithers BM, et al. Survival benefits from neoad-juvant chemoradiotherapy or chemotherapy in oesophageal carcinoma:a meta-analysis. Lancet Oncology 2007; 8:226234.

20. Trivella M, Pezzella F, Pastorino U, et al. Microvessel density as a prognos-tic factor in non-small-cell lung carcinoma: a meta-analysis of individualpatient data. Lancet Oncology 2007; 8:488499.

21. Kunz R, Vist G, Oxman AD. Randomisation to protect against selectionbias in healthcare trials (Cochrane Methodology Review). In: TheCochrane Library, Issue 1, 2003. Oxford: Update Software.

22. Gillum LA, Mamidipudi SK, Johnston SC. Ischemic stroke risk with oralcontraceptives: a meta-analysis. JAMA 2000; 284:7278.

23. Chan WS, Ray J, Wai EK, et al. Risk of stroke in women exposed to low-dose oral contraceptives. Arch Intern Med 2004; 164:741747.

24. Gillum LA, Johnston SC. Oral contraceptives and stroke risk: the debatecontinues. Lancet 2004; 3:453454.

25. Bhutta AT, Cleves MA, Casey PH, et al. Cognitive and behavioral out-comes of schoolaged children who were born preterm. JAMA 2002;288:728737.

26. De Luca G, Suryapranta H, Stone GW, et al. Adjunctive mechanicaldevices to prevent distal embolization in patients undergoing mechani-cal revascularization for acute myocardial infarction: a meta-analysis ofrandomized trials. Am Heart J 2007; 153:343353.

27. Wang R, Lagakos SW, Ware JH, et al. Statistics in medicinereporting ofsubgroup analyses in clinical trials. N Engl J Med 2007; 357:21892194.

28. Shuster JJ, Jones LS, Salmon DA. Fixed vs random effects meta-analysis inrare events: the rosiglitazone link with myocardial infarction and cardiacdeath. Stat Med 2007; 26:43754385.

29. Bracken MB. Rosiglitazone and cardiovascular risk. N Engl J Med 2007;357:937938.

30. Sutton AJ, Lambert PC, Hellmich M, et al. Meta-analysis in practice: a crit-ical review of available software. In: Stangl DK, Berry DA, editors. Meta-Analysis in Medicine and Health Policy. New York: Marcel Dekker, 2000.

ADDRESS: Esteban Walker, PhD, Quantitative Health Sciences, Wb4,Cleveland Clinic, 9500 Euclid Avenue, Cleveland, OH 44195; [email protected].

WeWelcome

YourLetters

WeWelcome

YourLetters

WE ENCOURAGE YOU TO WRITE, eitherto respond to an article published in theJournal or to address a clinical issue ofimportance to you. You may submitletters by mail, fax, or e-mail.

MAILING ADDRESSLetters to the EditorCleveland Clinic Journal of Medicine9500 Euclid Ave., NA32Cleveland, OH 44195FAX 216.444.9385E-MAIL [email protected]

Please be sure to include your full address,phone number, fax number, and e-mailaddress. Please write concisely, as space islimited. Letters may be edited for style andlength. We cannot return materials sent.Submission of a letter constitutes permissionfor the Cleveland Clinic Journal of Medicineto publish it in various editions and forms.


Documents

Meta-Analysis_ Its Srengths and Limitations. 2013