Meta-analysis of randomized trials: Looking back and looking ahead

ELSEVIER

Meta-Analysis of Randomized Trials:

Looking Back and Looking Ahead

Salim Yusuf, FRCPC, DPhil Professor of Medicine and Director, Division of Cardiology, McMaster University, Hamilton, Ontario, Canada

ABSTRACT: Meta-analyses as currently practiced are usually retrospective. They can be made more rigorous by developing a protocol that incorporates prospectively the elements that are usually necessary in a well-designed trial. Meta-analysis and large trials are complementary. Meta-analysis of small trials is useful in generating the hypotheses and assisting in the design of the large trials that are needed. Once the large trials have been completed, they could be brought together within the framework of a meta-analysis to estimate the overall treatment effect with greater confidence and to explore the effects in various subgroups. This article explores the value and limitations of meta-analyses and suggests ways of improving their conduct and interpretation. Controlled Clin Trials 1997;18:59&601 0 Elsevier Science Inc. 1997

KEY WORDS: Meta-analysis, randomized clinical trials

INTRODUCTION

Meta-analyses of randomized clinical trials have increased in popularity. The initial meta-analysis in the medical field was probably conducted by Chal- mers in the 1970s [ 11. This meta-analysis of anticoagulant trials in acute myocardial infarction (AMI) drew not only a great deal of attention but also significant criticism [2]. Some of the criticisms were due to the fact that the results were counter to the prevailing medical dogma at that time. Moreover, the methodology was still emerging and not familiar to most physicians. Subsequently, several investigators undertook to systematize the methodology and applied it to a number of clinical areas. The first of these publications, which outlined both the statistical as well as the non-statistical methodology in some detail, was applied to the topic of beta-blockers following myocardial infarction [3]. This meta-analysis was widely accepted, partly because it confirmed what clinicians already believed and perhaps partly because of the detail and rigor with which the methodological aspects were addressed. A subsequent and equally rigorous meta-analysis by the same group on thrombolytic therapy after acute myocardial infarction had some difficulty in getting accepted for publication by journals, perhaps in part because it was counter to the prevailing

Address reprint requests to: Salim Yusuf, FRCPC, DPhil, Hamilton General Hospital, 237 Barton Street East, Hamilton, Ontario, Canada, L8L 2X2.

Received August 3, 1996; revised October 31, 1996; accepted December 24, 1996.

Controlled Clinical Trials l&594-601 (1997) 0 Elsevier Science Inc. 1997 655 Avenue of the Americas, New York, NY 10010

0197-2456/97/$17.00 PI1 SOl97-2456(97)00052-4

Perspectives on Meta-Analysis 595

expert opinion. After being rejected twice, it was finally published in the Euro- pean Heart Journal [4]. This meta-analysis of the trials of streptokinase predicted the benefits of thrombolytic therapy fairly accurately, indicated that thrombolytic therapy after acute myocardial infarction was applicable to a broader group of patients than those presenting only very early after the onset of symptoms, and showed that anticoagulant therapy with heparin was probably not essential. These two latter aspects were criticized in an accompanying editorial and continued to be viewed with skepticism chiefly because they were inconsistent with the then current framework of thought in this field, based upon available experimental data at that time [5]. The findings of this meta- analysis, however, formed the basis of the design of a number of large and well-designed studies that confirmed the several hypotheses generated by the meta-analysis [6,7]. In cancer, the Oxford group in particular has pioneered extensive efforts related to breast cancer trials [S]. Although the results of these meta-analyses were intensively questioned and debated, their findings have subsequently been more widely accepted.

The methodology of meta-analysis has evolved from utilization of summary data of published trials to inclusion of unpublished data from published trials, to extensive searches to identify unpublished trials, and more recently, to utilizing individual patient data from both published and unpublished trials [9,10]. Two studies have demonstrated the value of a rigorous approach utilizing individual patient data by indicating important limitations with the use of summary data from the published literature such as incomplete reporting, unavailability of unpromising results, etc. [11,12]. These biases were overcome by incorporation of individual patient data from published and unpublished trials. Furthermore, individual patient data are important if one is interested in subgroup analyses across trials. At the same time, with the growing popularity of meta-analysis, some of the methodology that was originally developed based on summary data from published trials has become quite widely used.

Unfortunately, the widespread use of meta-analysis has not been accompa- nied by an appreciation of its limitations. The current “respectability” of the methodology, a perception that meta-analysis is a “quick” way to write a paper, and the availability of software programs that allow the statistical calculations to be conducted rapidly and uncritically have facilitated the increasing use of meta-analyses. Sometimes individuals with only limited knowledge of the pharmacologic aspects of a treatment, the biology of the disease, or clinical circumstances that relate to the specific question may perform meta-analyses leading to an analysis with little clinical relevance. Therefore, while the methodology of meta-analysis has improved considerably over the last decade, there has also been an increase in the number of meta-analyses utilizing subopti- ma1 methods.

A parallel methodologic development in the last decade in clinical trials has been the recognition that very large trials are needed to detect plausible differences on major outcomes (e.g., death, myocardial infarction, stroke, or tumor recurrence) especially in some common diseases such as cardiovascular diseases and common cancers [13]. Further, many people recognize that large trials are feasible only if protocols are simplified and data collection procedures are minimized. In many respects, the approaches to meta-analysis and the conduct of large simple trials are similar [3,13]. Both approaches are based on

596 s. Yusuf

collecting only key outcome data, both trust that randomization tends to balance key variables and that covariate adjustments add little to the sensitivity of the analysis. The common aspects of these approaches are based on two principles: first, only moderate effects on major outcomes are plausible; therefore, large amounts of information (a large number of events) are required to obtain reliable results; second, interactions that do occur are more likely to be quantitative (differences in degree) than qualitative (differences in direction of effect). Therefore, the collective information from a large trial or a meta-analysis is more reliable than some data-derived emphasis on a subset of information. Both of these fundamental judgments are important to bear in mind as we evaluate the roles of meta-analyses and large trials. The main difference between the conduct of large trials and meta-analysis is that the former specifies all key aspects of the study in advance (the hypotheses to be tested, sample size, analytical methods, etc.) whereas the latter is generally retrospective (hypotheses are often data-derived, no formal sample size is calculated in advance, the outcomes of interest in a meta-analysis may not be uniformly or completely available). Further, only when well-designed trials exist is a reliable meta- analysis possible. In general, under ideal circumstances individual well- designed trials can be used for hypothesis testing, whereas meta-analyses of these trials can be used for estimating treatment effects.

COMPARISON OF RESULTS OF META-ANALYSIS AND LARGE TRIALS

In recent years, there have been meta-analyses of the available small trials in many clinical areas. When these meta-analyses have led to promising results, they have then been used to design more definitive studies. It is worth reviewing some of them as general methodologic lessons emerge. While there are numer- ous examples, I will review four examples: magnesium for acute myocardial infarction [14]; nitrate therapy for acute myocardial infarction [15]; aspirin in the prevention of preeclampsia [16]; and fibrinolytic therapy for acute myocardial infarction [4]. The first three meta-analyses were based on small numbers of events but suggested a fairly large reduction in major clinical outcomes. These hypotheses were then tested in large prospective randomized trials. Unfortu- nately, in all three cases, the large trials either refuted the possibility of any benefit (magnesium) or suggested much smaller sized benefit (nitrates in acute myocardial infarction and aspirin in preeclampsia). This has led to considerable debate as to whether it was the meta-analysis or the large trials that were unreliable. Reviewing the two principles that were summarized in the previous section, if one assumed that only moderate sized effects were possible, the apparent large effects observed in the meta-analyses of small trials with magnesium, nitrates, and aspirin in preeclampsia should perhaps have been tempered by this general judgment. If a result appears to be too good to be true, it probably is. If such a judgment were applied, then one would not have accepted these meta-analyses based on small numbers of events as being conclusive, especially when they suggested very large effect sizes that were unlikely to be plausible. Therefore, if appropriate judgments were used, meta-analysis of small trials could trigger the development of large trials and perhaps even assist in their design.


Each specific area is worth examining in detail to assess clues for the apparent discrepancy. The meta-analysis we performed of the available trials of magnesium in acute myocardial infarction was based only on a small number of events (about 80 deaths). Our decision to conduct this specific meta-analysis likely was based on an informal impression that the meta-analysis would be “statistically significant” with “interesting” publishable results. (Note that at the same time, we were aware of “interesting” but not “statistically significant” results if we were to combine the mortality data from trials evaluating glucose, insulin, and potassium in acute myocardial infarction. Although these risk reductions were described as part of our general published review, we did not develop this into a separate manuscript. Perhaps, we judged that the results were not “exciting” enough to warrant a separate publication). This situation is similar to data-derived emphasis on subgroups. Therefore, the meta-analysis of magnesium trials should have been seen as developing a hypothesis (rather than proof) that magnesium might reduce mortality. We should, therefore, not be surprised that such hypotheses derived from a meta-analysis may not be confirmed by large trials [15].

With regard to the role of nitrates in acute myocardial infarction, the meta- analysis used intravenous nitrate therapy titrated carefully to avoid hypotension. Very few patients allocated to control received intravenous nitrates. The mortality rates of these trials was high and suggests that the patients included in these trials were of moderate to high risk. By contrast, ISIS-4 [17] utilized oral nitrates against a background of high use of intravenous nitrates in both active and control groups, thereby diluting any potential for benefit (a bias to the null). There was a greater potential for hypotension (which may be detrimental) due to the simultaneous use of angiotension converting enzyme (ACE-I), thrombolytic drugs and, in some patients, beta-blockers. Aspirin was used widely and its antiplatelet effects may overlap with the antiplatelet effects of nitrates. The population studied was at substantially lower risk of death compared to earlier trials on which the meta-analysis was based. Therefore, the questions that ISIS-4 addressed were dissimilar to the meta-analysis, the patient populations were at much lower risk of death, and ancillary management of the patients differed substantially. Furthermore, the high use of intravenous nitrates in the control group may have biased the results of ISIS-4 to the “null.”

The apparent large effects of a potential 50% reduction in preeclampsia by aspirin in a meta-analysis of several small trials may have been biased by several factors such as those outlined above with magnesium. Additionally, there may have been substantial publication biases as information on eclampsia from about half the trials was not available [16]. One should therefore not be surprised if the large trial showed a much smaller effect [16].

In contrast to the above three meta-analyses, a meta-analysis of about 5000 patients (about 1000 deaths) entered into randomized trials of thrombolytic therapy indicated a highly significant reduction in mortality [4], which later large trials confirmed [6,7]. The database for this meta-analysis, which included several times the number of patients in the magnesium overview, suggested a quite moderate treatment effect (20% risk reduction), which is more plausible than the 50% suggested by the magnesium overview.

598 s. Yusuf

These examples emphasize the importance of adhering to the fundamental judgment that, in general, treatments either have moderate benefits or no effect and that deviations from this should be viewed cautiously (unless confirmed by a large body of evidence).

IMPLICATIONS OF WIDESPREAD USE OF META-ANALYSIS

Very recently, there has been a major development in the area of meta- analysis in clinical trials (Cochrane Collaboration) in the form of an organized attempt to conduct formal meta-analyses of every possible intervention in as many areas as possible [20]. What are the benefits and the dangers of widespread use of meta-analysis? Clearly, the Cochrane Collaboration and its pro- mulgation of meta-analyses could potentially lead to many benefits including wider acceptance of the methodology, a realization on the part of clinicians and policy makers of the importance of bringing together all relevant data using quantitative methods, and if meta-analyses are done well, with due care and rigor and interpreted with appropriate caution, useful guidelines for patient management, as well as for health policy, would emerge. If the magnitude of the available data is modest and potentially unreliable, it would identify areas that require further investigation. Further, meta-analysis of small trials may at least suggest hypotheses that would help design better and larger studies.

Despite the many potential benefits of meta-analyses, widespread and uncrit- ical meta-analyses can lead to inappropriate conclusions and can sometimes cause more harm than good. The increasing acceptance and respectability of meta-analysis may mean that some people with little knowledge of many relevant aspects of the disease or intervention may use it mechanically and uncritically. It is important to include content experts as well as experts in methodology to strike the balance that is needed: all methodologically sound efforts should also be biologically sensible and clinically applicable. The most convincing level of evidence occurs when the architecture of information from diverse sources such as large randomized trials (or their meta-analysis), biologi- cal knowledge, and epidemiologic studies provides a coherent and consistent body of information.

Another problem of conducting a large number of meta-analyses relates to the issue of multiplicity. There are two kinds of multiplicity in the areas of meta-analysis. First, a large number of carefully conducted meta-analyses lead to an increased chance of false-positive (Type I error) results. Second, this inflation is further exaggerated since multiple endpoints or multiple subgroups may be examined and many of the decisions to emphasize specific analyses may be entirely data-derived. One such area may be the emphasis on the effects of a specific kind of calcium antagonist on reinfarction (but not death) in patients with non-Q wave myocardial infarction [19]. Therefore, it should not be surprising that when a large number of meta-analyses are done, even if carefully conducted, occasionally they may come up with the wrong answer. The play of chance may exaggerate the true benefits when there are only a small number of events or few trials. Therefore, claims of benefit based on meta-analysis with limited data should be viewed with skepticism.

Readers of meta-analyses may be lulled into a false sense of security that after all, a meta-analysis (a “gold” standard for evaluating therapy) has been


conducted and therefore there must be some intrinsic validity with the approach. One such example has been a poorly conducted meta-analysis of a variety of anti-hypertensive treatments in the prevention of left ventricular hypertrophy [20]. The investigators who conducted this meta-analysis claimed that ACE-inhibitor therapy reduced the risk of left ventricular hypertrophy to a greater extent than any other treatment. This conclusion was not based upon trials that directly compared various anti-hypertensive agents (which contra- dicts this) [21], rather upon indirect comparisons across quite different studies, some of which were observational. Although the available randomized trials do not support this hypothesis, the conclusion that ACE-inhibitors reduce left ventricular hypertrophy to a greater extent compared to other treatments has gained widespread currency.

CUMULATIVE META-ANALYSIS

A recent school of thought suggests that when one achieves some degree of statistical significance based on cumulating the results of trials (cumulative meta-analysis), further trials become unethical [22]. Evidence is indeed cumulative. At some time the collective evidence is so strong that further information is generally not required and patients should be treated with the therapy. There are, however, many factors to be considered in making such a general claim based solely on a meta-analysis. First, since meta-analysis is a retrospective exercise, formal methods of evaluating the strength of evidence (e.g., interim monitoring boundaries, specified number of looks at accumulating data) do not exist. In this symposium, Pogue and Yusuf [23] have tried to develop some methods based on modifying established methods of statistical monitoring of trials. This general approach provides a framework for utilizing levels of proof similar to that required in large trials. Second, even if the overall results are clear-cut for a specific patient population, in some circumstances physicians may wish to obtain more information regarding the balance of benefit versus risk in specific subgroups or in broader populations. Therefore, further trials may be needed. Third, sometimes the results may be extrapolated too widely, preventing the development of proper trials in this expanded population. Ide- ally, in such circumstances, meta-analyses should facilitate the development of more definitive trials and not impede them, unless the results of the meta- analysis are overwhelmingly clear.

MAKING A RETROSPECTIVE METHODOLOGY PROSPECTIVE

I have pointed out several times that meta-analyses are generally retrospective. By this I mean most meta-analyses do not identify a primary hypothesis before examining the data. Therefore, prospective protocols for the meta-analysis may not have been developed. Data accumulate in unplanned sequences, not all relevant studies may be known, and there may be variations in the types of data collected so that the outcomes of interest may or may not have been collected in some trials. If these are collected, they may not be reported and, if reported, they may have been defined variably. Minor variations in endpoint definition probably do not matter, but if these variations are data- derived, then the potential for significant bias exists. Furthermore, the choice

600 s. Yusuf

of an area to be the subject of a meta-analysis or the choice of which trials to include in the meta-analysis may be influenced by the results and expectations of the meta-analyst. Many of these problems can be overcome by developing a prospective protocol for meta-analysis at a time that several similar trials are planned or at least under way (but before all their results are available), register- ing the trials that would be included based on similarities in design, setting out specific hypotheses, data collection procedures, and analytic strategies in advance. Such a prospective effort in which the principal investigators of the studies themselves are involved will lead to methodologically sound and biologically based approaches that are more likely to yield reliable medically relevant answers.

What, in conclusion, is the role of meta-analysis and large trials in the evaluation of therapy? Meta-analyses are not replacements for large trials nor are large trials replacements for meta-analyses. Both are excellent ways of assessing the efficacy and adverse effects associated with specific interventions. Nevertheless, because each method possesses certain limitations, both large trials and meta-analyses are needed. Their complementary role is evident from the common principles that underlie them. Meta-analyses of small trials should help in the design of more definitive trials. Once these more definitive trials are completed, bringing these large trials together in a further formal meta- analysis will assist in obtaining more precise estimates of treatment effects overall as well as in subgroups. The information derived from large trials and meta-analyses should be interpreted in the context of knowledge from epidemiology, biology, physiology, and pharmacology, and applied according to the clinical context.

REFERENCES

1. Chalmers TC, Matta RJ, Smith J Jr, et al. Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction. N Engl JMed 1977;297:1091-1096.

2. Goldman L, Feinstein A. Anticoagulants and myocardial infarction: the problems of pooling, drowning and floating. Ann Int Med 1979;90:92-94.

3. Yusuf S, Peto R, Lewis J, Collins R, Sleight I’. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis 1985;27:335-371.

4. Yusuf S, Collins R, Peto R, Furberg C, et al. Intravenous and intracoronary fibrinolytic therapy in acute myocardial infarction: overview of results on mortality, reinfarction and side-effects from 33 randomized controlled trials. EW Heart J 1985;6:556-585.

5. Abildgaard U, Bjerkelund C. Coronary thrombolysis made easy? Eur Heart J 1985;6:584-585.

6. ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Ran- domized trial of intravenous streptokinase, oral aspirin, both or neither among 17,187 cases of suspected acute myocardial infarction. Lancet 1988;ii:349-360.

7. Gruppo Italian0 per lo Studio Della Streptochinasi Nell ‘lnfarto Miocardico (GISSI): effectiveness of IV thrombolytic treatment in acute myocardial infarction. Lancet 1986;1:397401.

8. Early Breast Cancer Trialists’ Collaborative Group. Treatment of Early Breast Cancer. Oxford: Oxford Medical Publications; 1990.

9. Antiplatelet Trialists’ Collaboration (1994). Collaborative overview of randomized trials of antiplatelet therapy. 1: prevention of death, myocardial infarction and stroke


by prolonged antiplatelet therapy in various categories of patients. BM] 1994; 308:81-106.

10. Yusuf S, Zucker D, Peduzzi I’, et al. Effect of coronary artery bypass graft surgery on survival: overview of 10 year results from randomized trials of Coronary Artery Bypass Graft Surgery Trialists Collaboration. Lancet 1994;334:563-570.

11. Jeng GT, Scott JR, Burmeister LF. A comparison of meta-analytic results using literature vs individual patient data. Paternal cell immunization for recurrent miscar- riage. JAMA 1995;274:830-836.

12. Stewart L, Parmar M. Meta-analysis of the literature or of individual patient data: is there a difference? Lancet 1993;341:418-422.

13. Yusuf S, Collins R, and Peto R. Why do we need some large, simple randomized trials? Stat Med 1984;3:409420.

14. Yusuf S, Flather M. Magnesium in acute myocardial infarction. ISIS-4 provides no grounds for its routine use. BMI 1995;310:751-752.

15. Flather MD, Farkouh ME, Yusuf S. Meta-analysis in the evaluation of therapies. In: Julian D, Braunwald E, eds. Management of Acute Myocardial Infarction. Toronto: WB Saunder Company Ltd; 1995, pp 393406.

16. CLASP Collaborative Group. CLASP: a randomized trial of low-dose aspirin for the prevention and treatment of pre-eclampsia among 9364 pregnant women. Lan- cet 1994;343:619-629.

17. ISIS-4 Collaborative Group: a randomized factorial trial assessing early oral capto- pril, oral mononitrate and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet 1995;345:669-85.

18. Chalmers I. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann NY Acad Sci 1993;703:156-163.

19. Gibbons RS, Boden WE, Theroux I’, et al. Diltiazem and reinfarction in patients with non-Q wave myocardial infarction. N Engl I Med 1986;315:423.

20. Dahlof B, Pennert K, Hansson L. Reversal of left ventricular hypertrophy in hypertensive patients. A meta-analysis of 109 treatment studies. American journal of Hyperten- sion 1992;5:95-110.

21. Neaton JD, Grimm RH Jr, Prineas RJ, et al., for the Treatment of Mild Hypertension Study Research Group. Treatment of Mild Hypertension Study. Final results. JAMA 1993;270:713-724.

22. Lau J, Antman EM, Jimenez-Silva J, et al. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N En@ J Med 1992;327:248-254.

23. Pogue J, Yusuf S. Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Controlled Clin Trials 1997; 18:580-593.

Documents

Meta-analysis of randomized trials: Looking back and looking ahead