An alternative method for meta analysis

An Alternative Method for Meta-Analysis1

Joachim Hartung

Department of StatisticsUniversity of DortmundGermany

Summary

In many fields of applications, test statistics are obtained by combining estimates from several experi-ments, studies or centres of a multi-centre trial. The commonly used test procedure to judge the evi-dence of a common overall effect can result in a considerable overestimation of the significance level,leading to a high rate of too liberal decisions. Alternative test statistics are presented and better approx-imating test distributions are derived. Explicitly discussed are the methods in the unbalanced heterosce-dastic 1-way random ANOVA model and for the probability difference method, including interactiontreatment by centres or studies. Numerical results are presented by simulation studies.

Key words: Meta-analysis; Combining experiments; Multi-centre study; Interactiontreatment by centres; Heteroscedastic ANOVA model; Random effects;Probability difference method; Log odds ratio; Log relative risk.

Zusammenfassung

In vielen Anwendungsgebieten werden Teststatistiken mittels Kombination von SchaÈtzungen ausverschiedenen Experimenten, Studien oder Zentren einer multizentrischen Studie erhalten. Die allge-mein verwandte Testmethode zur Beurteilung der Evidenz eines Gesamteffektes kann zu einer be-traÈchtlichen ÛberschaÈtzung des Signifikanz-Niveaus fuÈhren, mit der Folge eines hohen Anteils zuliberaler Entscheidungen. Es werden alternative Teststatistiken vorgestellt und besser approximierendeTestverteilungen hergeleitet. Die Verfahren werden in der unbalancierten heteroskedastischen Ein-fachklassifikation der Varianzanalyse mit zufaÈlligen Effekten und fuÈr die Wahrscheinlichkeits-Diffe-renzen-Methode ausfuÈhrlich diskutiert, unter Einbeziehung einer Wechselwirkung zwischen Behand-lung und Zentren bzw. Studien. Numerische Ergebnisse werden aufgezeigt mittels Simulations-studien.

1. Introduction

The problem of judging an overall effect from several studies or experimentsarises in a variety of application fields. However, not only in meta-analysis of

Biometrical Journal 41 (1999) 8, 901±916

1 Presented at the 45th Biometrical Colloquium of the German Region of the International BiometricSociety, Dortmund, 1999.

composed individual results but also in analyzing multi-centre trials, for instanceone has separated samples with heterogeneous error variances and possibly aninteraction of treatments with centres to asses the overall effect. Taking the effectsof this interaction as random, one gets the so-called random effects model ofmeta-analysis, cf. sec. 2.

In the commonly used method of meta-analysis, tracing back to Cochran(1937, 1954), one gets the variance of the common estimate of the overalleffect by estimating its components separately, and for the corresponding teststatistic the standard normal distribution is taken as test distribution, cf. sec. 3.Now, this procedure is observed not to hold the prescribed significance level,which can lead to a high rate of too liberal decisions. That phenomenon mainlyis not a problem of the type of variance estimator involved. Computational ex-perience with other estimators than the usual unbiased one, for instance also witha kind of nonnegative minimum biased estimator, as discussed by Hartung(1981), yield qualitatively no essential improvements in the significance levelsobtained.

Therefore in the following, cf. sec. 4, estimation functions are introduced esti-mating the variance of the weighted mean directly, based on weights which on afirst stage are estimated upon some other estimation principle, for instance here ischosen the classical one.

Furthermore, their distributions are approximated by equating the first two mo-ments to that one of a c2-distribution, such that the test of significance for theoverall effect becomes an approximate t-test that is more able to hold the actualsignificance level near the prescribed one.

The performance of the test procedures is discussed, including simulation stud-ies, in the unbalanced heteroscedastic random 1-way ANOVA model, cf. sec. 5,and, finally, in order to demonstrate the application to data that don't follow anANOVA model, to the probability difference method, cf. sec. 6, comparing twoproportions that are observed several times.

Let us refer e. g. to Hardy and Thompson (1996), Biggerstaff and Tweedie(1997) for likelihood approaches, needing however reasonable sample sizes for theasymptotic results involved.

Sometimes there is the opinion that one should avoid the random effects modelbecause it would be too conservative, and one should better work with the so-called fixed effects model, neglecting the interaction effects and assuming homo-geneity with respect to a common mean. Of course, this would lead to a higheractual significance level. However, here it is to say that, even if the fixed effectsmodel is the correct one for the data situation given, also the method commonlyused there can yield a high rate of too liberal decisions, e.g. Li, Shi, and Roth(1994), BoÈckenhoff and Hartung (1998).

With respect to combining dependent results, as for instance so-called multipleendpoints in a clinical trial, we refer to Hartung (1999).

902 J. Hartung: An Alternative Method for Meta-Analysis

2. The Model

Let mi for i � 1; . . . ; k; k � 2, be stochastically independent normally distributedunbiased estimators for the common mean m of k independent experiments, studiesor centres of a multi-centre trial, which also let provide stochastically independent

unbiased estimators bs2i > 0, (a.e.), for the partial variance s2

i > 0 ofbmi; i � 1; . . . ; k. Due, for instance, to an `interaction of response with centres',there may be a common part s2

a � 0 of the variance of the bmi that can not beestimated in the ith study, i � 1; . . . ; k; that is, we have the so-called randomeffects model of parametric meta-analysis, respectively of combining of experi-ments:

bmi � N�m; s2a � s2

i �; bs2i given; i � 1; . . . ; k ; �1�

e.g. Cochran (1937, 1954), Yates and Cochran (1938), Hedges and Olkin(1985), DerSimonian and Laird (1986), Whitehead and Whitehead (1991),Draper et al. (1992).

Of main interest here is to test a hypothesis like

H0 : m � 0 against H1 : m > 0; �2�respectively to derive a confidence interval for the common mean m. Denote

ti � 1

s2a � s2

i

; i � 1; . . . ; k; t �Pki�1

ti; �3�

the best unbiased estimator of m would be

~m �Pki�1

ti

tbmi �4�

with the variance var �~m� � 1=t, leading, under m � 0, to the test statistic

~m��1=t

p � N�0; 1� : �5�

Now, for a realization, the involved parameters have to be estimated.Note that in applications often bmi is a function of further parameter estimates,

for instance the z-transformed of a correlation, a mean difference or an effect sizeof two treatments, or e.g. the difference, cf. sec. 6, the (log) odds ratio or relative

risk of two observed proportions, cf. the references cited above, and bs2i frequently

is only an approximation, e.g. via the delta-method. The general assumptions forbmi and bs2i then can be fulfilled only in approximation, of course, implying the

same for resulting properties.

Biometrical Journal 41 (1999) 8 903

3. The Commonly used Method

Let denote wi � 1=s2i ; i � 1; . . . ; k, and w �Pk

i�1wi, then, e.g. Chochran (1954),

DerSimonian and Laird (1986), Whitehead and Whitehead (1991), an unbiasedestimator of s2

a is given by

s2a :� w

w2 ÿPki�1

w2i

Pki�1

wi bmi ÿPkj�1

wj

wbmj

!2

ÿ k � 1

8<:9=;; �6�

with the realization es2a, replacing s2

i by bs2i ; i � 1; . . . ; k, in s2

a. This estimator canbecome negative with positive probability, and so one truncates it at zero,bs2

a :� max f0; es2ag;

such that with

bti � 1bs2a � bs2

i

; i � 1; . . . ; k; and t �Pki�1

bti �7�

the common mean is estimated by

m �Pki�1

bti

t� bmi ; �8�

and the test statistic under m � 0 is taken as

T1 :� m��1=t

p �appr:N�0; 1�: (9)

Caused by distributional deficiencies, cf. also Li et al. (1994), BoÈckenhoff andHartung (1998), the resulting test procedure is not satisfactory, because of theobservation that the actual levels attained by the test can rise much above theprescribed level, leading to a high rate of too liberal decisions, cf. the simulationresults in sec. 5 and 6.

4. A Refined Method

Let bi � ti=t; i � 1; . . . ; k; b � �b1; . . . ; bk�0, and xi � bmi; i � 1; . . . ; k;x � �x1; . . . ; xk�0 (where c0 denotes the transpose of a vector c), then we considerthe following quadratic form in x:

S�b� :�Pki�1

bi�xi ÿ b0x�2: �10�


Theorem 4.1:(i) t � S�b� has a (central) c2-distribution with (k-1) degrees of freedom,(ii) ~m � b0x and S�b� are stochastically independent.

Proof: Denote D � diag �s2a � s2

i ; i � 1; . . . ; k� the diagonal covariance matrixof x and 1 :� �1; . . . ; 1�0 2 Rk, i.e.

x � N�m � 1; D�;and let 1i :� �0; . . . ; 0; 1; 0; . . . ; 0�0 2 Rk, with the 1 at the ith place, mi :� 1i ÿ b

and the matrix M :� t �Pki�1

bimim0i. So we can write

tS�b� � t �Pki�1

bi�m0ix�2

� t �Pki�1

bix0mim

0ix

� x0Mx;

and if MD � �MD�2, then x0Mx is c2-distributed with trace �MD� degrees of free-dom, and if MDb � 0, then x0Mx and b0x are stochastically independent, e.g.Mathai and Provost (1992, p. 197, 227); note that m0i1 � 0 and thusME�x� � 0.To (i): We have

MD �Pki�1

tbimim0iD

�Pki�1

tbi�1i ÿ b��1i ÿ b�0 diag1

ti; i � 1; . . . ; k

� ��Pk

i�1t�bi1i ÿ bib�

1

ti1i ÿ 1

t1

� �0�Pk

i�1t

1

t1i10i ÿ

1

tb10i ÿ

1

tbi1i1

0 � 1

tbib10

� ��Pk

i�11i10i ÿ b

Pki�1

10i ÿPki�1

bi1i

� �10 � Pk

i�1bi

� �b10

� I ÿ b10 ÿPki�1

bi1i10 � b10

� I ÿPki�1

bi1i10;

where I denotes the �k � k�-identity matrix, and


�MD�2 � I ÿPki�1

bi1i10

� �I ÿPk

i�1bi1i1

0� �

� I ÿ 2Pki�1

bi1i10 �Pk

i�1

Pkj�1

bibj1i101j1

0

� I ÿ 2Pki�1

bi1i10 � Pk

j�1bj

!Pki�1

bi1i10

� I ÿPki�1

bi1i10

� MD;

noting thatPkj�1

bj � 1 and 101j � 1. Further, we get trace �MD� � k ÿPki�1

bi

� k ÿ 1, which completes the proof of (i).

To (ii): There is with (i)

MDb � I ÿPki�1

bi1i10

� �b

� bÿPki�1

bi1i10b

� bÿPki�1

bi1i�28� � 0 ;

which yields (ii), completing the proof.Note that for the special case of a usual k-sample problem, i.e. s2

a � 0, a resultlike (i) above is stated already by Cochran (1937, p. 111).

Defining now

l�b� � b0b1ÿ b0b

; and wi�b� � bi ÿbi ÿ b2

i

1ÿ b0b; i � 1; . . . ; k ; �11�

we consider the affine quadratic (with respect to the random variables) form

Q�b� :� l�b� S�b� �Pki�1

wi�b� bs2i ; �12�

and get the following theorem.

Theorem 4.2:(i) Q�b� is an unbiased estimator of 1=t � var �~m�:(ii) If the estimators bs2

i of s2i are also stochastically independent of the estima-

tors bmj of m; i; j � 1; . . . ; k, then


(a) Q�b� and ~m are stochastically independent, and(b) an approximate (central) c2�n�-distribution of n � Q�b�=�1=t� has the

degrees of freedom

n � nQ�b� � 2 � �1=t�2

2�k ÿ 1� �1=t�2 l�b�2 �Pki�1

wi�b�2 var �s2i �:

Proof: To (i): For the expectation we get with (i) of theorem 4.1, noting that

Pki�1

wi � 1ÿ1ÿPk

i�1b2

i

�1ÿ b0b� � 0 ;

E�Q�b�� l�b� 1

t�k ÿ 1� �Pk

i�1wi�b� s2

i �Pki�1

wi�b��

s2a

� �k ÿ 1� 1

tl�b� �Pk

i�1

ti

tÿ �ti=t� ÿ �t2

i =t2�1ÿ b0b

� �1

ti

� �k ÿ 1� 1

tl�b� �Pk

i�1

1

tÿ �1=t� ÿ �ti=t2�

1ÿ b0b

� �� 1

t�k ÿ 1� b0b

1ÿ b0b� 1

tk ÿ �k=t� ÿ �1=t�

1ÿ b0b

� 1

tk � �k ÿ 1� b0bÿ k � 1

1ÿ b0b

� �� 1

t

k ÿ kb0b� kb0bÿ b0bÿ k � 1

1ÿ b0b

� �� 1

t:

To (ii): Part (a) follows from theorem 4.1 (ii), together with the additional assump-tions above. Now we come to part (b).

If the random variable Q* � n � Q=E�Q� follows a c2�n�-distribution, then forthe variance we have

Var �Q*� � n2 � var �Q��E�Q��2 � 2 � n;

that is

n � 2 � �E�Q��2

var �Q� ; �13�

which conversely can be used as an estimate of the degrees of freedom of anapproximate c2-distribution, i.e. the first two moments of the distributions are


equated, cf. Patnaik (1949). So again, with (i) of theorem 4.1, we get for thevariance of Q�b�

VQ�b� :� l�b�2 1

t

� �2

2�k ÿ 1� �Pki�1

wi�b�2 var � bs2i � ; �14�

and by (i): E�Q�b�� 1=t, yielding now the desired estimate for the degrees offreedom, which completes the proof.

Corollary 4.1: Either at least one of the wi�b�; i � 1; . . . ; k, is negative or allwi�b� are zero and all bi � 1=k; i � 1; . . . ; k.

Proof: Assume that for all i � 1; . . . ; k : wi�b� � 0. Now, cf. (11),Pki�1

wi�b� � 0,

implying, by our assumption, that for all i � 1; . . . ; k, there holds wi�b� � 0, andtherefore for all i � 1; . . . ; k,

bi�1ÿ b0b� � bi ÿ b2i ;

respectively dividing by bi�bi > 0� yields

1ÿ b0b � 1ÿ bi ;

i.e. all bi are identical and byPki�1

bi � 1 there has to be bi � 1=k for alli � 1; . . . ; k, completing the proof.

Now we consider simultaneously the idea of preserving a pointwise order fortwo estimators as induced by their expectations and the conception of combiningestimators. There is

var �~m� � varPki�1

bi bmi

� ��Pk

i�1b2

i �s2a � s2

i �

�Pki�1

b2i s2

i ;

so defining

R�b� :�Pki�1

b2ibs2

i ; �> 0 a.e.� ; �15�

we have by R�b� a lower estimator in the following sense:

E�Q�b�� var �~m�� E�R�b��:

Hence, a truncated estimator of var �~m� can be given by the order (pointwise)preserving estimator

qm�b� � max fQ�b�;R�b�g; �16�


or, more generally, by a linear interpolation near the switching point. That means,for some real values A and B with

0 < A � 1 � B �17�let denote

L�b� :� min 1;max 0;�Q�b�=R�b�� ÿ A

Bÿ A

� �� ; �18�

where x=0 :� �1 for x � 0 and x=0 :� ÿ1 for x < 0, then 0 � L�b� � 1, andwe define the convex combination of Q and R:

qL�b��b� :� L�b� Q�b� � f1ÿ L�b�g R�b� ; �19�and, regarding b; t; and var� bs2

i �; i � 1; . . . ; k; as known, the variance of qL canapproximately be estimated (L�b� random) by

Vq :� L�b�2 VQ�b� � f1ÿ L�b�g2 Pki�1

b4i var � bs2

i �

� 2 � L�b�f1ÿ L�b�gPki�1

wi�b� b2i var � bs2

i �

�: v�b; t; var � bs21�; . . . ; var � bs2

k�� ;

�20�

where the last term in (20) corresponds to the approximate covariance of L � Q and�1ÿ L� � R, and V Q�b� is given in (14). Note that L � 0 for Q � A � R and L � 1for Q � B � R, if A < B; for A � B � 1 : qL � qm. Further, qL > 0, a. e.

Applying again the Patnaik-approximation, cf. (13), an approximate c2�E~n�-dis-tribution of �E~n� � qL�b��b�=EqL�b��b� is given for

~n � ~nqL�b��b� � 2 � fEqL�b��b�g2

Vq: �21�

If the bs2i are stochastically independent of the bmj; i; j � 1; . . . ; k; then qL and ~m are

also stochastically independent, cf. theorem 4.2 (ii), and Vq is a better estimate ofvar �qL� with respect to bias.

For a realization, now b is replaced in qL and in ~nqL by its estimator, cf. (7),

b � 1

t�bt1; . . . ; btk�0

and with

q :� qL�b��b� ; cf: �19� ; �22�

v�q� :� v�b; t; cvar � bs21�; . . . ; cvar � bs2

k�� ; cf: �20� ; �23�where cvar � bs2

i � denotes an estimator of var � bs21�; i � 1; . . . ; k, the results above can

be summarized to state the test procedure in a compromised form as follows:


Theorem 4.3: Under m � 0, there holds for the test statistic, cf. (8), (22), (23),

T2 :� m��qp �appr:

t�n� ; (24)

with

n � 2 � q2

v�q� : �25�

Remark 4.1: The approximation in (24) is better for the case that the bs2i are

stochastically independent of bmj, implying also that bbi depends on bmj only viabs2a; i; j � 1; . . . ; k.

Remark 4.2:(i) If an estimate of var � bs2

i � is not reported in the i-th study or can not becomputed upon the knowledge of bmi and bs2

i , for instance by the delta-meth-od, cf. sec. 6, then in the formulas above we put cvar � bs2

i � � 0; i � 1; . . . ; k;defining then in n : x=0 :� 1 for x > 0 and the t�1�-distribution as thestandard normal distribution, cf. also sec. 5 and 6.

(ii) Defining, cf. (15),

nR � 2R�b�2Pki�1

b4i cvar � bs2

i �; �26�

then c2�EnR� is an approximate distribution of �EnR� � R�b�=E�R�b��,cf. (13), so that for 0 < 2j < 1, an approximate �1ÿ 2j�-confidence inter-val for E�R�b�� is given by

nR

c2�nR�1ÿjR�b� � E�R�b�� R�b� nR

c2�nR�j;

where c2�n�j denotes the j-quantile of the c2�n�-distribution.Now E�R�b�� var �~m� under s2

a � 0, and E�R�b�� E�Q�b��, so onecan use the bounds of that interval to define the `changing points' A and B,

A � nR

c2�nR�1ÿj; and B � nR

c2�nR�j; �27�

where in the following sections we have chosen j � 0:25. There also wehave ignored partially the knowledge about the estimates of var � bs2

i � andworked with fixed values for A and B: A � 0:8;B � 1:2, andA � 0:95;B � 1:05.

The different choices of A and B 'in the neighbourhood of 1' do not seemto have much influence on the results.

By theorem 4.1 we get, replacing in S�b�, cf. (10), the parameter b by its esti-mator b stated above, a test in the following way.


Theorem 4.4: Under m � 0, there holds for the test statistic

T3 :� m��1

k ÿ 1� S�b�

r �appr:tkÿ1 : (28)

In simulations, T3 shows a performance comparable to that of T2, of which severalvariants are demonstrated in the following sections 5 and 6.

5. The Unbalanced Heteroscedastic 1-way Random ANOVA Model

Here let us consider the model

yij � m� ai � eij ; i � 1; . . . ; k; j � 1; . . . ; ni � 2; �29�where a1; . . . ; ak; e11; . . . ; e1n1 ; . . . ; eknk are stochastically independent normally dis-tributed random variables with E�ai� � E�eij� � 0; var �ai� � s2

a � 0;var �eij� � x2

i > 0, and m � E�yij�; i � 1; . . . ; k; j � 1; . . . ; ni.For the ith estimate bmi; i � 1; . . . ; k, of m we get

bmi �1

ni

Pni

j�1yij � N�m;s2

a � s2i � ; with s2

i �1

nix2

i ;

and s2i is estimated bybs2

i �1

ni� 1

ni ÿ 1

Pni

j�1�yij ÿ bmi�2 ; i � 1; . . . ; k ;

which is stochastically independent of mi; i � 1; . . . ; k; and further

var � bs2i � � 2

1

ni ÿ 1�s2

i �2, of which the best unbiased estimator, cf. Hartung and


Table 5.1

Unbalanced heteroscedastic random 1-way ANOVA model: sample designs D�d; k�, ford � 1; 2; 3; 4, and k � 3, k � 6, with s2

a � 0:1; 1:0, and 10 for the simulation results intable 5.2

D�d; k� k � 3 k � 6

d i 1 2 3 1 2 3 4 5 6

1 ni 5 10 15 5 10 15 5 10 15x2

i 1 3 5 1 3 5 1 3 5

2 ni 10 20 30 10 20 30 10 20 30x2

i 1 3 5 1 3 5 1 3 5

3 ni 5 10 15 5 10 15 5 10 15x2

i 5 3 1 5 3 1 5 3 1

4 ni 10 20 30 10 20 30 10 20 30x2

i 5 3 1 5 3 1 5 3 1

Voet (1986), is given by

cvar � bs2i � � 2 � 1

ni � 1� bs2

i �2; i � 1; . . . ; k: �30�

In k � 3 illustrative samples of sizes ni, respectively 2ni, and in k � 6 samplesby independent replications of the first samples, cf. table 5.1, with different con-stellations of the residual variances x2

i , for s2a � 0:1; 1:0, and 10, a simulation

study (10 000 runs each) is performed in order to get estimates a of the actuallevels attained by the various test statistics, at the prescribed nominal levela � 0:05, for the one-sided hypothesis H01 : m � 0 against H11 : m > 0 and forthe two-sided hypothesis H02 : m � 0 against H12 : m 6� 0. This is done for the


Table 5.2

Unbalanced heteroscedastic random 1-way ANOVA model: realized significance levels a%at the nominal level a � 5% for the one sided H01 [1st number] and for the two-sidedH02 �second number; cursive] with the test statistics T1; T2; 1; T2; 2 and T2; 3 in the sampledesigns D�d; k� from table 5.1

a � 5% a%

s2a d H0l k � 3 k � 6

l T1 T2; 1 T2; 2 T2; 3 T1 T2; 1 T2; 2 T2; 3

0.1 1 1 8:0 6:1 5:8 5:2 7:8 4:7 5:2 4:52 10:4 7:4 7:6 5:9 9:8 6:0 6:1 5:2

2 1 7:7 5:3 5:5 5:2 7:1 4:9 4:6 4:72 9:6 7:0 7:4 6:8 8:6 5:3 4:7 5:6

3 1 9:2 6:6 7:2 6:2 8:4 4:6 4:9 5:32 12:4 9:5 9:3 8:3 11:0 6:0 6:6 6:9

4 1 11:0 7:8 6:7 7:5 8:7 4:8 4:8 4:72 15:7 11:7 9:8 11:5 12:3 6:2 6:6 6:4

1.0 1 1 11:4 5:5 5:2 5:1 8:0 4:9 4:5 4:92 16:7 7:7 7:6 6:9 10:9 5:1 4:6 4:7

2 1 12:2 5:1 5:3 5:1 8:3 4:8 5:1 5:22 18:4 6:3 6:9 6:8 11:3 4:9 4:9 4:9

3 1 12:8 6:2 5:9 6:5 9:5 4:4 4:2 4:42 20:2 10:3 10:5 10:4 13:1 4:3 4:2 4:5

4 1 13:2 5:0 4:9 5:2 9:8 4:8 4:5 4:52 20:6 8:1 8:2 8:4 13:6 4:2 4:3 4:4

10 1 1 12:2 4:9 5:0 5:0 8:7 5:2 4:9 5:22 19:1 5:6 5:6 5:5 12:2 5:3 5:1 5:2

2 1 12:4 5:4 4:8 4:9 8:3 4:7 5:0 4:92 19:3 5:5 5:2 5:1 11:2 5:0 5:0 5:2

3 1 13:4 4:4 5:0 4:4 9:4 4:9 4:9 4:52 21:4 5:6 6:1 5:4 13:6 5:1 5:6 5:2

4 1 13:7 5:0 4:7 4:8 9:8 5:0 5:1 4:92 21:6 5:9 5:5 5:2 14:1 5:5 5:3 5:2

commonly used statistic T1, cf. sec. 3, and for some variants of T2, cf. (24),where T2; 1 � T2 with A � 0:8;B � 1:2; T2; 2 � T2 with A � 0:95;B � 1:05, and

in the correspondent test procedures, the knowledge of an estimate for var� bs2i �

is ignored for both, i.e. we put there cvar� bs2i � � 0; i � 1; . . . ; k, cf. remark 4.2

(i). Finally, T2; 3 � T2 with A;B chosen in accordance with remark 4.2 (ii),

where j in (27) is taken as j � 0:25; here the information cvar � bs21�, given by

(30), is used.The simulation results are shown in table 5.2, where in each package the first

number gives a for H01 and the second number (cursive) a for H02.For T1, the realized levels a are lying partially much above the 5%-level,

whereas the results of the other test procedures don't differ very much and aresatisfactory, at the whole, cf. table 5.2.

6. The Probability Difference Method

Just for demonstrating the application of the procedures discussed in sec. 3 and 4to data not following an ANOVA model, we here consider the problem of testingthe difference of two proportions. To this let for i � 1; . . . ; k and j � 1; 2 the sto-chastically independent random variables zij be binomially distributed with param-eters nij � 2 and pij; 0 < pij < 1, where it is assumed that in all studies i the prob-ability difference is identical: pi1 ÿ pi2 �: m. An unbiased estimator of pij isbpij � zij=nij with var � bpij� � pij�1ÿ pij�=nij �: s2

ij, and bpij�appr:N�p; s2

ij�. For the dif-ference of the empirical rates there is allowed that one really observesbmi � ai � �cpi1 ÿcpi2� �: cpi1*ÿcpi2*; i � 1; . . . ; k; �31�where the stochastically independent ai�appr:

N�0; s2a�; s2

a � 0, corresponding to`random deviations' of the assumption of identical probability differences, repre-sent the random interaction effects, also assumed as stochastically independent ofthe zij; i � 1; . . . ; k, j � 1; 2. Thus we havebmi�

appr:N�m; s2

a � s2i �; s2

i :� s2i1 � s2

i2; i � 1; . . . ; k : (32)

An unbiased estimator of s2ij isbs2

ij �1

nij ÿ 1� bpij ÿ bp2

ij�; �33�

and for its variance it is sufficient here to take the approximation given by thedelta-method,

var � bs2ij� �

appr: @ bs2ij

@ bpij

��pij

0@ 1A2

� s2ij �

1ÿ 2pij

nij ÿ 1

� �2

� 1

nij� pij � �1ÿ pij� ; �34�

which is estimated by replacing pij with bpij.


Now bs2i � cs2

i1 �cs2i2 and var � bs2

i � � var �cs2i1� � var �cs2

i2�, so that (34) yields anestimate cvar � bs2

i �.The estimates above are only approximately correct, if one uses bpij* instead ofcpi1, i.e. if ai is different from zero; note that then bpij can not be identified.Hence, all our test procedures can approximately be applied. This is illustrated

in a simulation study (10 000 runs each) for k � 3 groups of paired samples�ni1; ni2�, with sizes (15, 25), (20, 15), (30, 20), and for k � 6 groups by an inde-pendent replication of the first samples to get estimates a of the actual levelsattained by the various test statistics ±± with the prescribed nominal level a � 0:05±± for the one-sided hypothesis H01 : m � 0 versus H11 : m > 0 and for the two-sided hypothesis H02 : m � 0 versus H12 : m 6� 0, where under 0 � m � pi1 ÿ pi2,the probabilities are taken as pij � 0:2, whereas s2

a here is chosen as 0.01, 0.1,and 0.5. The test procedures, corresponding to T1; T2; 1; T2; 2 and T2; 3, are chosenidentically as in sec. 5, where for T2;3 here the approximate estimate cvar� bs2

i �derived from (34) is used. The results are put together in table 6.1.

Now again we observe the realized levels a for T1 partially to increase muchover the 5%-level, and that the T2-variants produce quite satisfactory results.

It should be remarked that with the log odds ratio and log relative risk criteriain samples of the order above we observe a much more broader range of defi-ciency with the commonly used method T1, starting for small s2

a with a clearunderestimation and ending for larger s2

a with a strong overestimation of the sig-nificance levels.


Table 6.1

Probability difference method: realized significance levels a% at the nominal level a � 5%for the one sided H01 [1st number] and for the two-sided H02 [2nd number, cursive] withthe test statistics T1;T2; 1; T2; 2 and T2; 3 in the sample designs SD�k� for k � 3 and k � 6.

Sample designs SD�k� for s2a � 0:01; 0:1; 0:5, and pij = 0.2

k � 3: (15, 25), (20, 15), (30, 20)k � 6: (15, 25), (20, 15), (30, 20), (15, 25), (20, 15), (30, 20)

a � 5% a%

s2a H0l k � 3 k � 6

l T1 T2; 1 T2; 2 T2; 3 T1 T2; 1 T2; 2 T2; 3

0.01 1 7:3 5:0 5:0 4:8 6:7 4:9 4:8 4:82 8:8 6:0 6:1 5:8 7:7 5:1 5:0 4:9

0.1 1 11:8 5:5 5:6 5:3 8:1 5:1 5:5 5:02 17:4 7:1 6:8 6:8 11:1 5:1 5:0 5:0

0.5 1 12:1 4:5 4:3 4:3 8:4 5:1 5:1 5:12 18:7 3:9 3:7 4:0 11:7 5:2 5:4 5:2

7. Final Remark

In this paper we have shown the consequences of the commonly used method fortesting hypotheses about the common effect in combining estimates from severalindependent studies, experiments or centres of a multi-centre trial, where the oc-currence of a random interaction of response with centres or studies is included inthe considerations.

We recommend the use of the proposed alternative test procedures with thebetter approximating test distributions.

Acknowledgement

Thanks are due to Kepher Makambi, Dortmund University, for assistance with thesimulations.

References

Biggerstaff, B. J. and Tweedie, R. L., 1997: Incorporating variability in estimates of heterogeneity inthe random effects model in meta-analysis. Statistics in Medicine 16, 753±±768.

BoÈckenhoff, A. and Hartung, J., 1998: Some corrections of the significance level in meta-analysis.Biometrical Journal 40, 937±±947.

Cochran, W. G., 1937: Problems arising in the analysis of a series of similar experiments. Journal ofthe Royal Statistical Society B 4, 102±±118.

Cochran, W. G., 1954: The combination of estimates from different experiments. Biometrics 10,101±±129.

DerSimonian, R. and Laird, N., 1986: Meta-analysis in clinical trials. Controlled Clinical Trials 7,177±±188.

Draper, D., Gaver, D. R. jr., Goel, P. K., Greenhouse, J. B., Hedges, L. V., Morris, C. N., Tucker,J. R., and Waternaux, C. M., 1992: Combining Information. National Academic Press, Washing-ton, D. C.

Hardy, R. J. and Thompson, S. G., 1996: A likelihood approach to meta-analysis with random effects.Statistics in Medicine 15, 619±±629.

Hartung, J., 1981: Non-negative minimum biased invariant estimation in variance component models.The Annals of Statistics 9, 278±±292.

Hartung, J., 1999: A note on combining dependent tests of significance. Biometrical Journal, toappear.

Hartung, J. and Voet, B., 1986: Best invariant unbiased estimators for the mean squared error ofvariance component estimators. Journal of the American Statistical Association 81, 689±±691.

Hedges, L. V. and Olkin, I., 1985: Statistical Methods for Meta-Analysis. Academic Press, Orlando.Li, Y., Shi, L., and Roth, H. D., 1994: The bias of the commonly used estimate of variance in meta-

analysis. Communications in Statistics ±± Theory and Methods 23, 1063±±1085.Mathai, A. M. and Provost, S. B., 1992: Quadratic Forms in Random Variables. Marcel Decker,

New York.Patnaik, P. B., 1949: The non-central c2- and F-distributions and their applications. Biometrika 36,

202±±232.


Whitehead, A. and Whitehead, J., 1991: A general parametric approach to meta-analysis of random-ized clinical trials. Statistics in Medicine 10, 1665±±1677.

Yates, F. and Cochran, W. G., 1938: The analysis of groups of experiments. Journal of AgriculturalSciences 28, 556±±580.

Joachim Hartung Received, November 1998Department of Statistics, SFB 475 Revised, August 1999University of Dortmund Accepted, August 1999Vogelpothsweg 87D-44221 DortmundGermany


Documents

An alternative method for meta analysis