GMDS Jahrestagung

Essen, September 7-10, 2009

Network Meta-analysis, Indirect Comparisons:

are they so controversial ?

Tony Ades

with thanks to

Debbi Caldwell, Sofia Dias, Guobing Lu, Nicky Welton

Academic Unit of Primary Health Care,

Department of Community Based Medicine

Network meta-analysis

1. What is it ?

2. Is it complicated ?

3. Does it make difficult assumptions ?

4. Is it biased ?

Pair-wise meta-analysis

One or more trials comparing treatments B and C

Inferences about relative treatment effect dBC

B C

Indirect Comparisons

One or more trials comparing treatments A and B

One or more trials comparing treatments A and C

Inferences about relative treatment effect dBC ????

.

A

C

B

Indirect Comparisons

One or more trials comparing treatments A and B

One or more trials comparing treatments A and C

Simultaneous Inference about relative treatment effects

dAB, dAC, dBC

A

C

B

Indirect Comparisons: K treatments

Simultaneous inference on all K(K-1)/2 pair-wise contrasts

dAB, dAC, dAD dAE, dBC, dBD dBE dCD dCE , dDE

A

C

B

D

EA

B D

C

E

Mixed Treatment Comparisons = Multiple

treatment comparisons = Network Meta-analysis

A

C

B

D

E

Any connected

network:

simultaneous

inference on all

K(K-1)/2 contrasts.

Combines “direct” and

“indirect” evidence on

each contrast.

Loops: possibility of

“inconsistency”.

Example: Smoking Cessation

(Hasselblad, 1998)

• 24 trials, 4 treatments, and 50 data points (two 3-arm trials, the remaining 2-arm trials)

• Treatments:

A: No Contact

B: Self-Help

C: Individual Counselling

D: Group Counselling

Smoking Cessation Data Structure

1 trial

1 trial

2 trials

15 trials

1 trial

1 trial

1 trial

2 trials

A C D

B C D

A B

A C

A D

B C

B D

C D

MTC model for 4 treatments

Binomial data, treatments A,B,C,D

Data informing all 6 contrasts, AB, AC, AD, BC, BD, CD

Take A as the reference treatment

Make dAB, dAC, dAD the basic parameters, to be estimated.

Remaining contrasts are functional parameters:

dBC= dAC – dAB

dBD= dAD – dAB ”Consistency equations”

dCD= dAD - dAC

MTC model for 4 treatments

Binomial data, treatments A,B,C,D

Data informing all 6 contrasts, AB, AC, AD, BC, BD, CD

Take A as the reference treatment

Make dAB, dAC, dAD the basic parameters, to be estimated.

Remaining contrasts are functional parameters:

dBC= dAC – dAB

dBD= dAD – dAB ”Consistency equations”

dCD= dAD - dAC

A B C D

Is it complicated ?

Model for pair-wise meta-analysis. Treatments k, Trials j, comparing treatments A and B:

, ,2

,

, , ,2

Link Function: Logit( ) ( )

Random Effect: ~ ( , )

Likelihood: ~ Binomial( , )

Priors: , , ~ ( , )

j k j j AB

j AB AB

j k j k j k

j AB

p I k A

N d

r p n

d Dist

treatment effect

Is it complicated ?

Model for MTC. Trials j, treatments k in {A,B,C,..…..S }: comparing treatments b and k:

, , ,2

, ,

, , ,2

Link Function: Logit( ) ( )

Random Effect: ~ ( , )

Likelihood: ~ Binomial( , )

Priors: , , ....... , ~ ( , )

j k j j b k

j b k Ak Ab

j k j k j k

j AB AC AS

p I k b

N d d

r p n

d d d Dist

K-1 treatment effects: “basic” parameters

Same code for MTC and pair-wise meta-

analysis

A single program code can estimate

• Pair-wise meta-analysis

• Indirect comparisons

• Mixed Treatment Comparisons

= Network Meta-analysis

• Multi-arm trials

• Any combination of the above

Does MTC require difficult assumptions ?

Pair-wise meta-analysis assumes that the trial specific treatment effects are “exchangeable”

MTC assumes that the trial specific treatment effects

are “exchangeable” .

…. meaning that, if ALL the trials had included ALL the treatments, then each trial would have estimated the same, exchangeable effects.

MTC is based on a “missing at random” assumption

Consistency not an additional assumption: but follows from exchangeability

,j AB

, , ,, ,j AB j AC j AD

Is it biased ?

“Between-trial [ ie Indirect] comparisons are

unreliable. Patient populations may differ in

their responsiveness to treatment. Therefore

an apparently more effective treatment may

have been tested in a more responsive

population”Cranney, Guyatt et al. Endocr Rev 2002,

23; 570-8. Summary of meta-analyses of

therapies for postmenopausal osteoporosis

Is it biased ?

“Placebo controlled trials lacking an active control give little useful information about comparative effectiveness. Such information cannot reliably be obtained from cross-study comparisons, as the conditions of the studies may have been quite different”

International Council of Harmonisation E10 2.7.1.4

IS it biased ?

“Indirect comparisons are observational studies

across trials, and may suffer the biases of

observational studies, for example confounding”

Cochrane Handbook for systematic reviews of interventions 4.2.4.

Cochrane Library Issue 2

… But Victor, Egger, and Moher have each claimed

that pair-wise M-A is “observational”.

Are indirect estimates vulnerable to bias ?

If is an unbiased estimate of dAC, and

is an unbiased estimate of dAB, then

must be an unbiased estimate of dBC

“Indirect” can only be biased if “direct” is biased !

ˆ ˆ ˆIndirect Direct Direct

BC AC ABd d d

ˆDirect

ACd

ˆDirect

ABd

ˆ Indirect

BCd

“…. But suppose the AC and the AB trials were

on different patient populations? “

If we want a pair-wise meta-analysis to provide an

estimate of dBC in a patient group X, then we

should include only trials on group X.

If we want an indirect comparison to provide an

estimate of dBC in a patient group X, then we

should include only trials on group X.

“…. But suppose there are UNRECOGNISED

treatment modifiers in the AC trials, and not in the AB

trials ? “

Good question!

Heterogeneity is very common in pair-wise meta-analysis, and indicates the presence of often unrecognised relative effect modifiers (UREMs)

At the same time, the concern with indirect comparisons is that AB and AC trials might differ with respect to the presence UREMs.

Perhaps the heterogeneity issue in MA and the ‟bias‟ issue in IC are the same problem ?

We live in a world with UREMs !

… that is why we discover “heterogeneity” in

meta-analysis so often

Two kinds of UREM

(a) „Fixed‟ bias due to a trial-level treatment-by-covariate interaction: Some trials estimate , others h

… external validity problem

(b) „Random bias‟ associated with markers for low quality. Trials without such markers estimate .

With, they estimate b, b ~ N(B,2B)

… internal validity problem

„thought experiments‟

Suppose there is an Unrecognised Relative Effect Modifier

(treatment-by-trial interaction) with effect h and it is

present in a proportion p of infinitely large trials:

Or suppose there was a random UREM b ~ N(B,2B)

What estimate would we expect from a meta-analysis, or an

indirect comparison. How might the estimate deviate from

the target parameter ?

With „fixed‟ UREMs, what IS the

target parameter ?

UREM effect size: h

Treatment effects : no UREM , with UREM

UREM present in a proportion of trials: p

Target parameter *: !!!!!

* The pooled average.

Between trials Variance

(1 ) ( ) p p h ph

2(1 )p p h

h

Thought experiment (1):

Pair-wise Meta-analysis, N=1A meta-analysis with N=1 trials: UREM=h, with prob. p = 0.5

Target parameter, the pooled effect = +h /2

Long-run consideration of estimates from N=1

• 50% of the time it estimates

• 50% of the time it estimates + h

• Expected absolute bias = h / 2

• Probability that the trial estimates pooled effect target = 0

In a world with UREMs, a meta-analysis with N=1

(ie an RCT) is ALWAYS BIASED

Pair-wise Meta-analysis, N=2

A meta-analysis with N=2 trial: UREM=h, with prob. p = 0.5

Target parameter = +h /2

Four possible outcomes:

Prob Trial 1 Trial 2 Expected Bias:

MA - Target

Expected

Abs Bias

1 .25 ( h/2) h/2

2 .25 h ( h/2) ( h/2) 0

3 .25 h ( h/2) ( h/2) 0

4 .25 h h h ( h/2) h/2

Expected Absolute Bias h/4

Pair-wise Meta-analysis, N=3

A meta-analysis with N=3 trial: UREM=h, with prob. p = 0.5

Target parameter = +h /2

Eight possible outcomes:

Prob 3 trials Expected Bias:

MA - Target

Expected

Abs Bias

1 .125 , , , ( h/2) h/2

2 .375 , , h ( h/3) ( h/2) h/6

3 .375 , h, h ( 2h/3) ( h/2) h/6

4 .125 h, h, h h ( h/2) h/2

Expected Absolute Bias h/4

Pair-wise Meta-analysis Absolute Bias, Pooled Effect Target

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 5 10 15 20 25 30 35 40

Number of Trials

Ex

pe

cte

d A

bs

olu

te B

ias

, in

Un

its

of

Bia

s

p =0.5

Pair-wise Meta-analysis Absolute Bias, Pooled Effect Target

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 5 10 15 20 25 30 35 40

Number of Trials

Ex

pe

cte

d A

bs

olu

te B

ias

, U

nit

s o

f B

ias

p=0.5

p=0.4, 0.6

p=0.3, 0.7

p=0.2, 0.8

p=0.1, 0.9

Indirect comparisons – How might UREM work ?

Suppose :

1. A is „placebo‟, B and C active treatments in same

class.

2. A UREM acts on B and C equally, but not A.

Therefore UREM acts on dAB and dAC, but not dBC;

In BC trials it acts the same on each arm

Indirect Comps – 3rd comparator “similar”

We have trials of A vs C, B vs C, and wish to make inferences

about A vs B: dAB = dAC – dBC

UREM h present with probability 0.5, Target = dAB+h/2

In the absence of UREM,

AB trials estimate dAB, IC estimates dAC-dBC = dAB

In the presence of UREM,

AB trials estimate dAB +h, IC estimates dAC+h – dBC = dAB+h

Now consider IC syntheses with N AC and N BC trials ….

One AC trial & One BC trial, UREM=h, p=0.5

Four possible outcomes: Pooled effect Target: dAB+h/2

Prob AC trial BC trial Indirect

Comparison

Equal to Abs Bias

1 0.25 No UREM No UREM dAC-dBC dAB h / 2

2 0.25 No UREM UREM dAC-dBC dAB h / 2

3 0.25 UREM No UREM dAC+ h -dBC dAB + h h / 2

4 0.25 UREM UREM dAC+ h -dBC dAB + h h / 2

Expected absolute bias h / 2

Pairwise meta-analysis and Indirect Comparisons. p =0.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0 10 20 30 40

Number of Trials per comparison

Exp

ecte

d A

bso

lute

Bia

s,

un

its o

f h

Pair-wise MA

AND IC, 3rd comparator „similar‟ dAB = dAC – dBC

IC, 3rd comparator „different‟

dBC = dAC – dAB

Random effect modifiers.

Random bias has been associated with indicators of lower study quality (Schultz 1995)

Welton, Sterne: JRSS(A), 2009. Trials with lower quality indicators estimate: dAB + b, b ~ Normal(B,2)

What is known about this?

• Higher between-trials variation “lower quality” RCTs

• B depends on the meta-analysis.

• On average ….equivalent to an OR= 1.6

• .

• Over-estimates in favour of „newer‟ treatment

B 0.50.25

Random bias in MA, N=2, mean bias B, p=0.5

Prob Trial 1 Trial MA result Mean, var

of random

bias in MA

Exp Abs

Bias*

1 0.25 No UREM No UREM d, d 0, 0 0

2 0.25 No UREM UREM d, d+b B/2, 2/4 0.257 *

3 0.25 UREM No UREM d +b, d B/2, 2/4 0.257 *

4 0.25 UREM UREM d+b, d+b B, 2/2 0.502 *

Expected absolute bias 0.254

* By numerical integration

How might quality-related random bias work in

MA and IC ?

1. Bias operates in favour of “newest”, in „vulnerable‟ trials.

2. Assume treatments in chronological order A, B, C

3. Indirect comparison: infer BC effect from AC and AB trials.

“lower quality” AB trials are biased to favour B,

“lower quality” AC trials favour C.

dBC= dAC+bj – (dAB+bk), bj, bk ~ N(B, 2)

MEAN Biases CANCEL, but Random elements do not

… Song (J Clin Epi 2008) argues that IC may be LESS biased than direct comparisons

Pair-wise meta-analysis and Indirect Comparisons, Quality-

Related Random Bias with p=0.5, =0.25

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 40

Number of Trials

Exp

ecte

d A

bso

lute

Bia

s

Pair-wise MA

Indirect comparisons dBC=dAC-dAB

bj~N(B,2)

B=0.5

B=0.25B=0

B=0.5

B=0.25

B=0

But HEY! …. That‟s when biases SUBTRACT:

….. suppose they ADD !!

1. Bias operates in favour of “newest”, in „vulnerable‟

trials.

2. But now: infer AC effect from AB and BC trials.

“lower quality” AB trials are biased to favour B,

“lower quality” BC trials favour C.

dAC = dAB+bj + (dBC +bk), bj, bk ~ N(B, 2)

MEAN Biases Now ADD.

Quality-related Random Bias: Pairwise meta-analyis and Indirect

Comparisons based on ADDITION. p=0.5, =0.25

0

0.1

0.2

0.3

0.4

0.5

0.6

0 5 10 15 20 25 30 35 40

Number of Trials

Exp

ecte

d A

bso

lute

Bia

s

Indirect comparisons dAC=dAB+ dBC

B=0.5

B=0.25

B=0

Pair-wise MAB=0.5

B=0.25

B=0

bj~N(B,2)

Findings

1. „Error of generalisation‟ (bias ?) in pair-wise MA under UREM qualitatively same as „bias‟ in IC.

Findings

1. „Error of generalisation‟ (bias ?) in pair-wise MA under UREM qualitatively same as „bias‟ in IC.

2. Quantitatively, also comparable. With FIXED UREMs, bias in IC either identical to Pair-wise MA, or a little worse.

Findings

1. „Error of generalisation‟ (bias ?) in pair-wise MA under UREM qualitatively same as „bias‟ in IC.

2. Quantitatively, also comparable. With FIXED UREMs, bias in IC either identical to Pair-wise MA, or a little worse.

3. Random “quality-related” UREMs: IC can be better than MA when subtracting effects, but far worse when adding !

Findings

4. Think of single trials as MA‟s with N=1.

When we use RCTs to make predictions about the future,

in a world with UREMs, where it is hard to duplicate RCT

conditions, RCTs are effectively „observational‟ …

…. (but still far better than non-randomised studies!)

Findings

4. Think of single trials as MA‟s with N=1.

When we use RCTs to make predictions about the future,

in a world with UREMs, where it is hard to duplicate RCT

conditions, RCTs are effectively „observational‟ …

…. (but still far better than non-randomised studies!)

5. If heterogeneity in MA, never say “we need one (or more) big trial(s)”. They won‟t help !

Findings

4. Think of single trials as MA‟s with N=1.

When we use RCTs to make predictions about the future,

in a world with UREMs, where it is hard to duplicate RCT

conditions, RCTs are effectively „observational‟ …

…. (but still far better than non-randomised studies!)

5. If heterogeneity in MA, never say “we need one (or more) big trial(s)”. They won‟t help !

6. Similarly, if you are worried ICs are biased because of UREMs, don‟t say “we need a head-to-head trial”. It will be biased / un-interpretable / un-generalisable, too.

In conclusion

Pair-wise meta-analysis is a special case of

network meta-analysis, with K=2 treatments.

Every time we ask a new question about

network meta-analysis, we learn something

new about pair-wise meta-analysis