Estimating Incremental Cost- Effectiveness Ratios from Cluster Randomized Intervention Trials M. Ashraf Chaudhary & M. Shoukri

Estimating Incremental Cost-Effectiveness Ratios from Cluster Randomized Intervention Trials

M. Ashraf Chaudhary & M. Shoukri

Sep 28, 2005 2CREATE Biostat Core Meeting, Cape Town

Incremental Cost-Effectiveness Ratio

E2E1

C2C1

R

21

21ˆEECC

R

Assuming numerator and denominator positive, R is the cost per additional outcome achieved by the treatment.


Statistical Properties

• Biased• Consistent• Positively skewed• Limiting distribution is normal• Very sensitive to changes in the denominator• No exact method of estimating variance


Methods

Parametric:

• 1) Taylor Series Expansion

• 2) Fieller's Method

Non-Parametric Bootstrap:

• 3) Percentile Method

• 4) BCa


Cluster Randomized Trials

i

ii

m

CmC

1

111

1111

212

1 11 CC

C mmk

2)(1

2)(1

2)(1

1eCbC

bCC

11111 ),cov( ECEC

For fixed cluster size , so that .11 mm i 111 kCC i


Coefficients of Variation

2

21

2C2

2C1

CC

sscnn

221

2E2

2E1

EE

sscdd

2121

E2C22111

EECC

ssrssrc EC

nd


Taylor Series Method

Large sample normal approximation yields,

2/1)ˆ(ˆ RvzRR

• Inaccurate if is far from normal or the sample is not large enough.• Affected by extreme values• Interval is symmetrical even if is not.

R̂

R̂

ndddnn2 2ˆ)ˆ( cccRRv


Fieller's Method

dd2

2/12ndddnn

2ndddnnnd

2

1)()2()1(ˆ

czccczccczcz

RR

» Not symmetrical if is not » Assumes bivariate normality» Assumes unbiasedness of » Imaginary roots for certain samples» Volatility of making a negative quantity leading in correct intervals.

R̂

R̂

ddcddcz21


Non-Parametric Bootstrap

1. a- resample observations or b- clusters or c- two stage bootstrap?a – not appropriate as observations within cluster are correlatedb – theoretically appropriate c – mathematically preferable but assumes no correlation within clusters

2. We use approach b and resample clusters retaining all observations in resampled clusters

3. A cluster level summary data is prepared with mean cost and mean effect in each cluster

4. boot package in R was used to implement bootstrap stratified by study arm and estimate intervals

5. bootstrap replications =1999


Percentile Method

• If is normal, agrees with delta method

• Interval not symmetrical if is not

• Transformation respecting

• Range preserving

• Robust to extreme replication

• No adjustment for bias due to for asymmetry

R̂

R̂


BCa All the advantages of percentile method

Adjustment for bias due to asymmetry

More accurate in terms of coverage


Simulation (1)– Balanced CRT with two treatment groups

• clusters of fixed size

• varying but equal ICC in cost in two groups and zero for the effectiveness

– Cost - Random effects model framework used separately for each arm, between and within cluster effects assumed normal

– Effect - A correlated normal variable generated within each cluster and dichotomized – 0.20 in control and 0.40 in treatment and cost-effect correlation 0.30 in each group

– Mean cost in control and treatment set at $20.00 and $30.00

– R = $50.00

– Box-Cox transformation of normal for positively skewed cost data


Simulation (2)

– 54 Scenarios:• number of clusters (12, 24, 48)• cluster sizes (25, 50, 100)• ICC (0.25, 0.10, 0.01)• Normal/positively skewed cost data

– Sum of between and within components of variance in cost data constrained to be 100

– 2000 simulation replications of data under each scenario– All four types of intervals computed for each replication– The programming for simulation and analysis in R


Skewness & VariabilityRho Clusters Size Skewness nnc

ddc ndc

0.25 12 25 0.40 0.47 0.44 0.31 0.25 12 100 0.08 0.46 0.30 0.25 0.25 48 25 0.10 0.33 0.30 0.21 0.25 48 100 0.17 0.32 0.21 0.18

0.01 12 25 0.63 0.31 0.43 0.26 0.01 12 100 0.26 0.24 0.30 0.19 0.01 48 25 0.24 0.21 0.30 0.18 0.01 48 100 0.04 0.17 0.21 0.13


Skewness & Variability

– The coefficient of skewness estimated as 3(mean-median)/sd

– Distributions of the simulation replications of are generally positively skewed.

– Increasing the number of clusters and the cluster sizes leads to more symmetrical distribution of

– Distribution of is more skewed with smaller ICC – with more within cluster variability

– The order of the numerator and denominator coefficients of skewness is about the same as observed in most cost-effectiveness studies.

R̂R̂


Histograms

k=12 m=25

Histogram of R2

R2

Fre

qu

ency

20 40 60 80 100 120

050

10

015

020

025

0

Histogram of R2

R2

Fre

qu

en

cy

40 60 80 100

05

01

00

15

02

00

25

0

k = 12, m = 100

Histogram of R2

R2

Fre

qu

en

cy

20 40 60 80

05

01

00

15

0

Histogram of R2

R2

Fre

qu

en

cy

40 50 60 70

01

00

20

03

00

k = 48 m = 25

Histogram of R2

R2

Fre

qu

en

cy

30 40 50 60 70 80

050

10

015

020

025

0

Histogram of R2

R2

Fre

qu

en

cy

40 50 60 70

01

00

20

03

00

40

0

k = 48 m = 100

Histogram of R2

R2F

req

uency

30 40 50 60 70

050

10

015

020

025

030

0

Histogram of R2

R2

Fre

qu

en

cy

45 50 55

05

01

00

15

0

Rho = 0.25 Rho = 0.01


95% Confidence Intervals

Rho Clus- Cluster Skew- Methods

ters Size ness Delta Fieller’s Percentile BCa

in R̂ Cov. Width Shape Cov. Width Shape Cov. Width Shape Cov. Width Shape

25 0.41 93.8 53.2 1.0 91.4 62.5 1.6 92.5 63.6 1.7 92.3 64.7 1.7 50 0.07 93.4 44.7 1.0 92.6 46.8 1.3 92.3 48.8 1.3 92.7 49.2 1.3 12

100 0.10 92.0 40.6 1.0 91.7 41.4 1.1 91.9 42.8 1.1 92.1 43.0 1.2

25 0.22 94.6 36.1 1.0 93.2 38.0 1.3 94.2 40.1 1.4 94.5 40.2 1.4 50 0.12 93.2 31.1 1.0 93.0 31.7 1.2 94.1 33.6 1.2 93.9 33.7 1.2 24

100 0.11 93.2 28.7 1.0 93.1 29.0 1.1 94.3 30.6 1.1 94.3 30.7 1.1

25 0.18 94.2 24.9 1.0 93.6 25.5 1.2 94.6 27.3 1.3 94.3 27.4 1.3 50 0.21 93.2 21.9 1.0 92.6 22.1 1.1 94.3 23.7 1.1 94.3 23.8 1.1

0.25

48

100 0.00 92.6 20.3 1.0 92.5 20.4 1.0 94.1 21.8 1.1 93.6 21.8 1.1

25 0.43 94.3 44.6 1.0 93.4 52.9 1.8 92.7 52.3 1.8 92.4 53.5 1.9 50 0.11 94.6 34.3 1.0 93.2 36.2 1.4 93.8 37.3 1.4 93.5 37.7 1.4 12

100 0.17 92.6 28.5 1.0 92.1 29.1 1.2 92.9 30.3 1.2 93.1 30.5 1.2

25 0.28 93.8 30.1 1.0 93.6 32.0 1.5 93.5 33.2 1.5 93.9 33.3 1.5 50 0.10 93.2 23.5 1.0 92.7 24.1 1.2 94.2 25.6 1.3 93.9 25.6 1.3 24

100 0.11 92.7 20.1 1.0 92.6 20.3 1.1 94.4 21.7 1.1 94.0 21.7 1.1

25 0.27 94.4 20.8 1.0 93.4 21.3 1.3 94.5 22.5 1.3 94.6 22.6 1.3 50 0.15 92.3 16.6 1.0 92.4 16.7 1.2 94.3 17.9 1.2 94.2 18.0 1.2

0.10

48

100 0.04 93.3 14.2 1.0 93.0 14.2 1.1 94.9 15.4 1.1 94.6 15.4 1.1

25 0.49 94.1 40.3 1.0 94.2 51.0 2.0 92.0 47.0 2.0 91.7 47.6 2.1 50 0.28 94.7 27.0 1.0 94.5 28.9 1.6 93.2 28.0 1.6 92.9 28.3 1.6 12

100 0.21 93.6 19.1 1.0 93.6 19.7 1.3 92.7 19.4 1.3 92.5 19.6 1.4

25 0.29 94.2 26.6 1.0 95.0 28.5 1.6 93.9 28.0 1.6 93.6 28.2 1.6 50 0.23 94.3 18.5 1.0 94.6 19.0 1.4 94.2 18.9 1.4 94.1 19.0 1.4 24

100 0.12 94.7 13.3 1.0 94.5 13.5 1.2 94.5 13.7 1.2 94.1 13.7 1.2

25 0.19 94.8 18.4 1.0 94.2 18.9 1.4 93.8 18.9 1.4 93.5 18.9 1.4 50 0.10 95.3 13.0 1.0 95.8 13.2 1.2 95.4 13.3 1.2 95.4 13.3 1.3

0.01

48

100 0.10 94.7 9.4 1.0 94.9 9.4 1.2 94.9 9.7 1.2 94.8 9.7 1.2

Overall 93.7 26.7 1.0 93.4 28.4 1.3 93.8 29.1 1.3 93.6 29.3 1.4


Results (1) Coverage: Proportion of intervals containing R

Width:

Shape:

ICER is highly unstable with small effect difference. If |R| < 0.0001 then R= +/- 0.0001. Replications of R are bounded by Chebychev’s inequality -

the probability of having a more extreme replication < 1/100. Possibly no effect on bootstrap confidence intervals.

lowup RR ˆˆ )ˆˆ/()ˆˆ( lowup RRRR


Results (2)

In terms of coverage all the four methods seem to perform equally well

Generally the coverage falls below the nominal value of 0.95

The coverage does not seem to improve with increase in sample size

The lower levels of ICC seem to be associated with better coverage of the intervals.

The width of the intervals shrinks with the increase in the number of clusters and the cluster size

The shape of the intervals tends to be more even with large cluster and of bigger size.


Results (3)

The shapes of the Fieller’s, Percentile and the BCa intervals are similar and reflect the direction of asymmetry in the distribution of ICER.

The distribution of simulation replications of R is more symmetrical as the number of clusters and the size of the clusters increase. This translates to Fieller’s, Percentile and BCa intervals to be more even around the R. On top of this trend, these intervals are more symmetrical with smaller ICC. The same trend is evident in the width of the intervals. Further the width of the intervals shrinks with reduced levels of ICC.

It is evident that confidence intervals ignoring the ICC will be shorter in length and would not provide the desired coverage probability and would be misleading.


Results (4) The big question – why all the methods perform equally well in

terms of coverage when the distribution of replications of R is clearly positively skewed? The overall sample is big in each combination? The cost data are assumed normally distributed? The cluster specific means are used in the analysis leading to normality by

CLT?


References1. Chaudhary, M.A., and Stearns, S.C., "Estimating Confidence

Intervals for Cost Effectiveness Ratios: An Example from a Randomized Trial", Statistics in Medicine, Vol. 15, 1447-1458 (1996)

2. O'Brien, B.J., M.F. Drummond, R.J. LaBelle, A. Willan 'In Search of Power and Significance: Issues in the Design and Analysis of Stochastic Cost-Effectiveness Studies in Health Care', Medical Care, 32(2):150-163 (1994)

3. Mullahy, J. and W. Manning 'Chapter Eight: Statistical Issues in Cost-Effectiveness Analyses', in Costs, Benefits, and Effectiveness of Pharmaceuticals and Other Medical Technologies, edited by Frank Sloan, Cambridge University Press. (1995)

4. Cochran, W. G. Sampling Techniques, John Wiley and Sons. N.Y. (1977)

5. Efron, B., and Tibshirani, R.J. An Introduction to the Bootstrap, Chapman and Hall, N.Y. (1993)

Documents

Estimating Incremental Cost- Effectiveness Ratios from Cluster Randomized Intervention Trials M. Ashraf Chaudhary & M. Shoukri