Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Empirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests Yemisi Takwoingi, Richard Riley and Jon Deeks
Outline
• Rationale • Methods • Findings • Summary
Motivating example
Gurusamy et al. Endoscopic retrograde cholangiopancreatography versus intraoperative cholangiography for diagnosis of common bile duct stones. Cochrane Database of Syst Rev 2013.
Motivating example 0.
00.
20.
4 0.
60.
81.
0
Sen
sitiv
ity
0.00.20.40.60.81.0
Specificity
Endoscopic retrograde cholangiopancreatographyIntraoperative cholangiography
Bivariate model 0.
00.
20.
4 0.
60.
81.
0
sens
itivi
ty
0.00.20.40.60.81.0
specificity
correlation
sensitivity
specificity
Models both logit(sensitivity) and logit(specificity) and the correlation between them
Combines two random effects meta-analysis of sensitivity and specificity in a single model
logit(sensitivity) and logit(specificity) are specified as random study effects
Bivariate model specification
Number not diseased
Number diseased
),(~ iSpifpitnbinomialitn +
),(~ iSeifnitpbinomialitp +
Level 1 (allows for within-study variability)
Level 2 (allows for between-study variability)
=∑
∑
2
2
,
, ,~BAB
ABA
B
A
iB
iA withNσσσσ
µµ
µµ
iAiSeit ,)(log µ= iBiSpit ,)(log µ=and
Bivariate model for comparative meta-analysis
Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be extended as
=∑
∑
++
2
2
,
, with,~BAB
ABA
iBB
iAA
iB
iA
ZvZv
Nσσσσ
µµ
µµ
Effect of test type on variance parameters can also be investigated
…results indicate that simpler hierarchical models are valid in situations with few studies or sparse data. For synthesis of sensitivity and specificity, univariate random effects logistic regression models are appropriate when a bivariate model cannot be fitted…
Applies to meta-analysis of a single test
Univariate model for comparative meta-analysis
Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be expressed as
=∑
∑
++
2
2
,
, with,~BAB
ABA
iBB
iAA
iB
iA
ZvZv
Nσσσσ
µµ
µµ 0
0
Bivariate model simplifies to 2 univariate random effects logistic regression models for sensitivity and specificity
Estimates from bivariate and univariate models comparing ERCP and IOC
Test Logit sensitivity
(SE)
Logit specificity
(SE)
Variance of random
effects for logit
sensitivity (SE)
Variance of random
effects for logit
specificity (SE)
Correlation of the logits
(SE)
Sensitivity (95% CI)
Specificity (95% CI)
Bivariate model
ERCP 1.55 (0.30) 5.35 (2.25) 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 (72.3–89.5) 99.5 (71.8–100)
IOC 7.06 (4.53) 4.15 (0.52) 16.7 (26.9) 0.25 (0.54) -0.73 (0.98) 99.9 (14.1–100) 98.5 (95.8–99.4)
Univariate model
ERCP 1.56 (0.30) 5.32 (2.19) 0.22 (0.26) 2.83 (5.87) 0 82.6 (72.6–89.5) 99.5 (73.8–100)
IOC 6.12 (3.28) 4.19 (0.57) 9.74 (12.7) 0.34 (0.67) 0 99.8 (42.3–100) 98.5 (95.6–99.5)
Estimates from bivariate and univariate models comparing ERCP and IOC
Test Variance of random
effects for logit
sensitivity (SE)
Variance of random
effects for logit
specificity (SE)
Correlation of the logits
(SE)
Sensitivity (95% CI)
Specificity (95% CI)
Bivariate model
ERCP 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 (72.3–89.5) 99.5 (71.8–100)
IOC 16.7 (26.9) 0.25 (0.54) -0.73 (0.98) 99.9 (14.1–100) 98.5 (95.8–99.4)
Univariate model
ERCP 0.22 (0.26) 2.83 (5.87) 0 82.6 (72.6–89.5) 99.5 (73.8–100)
IOC 9.74 (12.7) 0.34 (0.67) 0 99.8 (42.3–100) 98.5 (95.6–99.5)
Univariate or bivariate comparative meta-analyses: does it matter?
Aim • To investigate validity of assumption of equal variances
when comparing test accuracy in bivariate meta-regression models. – Are there important differences between findings from
bivariate meta-regression models that assume common variances across tests and those which allow variances to differ by test?
• To examine the impact of using univariate random effects logistic regression models. – Are findings from univariate meta-regression models similar to
those from bivariate meta-regression models?
Data source • Reviews identified in DARE from 1994-2012
Eligibility criteria
Included if
1. Diagnostic accuracy of 2 tests compared
2. Meta-analyses were performed
3. Possible to derive 2x2 tables for included studies
• Reviews identified in DARE from 1994-2012
Data analysis A. Preliminary meta-analysis of each test in a test
comparison performed – Bivariate model fitted to assess model stability and
estimation of correlation parameter
B. Comparative meta-analyses of each test comparison
– Bivariate model with and without equal variances – Univariate model with and without equal variances
Criteria for assessment of performance 1. Difference in magnitude of relative test performance
expressed as ratio of relative sensitivities and ratio of relative specificities
2. Difference in precision of measures of relative test performance expressed as a ratio of standard errors
3. Change in statistical significance at the 5% level: do confidence intervals include 1?
4. Change in direction of effect (qualitative change): is the ranking of a pair of tests in terms of superior sensitivity or specificity consistent between the two models?
Findings – cohort of reviews
• 57 reviews and test comparisons included
• Total number of studies in a test comparison ranged between 6 and 103
• At least one study had a zero cell in 53 test comparisons
Preliminary bivariate meta-analyses: Is correlation reliably estimated?
0
-0.5
-1.0
0.5
1.0
Inde
x te
st -
corr
elat
ion
para
met
er
0-0.5 0.5-1.0 1.0Comparator test - correlation parameter
Rarely similar for a pair of tests
Estimated within boundary of parameter space (> –1 < +1) for remaining 80 (70%)
Correlation = +1 for 12/114 (11%) Correlation = –1 for 22/114 (19%)
Comparative bivariate meta-analyses: Is it important for variances to differ by test?
Differences in magnitude
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
Rat
io o
f poi
nt e
stim
ates
Relative sensitivitiesRelative specificities
11 (22%) test comparisons had more than a 10% difference in relative sensitivity and/or relative specificity.
Across 49 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (0.99 to 1.01) and 1.00 (0.98 to 1.01).
Comparative bivariate meta-analyses: Is it important for variances to differ by test?
Differences in magnitude Differences in precision
0.2
0.6
1.0
1.4
1.8
2.2
2.6
3.0
3.4
Rat
io o
f sta
ndar
d er
rors
Relative sensitivitiesRelative specificities
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
Rat
io o
f poi
nt e
stim
ates
Relative sensitivitiesRelative specificities
Comparative bivariate meta-analyses: Is it important for variances to differ by test?
Differences in precision
0.2
0.6
1.0
1.4
1.8
2.2
2.6
3.0
3.4
Rat
io o
f sta
ndar
d er
rors
Relative sensitivitiesRelative specificities
Standard errors were on average higher for models with unequal variances compared to models with equal variances.
Median (IQR) = 1.37 (1.09 to 1.77) for ratios of standard errors of log relative sensitivities and 1.39 (1.15 to 2.05) for those of log relative specificities.
**
***
**************
**
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative sensitivity with 95% confidence interval
Equal variances Unequal variances
A
**
***
**************
**
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative specificity with 95% confidence interval
Equal variances Unequal variances
B
Comparative bivariate meta-analyses: Is it important for variances to differ by test?
For 21 (43%) of the 49 test comparisons, likelihood ratio tests indicated statistically significant differences in model fit.
**
***
**************
**
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative sensitivity with 95% confidence interval
Equal variances Unequal variances
A
**
***
**************
**
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative specificity with 95% confidence interval
Equal variances Unequal variances
B
15 (31%) test comparisons had a change in the statistical significance of relative sensitivity or relative specificity while 4 (8%) had a change in both measures.
Comparative bivariate meta-analyses: Is it important for variances to differ by test?
Qualitative differences were observed for 11 (22%) test comparisons.
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Differences in magnitude
0.900
0.925
0.950
0.975
1.000
1.025
1.050
1.075
1.100
Rat
io o
f poi
nt e
stim
ates
Relative sensitivitiesRelative specificities
Differences between both models were negligible.
Across 48 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (1.00 to 1.01) and 1.00 (1.00 to 1.00).
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Differences in magnitude Differences in precision
0.900
0.925
0.950
0.975
1.000
1.025
1.050
1.075
1.100
Rat
io o
f poi
nt e
stim
ates
Relative sensitivitiesRelative specificities
0.8
0.9
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
Rat
io o
f sta
ndar
d er
rors
Relative sensitivitiesRelative specificities
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Differences in precision
0.8
0.9
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
Rat
io o
f sta
ndar
d er
rors
Relative sensitivitiesRelative specificities
Standard errors tended to be higher for estimates from bivariate models relative to those from univariate models.
Median (IQR) ratios of standard errors for log relative sensitivities and log relative specificities were 1.00 (1.00 to 1.05) and 1.00 (1.00 to 1.01).
Univariate vs bivariate comparative meta-analyses: Are findings similar?
**
*
***
*
****
*
****
****
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative sensitivity with 95% confidence interval
Univariate model Bivariate model
A
123456789
101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
Test
com
paris
on ID
0.125 0.25 0.5 1 2 4 8
Relative specificity with 95% confidence interval
Univariate model Bivariate model
B
8 test comparisons where likelihood ratio tests indicated statistically significant difference in model fit between univariate and bivariate models.
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Change in statistical significance
Univariate vs bivariate comparative meta-analyses: Are findings similar?
Qualitative change
Summary and conclusions • Assumption of equal variances in comparative meta-
analyses is not always justified
• Validity of assumptions should be investigated if data permits
• Minimal impact of using a bivariate structure for comparative meta-analyses
• Univariate meta-regression is an alternative when bivariate meta-regression is not feasible – Provides a solution to non-convergence when data are sparse