Quality and Reporting Characteristics of Network Meta-analyses€¦ · Quality and Reporting Characteristics of Network Meta-analyses: A Scoping Review Andrea C. Tricco MSc, PhD Scientist,

Quality and Reporting Characteristics

of Network Meta-analyses: A Scoping Review

Andrea C. Tricco MSc, PhD

Scientist, Li Ka Shing Knowledge Institute of St. Michael’s Hospital

Assistant Professor, Dalla Lana School of Public Health, University of Toronto

CADTH Symposium 2016

Conflict of interest statement:

I have no actual or potential conflict of interest in relation to this presentation.

Funding:

This research was funded by the Canadian Agency for Drugs and Technologies in Health.

Definition of Network Meta-analysis (NMA)

NMA is an extension of indirect comparisons that allows the combination of direct with indirect comparisons, and also the simultaneous analysis of the comparative effects of many interventions

3

Cochrane Handbook, http://handbook.cochrane.org/

Network meta-analysis (NMA)

4

Steps for a systematic review in a NMA

5

Start with a systematic review and NMA protocol

Conduct a comprehensive literature search

Use pre-defined eligibility criteria, same types of data abstracted

Use the same risk of bias appraisal as recommended for reviews

Synthesize totality of evidence and report using PRISMA-NMA

Discuss strengths and limitations

Conduct each step by 2 reviewers, independently

Hutton et al., Ann Int Med, 2015

Methodological considerations

6

Broader PICOS than systematic reviews with a meta-analysis

Literature search must include all interventions/comparators

= Expect to include many more studies!

Decisions on how to “lump” and “split” treatments into nodes (e.g., dosing, administration, drug class versus specific drug analysis)

= A lot of clinical guidance!

Statistical considerations

7

Network geometry

Key assumptions:

1. Homogeneity – similar considerations as meta-

analysis

2. Similarity (transitivity) – effect modifiers

3. Consistency – agreement between direct and indirect

evidence

Assumptions underlying indirect

comparisons and NMA

Cipriani et al Ann of Int Medicine 2013

Assumption for indirect and mixed comparison

Conceptual definition (Transitivity)

Clinical Methodological

Property of parameters and data

(Consistency)

Statistical

Transitivity

Interpretation of transitivity:

1. Treatment A is similar when it appears in AB and AC

trials

2. The two sets of trials AB and AC do not differ with

respect to the distribution of effect modifiers

3. Participants included in the network could in principle

be randomized to any of the 3 treatments A, B, C

4. ‘Missing’ treatment in each trial is missing at random

5. There are no differences between observed and

unobserved relative effects of AB and AC beyond

what can be explained by heterogeneity

Salanti Res Synth Methods 2012

A

B

C

Consistency

B

C

A

B

C

Direct and indirect evidence are in

agreement

10

Consistency is a property of a

closed loop (path that starts and

ends at the same node)

Heterogeneity

The heterogeneity variance can be estimated using several approaches

─ E.g., DerSimonian and Laird (DL), Maximum Likelihood (ML), Restricted

Maximum Likelihood (REML)

In a Bayesian framework several priors can be assigned

─ E.g., Informative, Minimally informative, Vague

Several heterogeneity assumptions can be considered

1. Common within-network heterogeneity: all treatment comparisons in the

network are associated with the same magnitude of heterogeneity

2. Comparison-specific heterogeneity: each treatment comparison in the

network has its own amount of heterogeneity

3. Common within-loop heterogeneity: treatment comparisons included in each

closed loop in the network share the same amount of heterogeneity

11

Objective

To explore the quality and reporting characteristics

of all published NMAs through a scoping review

12

Methods

A priori protocol compiled using PRISMA-P

Methods guided by Joanna Briggs Institute Methods Manual

MEDLINE, EMBASE, PubMed and Cochrane searched from

inception to April 14, 2015 without any language or

publication status restrictions

Included NMAs with 4 treatments, randomized trials, and

using a valid statistical analysis

Pairs of reviewers independently performed screening of

citations and full-text articles; abstracted data; and assessed

quality of included studies (AMSTAR and ISPOR)

13

Results – study flow

Additional records identified through other sources

(n = 253)

Scre

en

ing

Incl

ud

ed

Elig

ibili

ty

Ide

nti

fica

tio

n

Citations identified through database

searching

(n = 3727)

Citations after duplicates removed (n = 3538)

Citations screened (n = 3538)

Records excluded (n = 2913)

•Not an NMA (n=2601) •NMA includes observational studies (n=61) •Includes <4 treatments (n=216) •Commentary/Protocol (n=35)

Full-text articles assessed for eligibility

(n = 625)

Full-text excluded (n = 382)

•Unable to locate (n = 3) •Duplicate (n=16) •(Commentary/Protocol (n = 91) •Invalid NMA (n = 98) •NMA includes observational studies (n = 21) •Includes <4 treatments (n = 70) •No. of treatments > no. of studies (n = 76) •Companion report (n = 7)

Studies included in qualitative synthesis (n = 496)

Full-text articles assessed for eligibility

(n = 878)

Results – geographic trends

Geographic distribution of publications by region (n=496)

31%

0.6%

1.2%

1.4%

North America

Africa

Central &

South America

Australia

13% Europe

52%

Asia

Multiple regions: 0.4%

0.2% 0.2% 0.4% 0.2% 0.2% 0.4% 1.0%

2.8%

6.3% 6.7%

11.7% 12.5%

18.3%

23.8%

10.5%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

1997 1999 2000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015*

Results – temporal distribution

Temporal distribution of publications (n=496)

2.4% 2.4%

*2015 sample is not complete

Results – frequency of terms

Word Cloud of NMA terminology based on frequency (n=496)

NMA terminology (n=496) Count (%)

Network meta-analysis 232 (38.9%)

Mixed treatment comparisons 122 (20.5%)

Indirect comparisons 57 (9.6%)

Results – cited methodology

Word cloud of cited methodology paper (n=496)

Most Commonly Cited Papers Count (%)

Lu & Ades 2004 149 (30%)

Caldwell 2005 82 (17%)

Bucher 1997 71 (14%)

Results – journal disciplines

Journal Discipline (n= 496) Count (%)

Medicine, General & Internal 121 (23.8%)

Health Care Sciences & Services 56 (11.0%)

Pharmacology & Pharmacy 35 (6.9%)

Cardiac & Cardiovascular Systems 29 (5.7%)

Endocrinology & Metabolism 25 (4.9%)

Oncology 21 (4.1%)

Multidisciplinary Sciences 19 (3.7%)

Rheumatology 17 (3.3%)

Clinical Neurology 15 (2.9%)

Gastroenterology & Hepatology 15 (2.9%)

Results – NMA characteristics

NMA Characteristics (n=496) Count (%)

Funding

Publicly-sponsored 186 (38%)

Industry-sponsored 103 (21%)

Non-sponsored 106 (21%)

Industry and publicly sponsored 9 (2%)

Funding source not reported 92 (19%)

Article Type

Application paper 438 (88%)

Application paper (no review method) 18 (4%)

Development paper 40 (8%)

Review Size (n=438)* median (Q1, Q3) 25 (14, 48)

*Review size reported for application papers with full review methods only

Results – review design

Method Characteristics (n=438) Count (%)

Re

vie

w T

yp

e

Knowledge Synthesis

Name

Systematic review 356 (81.3%)

Overview of reviews 7 (1.6%)

Not reported 75 (17.1%)

Review Duration

<6 months 55 (12.6%)

6-12 months 132 (30.1%)

>12-24 months 106 (24.2%)

>24 months 81 (18.5%)


A p

rio

ri p

roto

co

l a

nd

revie

w d

es

ign

A Priori Protocol

Use of a protocol mentioned 66 (15.1%)

Published 40 (9.1%)

Registered 25 (5.7%)

Available upon request 6 (1.4%)


Research Question Clearly Reported 437 (99.8%)

Unclear/inferred 1 (0.2%)

Eligibility Criteria

Clearly reported 430 (98.2%)

Unclear/inferred 1 (0.2%)


Results – literature search


Iden

tify

ing

rele

van

t stu

die

s

Search String

Complete literature search 207 (47.3%)

Medical subject headings only 173 (39.5%)


Databases Searched

Searched >1 database 407 (92.9%)

Searched only 1 database 29 (6.6%)


Additional Search

Strategy

Scanned references 309 (70.5%)

Grey literature searched 270 (61.6%)

Consulted topic experts 80 (18.3%)

Consulted librarian 67 (15.3%)

Performed updated search 62 (14.2%)

Manually searched select journals 37 (8.4%)

Limits Applied

Limited by date 135 (30.8%)

Limited by language 147 (33.6%)

Limited by study design 291 (66.4%)

Other limits (e.g., age, humans) 129 (29.5%)

Results – screening, abstraction, quality


Stu

dy S

ele

cti

on

Title & Abstract Screening

Reported 402 (91.8%)


Full-text Screening



Study flow

Completely in PRISMA-like flow diagram 374 (85.4%)

Completely in text/table only 20 (4.6%)

Partially reported 15 (3.4%)


Da

ta

Ab

str

ac

tio

n &

Qu

ali

ty

As

se

ss

me

nt Data Abstraction



Quality Appraisal



65%

2%

4%

0.5%

21%

8%

64%

3%

2%

0.2%

24%

7%

54%

21%

2%

0.0%

17%

5%

41%

5%

2%

0.5%

30%

21%

0% 10% 20% 30% 40% 50% 60% 70%

≥ 2 independent reviewers

1 reviewer & 1 verifier

1 reviewer only

Not done

Done but unclear # of reviewers

Not reported



1 reviewer only

Not done


Not reported



1 reviewer only

Not done


Not reported



1 reviewer only

Not done


Not reported

Tit

le &

Ab

str

act

Scre

en

ing

Fu

ll-t

ext

Scre

en

ing

D

ata

Ab

str

acti

on

Qu

ality

Assessm

en

t Results – conduct of review process

Method characteristics – conduct of review process (n=438)

Short-cuts taken (6%)




Inadequate reporting (29%)




Results – streamlined methods

NMAs that leveraged previous systematic review(s) Count (%)

Leveraged previous systematic reviews (n=456)*

Yes 78 (17%)

No 378 (83%)

Leverage Process (n= 78)

Updated literature search of previous systematic review(s) 37 (47%)

Used literature database of previous systematic review(s) 20 (26%)

Updated and expanded literature search of previous systematic review(s) 9 (12%)

Cochrane update 6 (8%)

Used database of previous systematic review(s) 3 (4%)

Expanded literature search of previous systematic review(s) 2 (3%)

Updated literature search of previous systematic review(s) and used data from previous

review 1 (1%)

*Includes all application papers

Results – AMSTAR components

Review methodological quality – AMSTAR (n=438)

31%

58%

84%

62%

18%

83%

62%

48%

86%

36%

16%

69%

39%

0.5%

33%

1%

13%

21%

1%

12%

0.5%

3%

15%

5%

82%

5%

17%

29%

2%

63%

84%

21%

0% 20% 40% 60% 80% 100%

A priori design

Duplicate screening and data extraction

Comprehensive literature search

Grey Literature

List of includes / excludes

Study characteristics

Quality Appraisal

Quality of evidence incorporated in conclusions

Appropriate pooling methods

Publication bias assessed

Conflict of interest declared

YES UNCLEAR NO NOT APPLICABLE

Results – AMSTAR scores overall

Review methodological quality – AMSTAR overall quality (n=438)

1% 1%

7%

8%

13% 12%

17%

15%

11%

7%

4% 3%

0%

4%

8%

12%

16%

20%

0 1 2 3 4 5 6 7 8 9 10 11

Low Quality Moderate Quality High Quality

Sc

ore

Dis

trib

uti

on

(%

)

AMSTAR Score

median (IQR)= 6 (4,7)

Low: 18%

Moderate: 57%

High: 25%

0%

20%

40%

60%

80%

100%

1999 2000 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Year of Publication

Low (0-3) Moderate (4-7) High (8-11)

100%

50%

100%

50%

21%

13%

31% 29% 25% 22% 19%

10% 4%

50%

100%

25%

64%

50%

90%

50%

64%

62% 59%

71%

79%

74%

25%

14%

38%

10%

19%

7% 13%

19%

10% 11%

21%

Results – AMSTAR scores over time

Review methodological quality vs. publication year (n=438)

Results – ISPOR components

NMA methodological quality – ISPOR (n=456)

54%

48%

31%

81%

12%

46%

21%

41%

73%

20%

20%

1%

19%

24%

27%

31%

27%

14%

10%

0% 20% 40% 60% 80% 100%

Consistency Assessed

Direct & indirect evidence combined

Accounted for inconsistency

Valid rationale for FE or RE model

Heterogeneity assumption explored

Accounted for heterogeneity

YES UNCLEAR NO Not Applicable

Summary

Sub-optimal quality of reporting was observed

PRISMA-NMA may lead to improvements in reporting over time

Less than a quarter of the published NMAs were considered high

quality (AMSTAR ≥8), however, this improved over time

Authors of published NMAs used streamlined methods:

23% streamlined data abstraction, 17% leveraged previous reviews

About 20% did not assess consistency, provide rationale for FE/RE model or

account for heterogeneity

ISPOR may lead to improvements in review/NMA conduct

Acknowledgements

Co-investigators on funding proposal: Dr. Sharon Straus, Dr. Areti Angeliki Veroniki

Research team:

Wasifa Zarin

Vera Nincic

Afshin Vafaei

Dr. Shannon Sullivan

Dr. Emily Reynen

Sanober Motiwala

Maria Petropoulou

CIHR (new investigator award, MAGIC team grant)

31

Myrsini Gianiasti

Jesmin Antony

Caitlin Daly

Joycelyne Ewusie

Dr. Adriani Nikolakopoulou

Dr. Anna Chaimani

Dr. Georgia Salanti

Questions?

Email: [email protected]

Office Location:

209 Victoria Street, 7th floor, Room 721, East building, Toronto, ON

Telephone: 416.864.6060 x77521

32

mailto:[email protected]

Documents

Quality and Reporting Characteristics of Network Meta-analyses€¦ · Quality and Reporting Characteristics of Network Meta-analyses: A Scoping Review Andrea C. Tricco MSc, PhD Scientist,