Addressing missing participant data in systematic reviews:

Part I – Dichotomous outcomes

Elie Akl, Shanil Ebrahim, Bradley Johnston, Pablo Alonso, Matthias Briel, Gordon Guyatt

Disclosures

• No conflicts of interest to disclose

• This work was funded by the Cochrane Methods Innovation Fund

Workshop plan

• Missing participant data at the RCT level

• Missing participant data at the SR level

• Practical issues

• Discussion

Workshop plan

• Missing participant data at the RCT level

• Missing participant data at the SR level

• Practical issues

• Discussion

Missing Participant Data

• MPD refers to:– participants excluded from the analysis of the effect

estimate in the primary study because no data are available

• MDP does not refer to:– Missing studies (e.g., unpublished studies);– Missing outcome data (e.g., unreported outcomes);– Missing summary data (e.g., unreported SD);– Missing study-level characteristics (e.g., mean age, for

subgroup or meta-regression analyses)

Dealing with MPD at the RCT level

• 87% of RCTs published in high impact medical journals had participants with missing data for the primary outcome

• The median percentage of participants with missing data was 6% (inter-quartile range 2% to 14%)

Akl et al. BMJ. 2012 May 18;344:e2809

STUDY DESIGN AND OBJECTIVE

CONCLUSION

Karl et al. BJU. 2010.106:24-6Primary outcome: overall success

Dealing with MPD at the RCT level

• Complete case analysis• Make assumptions about the outcomes of

participants with missing data:– None suffered the outcome– All suffered the outcome– Best case scenario– Worst case scenario

Karl et al. BJU. 2010.106:24-6

Karl et al. BJU. 2010.106:24-6

Karl et al. BJU. 2010.106:24-6

Dealing with MPD at the RCT level

• However, these assumptions are not plausible• More plausible assumptions: based on RIMPD/FU

– event incidence among those with MPD (not followed-up) relative to the event incidence among those followed up

– RIMPD/FU = 2: event incidence among those with MPD is double the event incidence among those followed up

Akl et al. BMJ. 2012 May 18;344:e2809

Exercise 1

20000

1000

20(not followed-up)

?

980(followed-up)

200events

1000

10(not followed-up)

?

990(followed-up)

240events

Exercise 1 solution

Dealing with MPD at the RCT level

• What are the advantages and disadvantages of:– Complete case analysis– Assuming none suffered the outcome– Assuming all suffered the outcome– Assuming best case scenario– Assuming worst case scenario– RIMPD/FU approach

Workshop plan

• Missing participant data at the RCT level

• Missing participant data at the SR level

• Practical issues

• Discussion

Dealing with MPD at the SR level

• What are the issues that systematic review authors need to deal with in relation to MPD?

Dealing with MPD at the SR level

• We will discuss how systematic reviewer authors need to:

– Deal with MPD when producing the pooled effect estimate for the primary analysis

– Assess risk of bias associated with MPD and the extent to which introduces confidence in results (quality of evidence)

Systematic review level - data analysis

Systematic review level - data availability

Trial level - data analysis

Trial level - data collection

Trial level - participant flow

Trial level -participant entry

Randomized

Adherent and followed-up

Collected

Included

Available

Nonadherent

Collected

Included

Available

According to trial

analysis

Collected

Excluded

Available

ITT, per protocol or as treated

Not collected

Excluded

Not available

Lost to follow-up

Not collected

Excluded

Missing

CCA, or make assumptions

Mistakenly randomized

Appropriately excluded

Exclude

Dealing with MPD at the SR level

• The Cochrane handbook encourages systematic reviewer authors to re-analyze a study’s effect estimate by including all randomized participants

• The handbook, however, fails to provide detailed guidance on how such analyses should be conducted

Proposal to handle MPD

• For the primary analysis: exclude participants with missing data (complete case analysis)

• When the primary analysis suggests important effect, we suggest sensitivity meta-analyses making different assumptions about the outcome of participants with missing data, to test the robustness of the results (the risk of bias)

Akl et al. PLoS One. 2013;8(2):e57132

Handling dichotomous MPD

Judging RoB dichotomous MPD

• Results robust to a worst case scenario missing data does not represent a risk of bias

• Results not robust to worst case scenario test progressively more extreme assumptions culminating in a "worst plausible case”

• Important changes in results with such sensitivity analyses suggest serious RoB

Example 1

• Meta-analysis assessing effects of probiotics for prevention clostridium difficile-associated diarrhea

Johnston et al. Ann Intern Med. 2012 Dec 18;157(12):878-88

Complete case analysis

Event rate: 1.5:1

Event rate: 3:1

Event rate: 5:1

Example 1

• Based on these findings– Would you judge the risk of bias as: low or high?

– Would you rate down the confidence in effect estimates (quality of evidence)?

Example 2

• Meta-analysis comparing oral direct factor Xa inhibitors to low-molecular-weight heparin for thromboprophylaxis in patients undergoing total hip or knee replacement

• The primary analysis: – complete case analysis – factor Xa inhibitors reduced the incidence of

symptomatic deep venous OR 0.46 (0.30-0.70) Neumann et al. Ann Intern Med. 2012 May 15;156(10)

Example 2

• Two sensitivity analyses based on extreme but plausible assumptions :– RILTFU/FU 2 and 3 for the intervention arm and 1 for

control arm. – The effect estimates did not change appreciably:

OR 0.54 (0.37-0.80), 0.59 (0.40-0.87) respectively

Neumann et al. Ann Intern Med. 2012 May 15;156(10)

Example 2

• The results would lose statistically significance, OR 0.84 (0.59-1.20), when we assumed:– the lowest incidence among intervention arms of

all included trials for those with missing data in the control group

– the highest incidence among control arms of all included trials for those with missing data in the intervention group

Neumann et al. Ann Intern Med. 2012 May 15;156(10)

Example 2

• Based on these findings– Would you judge the risk of bias as: low or high?

– Would you rate down the confidence in effect estimates (quality of evidence)?

Workshop plan

• Missing participant data at the RCT level

• Missing participant data at the SR level

• Practical issues

• Discussion

Practical issues

• Identifying in the RCT report, which participants were actually followed up, and which participants having data missing

• Automatic integration of MPD in the analysis

Identifying participants with missing data

Identifying participants with missing data

Using the Excel sheet

Workshop plan

• Missing participant data at the RCT level

• Missing participant data at the SR level

• Practical issues

• Discussion

Discussion

• Is the proposed approach reasonable?

• Is the proposed approach feasible?

Thank you!