Disproportionality statistics for signal detection

Andrew Bate Senior Director, Analytics Team Lead, Epidemiology

ISOP-UMC Training Course Mysore, 13 January 2015

Disclosures

•  I am a full time employee of Pfizer and hold stocks and stock options

Spontaneous reporting as a tool for post marketing surveillance

•  Analysis of spontaneous reports of suspected adverse drug reactions (ADRs) is a valuable tool in the detection of previously unknown drug adverse reactions

•  Reports of adverse events (AEs) associated with a drug are not necessarily true ADRs, that is, they may be temporally associated with a drug but not caused by the drug.

•  Hypothesis generation of new possible side effects from such data is referred to as signal detection

Ref Bate and Evans 2009 PDS

Signal of suspected causality - definition

“Information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, that is judged to be of sufficient likelihood to justify verificatory action.”

Ref CIOMS VIII Practical Aspects of Signal Detection in Pharmacovigilance 2009

Spontaneous reporting limitations

•  Often limited clinical information on reports and satisfactory secondary case evaluation is not always possible

•  Not all ADRs that occur will be recognized as drug induced by a healthcare professional

•  Even those that are suspected will not necessarily be reported •  Suspicion can mistakenly rest on the drug, coincidental spontaneous

ADR case reports resulting •  Control information is not collected as part of spontaneously reported

systems, the drug use is not known, and there is no direct information on disease incidence

Ref Bate et al 2008 FCP

Quantitative signal detection

•  To detect potential signals for further investigation that are not readily recognisable on a single case report nor otherwise readily apparent at case entry

•  Enhance rather than replace other methods of signal detection –  E.g. Initial case review of spontaneous reports

•  Overall help to detect signals earlier, more effectively and more efficiently

•  Primarily useful in large data sets

Quantitative signal detection –different approaches

•  Several methods for screening spontaneously reported data

•  Most common approach: “measures of disproportionality”

•  Different measures of disproportionality •  But all aim to detect the ‘unexpectedly’ frequently

reported relative to a background of other reports •  Quantitative outputs clinically reviewed and some

subsequently considered as ‘signals’

8

Metrics used in quantitative screening of spontaneous reports

AE of interest (y) Other AEs Drug of interest (x) a b

Other drugs c d Proportional Reporting Ratio (PRR) =

[a ÷ (a + b)] / [c ÷(c + d)]

Reporting Odds Ratio (ROR) = [a ÷ b] / [c ÷(d)]

Observed / Expected 1 = [a ÷ (a + b)] / [(a + c) ÷ (a + b + c + d)] 1 Frequentist basis of EBGM and IC

2 by 2 contingency table

See Bate and Evans 2009 PDS, for an overview

Bayesian methods in quantitative signal detection

-  Bayesian methods give framework for how to combine prior information with data -  Prior information/knowledge + data -  Account in a conceptually transparent way for external data and/or

subjective opinion -  Bayesian methods - not always more complex

-  High Observed and Expected, shrinkage no practical impact -  When Expected and Observed are low, IC tends to zero -  Note: Idea is that overall the estimates will be better – not that

each estimate will necessarily be better •  Informative priors also possible

ICObservedExpected

≈+

+log

//2

1 21 2

Refs Noren et al 2013, for IC formula Bate and Evans 2009 PDS, for an overview

-2

-1

0

1

2

3

4

5

6

79:1 81:1 83:1 85:1 87:1 89:1 91:1 93:1 95:1

Time(year)

Captopril - Coughing

IC

Change in Information Component over time

-6

-4

-2

0

2

4

6

67:1 70:1 73:1 76:1 79:1 82:1 85:1 88:1 91:1 94:1

Time(year)

Digoxin - Rash

IC

Change in Information Component over time

-6

-4

-2

0

2

4

6

67:1 70:1 73:1 76:1 79:1 82:1 85:1 88:1 91:1 94:1

Time(year)

Digoxin - Acne

IC

Change in Information Component over time

-6

-4

-2

0

2

4

6

8

1971197419771980198319861989 19921995 19982001

year

Practolol, ATC C07AB - Peritonitis sclerosing

PractololATC C07AB

Change in Information Component over time

14

The best metric

•  No single best method •  PRR arguably most easy to interpret •  Shrinkage in Bayesian approaches advantageous at low

counts, but more complex –  Van Puijenbroek et al 2001

•  Underreporting less effect for ROR than PRR –  Van Der Heijden 2002, Rothman et al 2004

•  ROR gives more volatile scores

Differences in metrics

6 24

54 916

ROR =4.24, PRR= 3.59, O/E= 3.33

Choice of metrics

3 27

27 943

ROR =3.88, PRR= 3.59, O/E= 3.33

Choice of metrics

Ongoing Misunderstanding

•  ”The ’PRR’ of 6 indicates that for this drug the risk of reporting this event is six times higher compared with reference drugs.”

•  Should be: ”The ’PRR’ of 6 indicates that for this drug the probability of reporting this particular event rather than any other event is six times higher compared to the probability for reference drugs

17

Background reading

•  Bate A et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998; 54: 315-321

•  Bate A, Lindquist M, and Edwards IR. The application of knowledge discovery in databases to post-marketing drug safety: example of the WHO database. Fundam Clin Pharmacol, 2008. 22(2): p. 127-40.

•  Bate A, Evans SJW. 2009 Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiology and Drug Safety. 18(6): 427-436

•  DuMouchel, W. (1999). Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. The American Statistician, 53(3), 177-190.

•  Evans, S. J. W., Waller, P. C., & Davis, S. (2001). Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and drug safety, 10(6), 483-486.

•  Gipson, G. (2012). A shrinkage‐based comparative assessment of observed‐to‐expected disproportionality measures. Pharmacoepidemiology and drug safety, 21(6), 589-596.

•  Lindquist M et al. From association to alert - A revised approach to international signal analysis. Pharmacoepidemiology and Drug Safety 1999; 8: S15-S25

•  Norén GN, Hopstadius J, Bate A. 2013 Shrinkage observed to expected ratios for robust and transparent large scale pattern discovery. Statistical Methods in Medical Research. 22 (1), 57-69

•  van Puijenbroek EM et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for Adverse Drug Reactions. Pharmacoepidemiology and Drug Safety 2002; Volume 11( 1) pp 3-10.

18

Conclusions

•  Several methods for screening spontaneously reported data

•  Most common approach: “measures of disproportionality” •  Different measures of disproportionality •  But all aim to detect the ‘unexpectedly’ frequently reported

relative to a background of other reports •  No one best method •  Quantitative outputs clinically reviewed and some

subsequently considered as ‘signals’ •  Quantitative signal detection needs to be part of robust

overall signal management programme