An agency of the European Union

Signal detection: le point de vue de l’EMA (EudraVigilance, CIOMS, nouvelle legislation)Ne soyez pas dupes … je vais vous donner MON point de vue

Presented by: François MAIGNENPrincipal scientific administrator (PhvRM)

Presentation title (to edit, click View > Header and Footer)2

Introduction & Disclaimers- Background (main objective of seminar)- Conflicts of interests & disclaimer- Apologies for the lack of French- Learning objectives:

- Fundamentals Disproportionality analysis- Evaluation / Comparison of the methods (limitations, stats vs clinical)- Fundamental issues included in CIOMS VIII / EudraVigilance guideline on

the use of signal detection methods in EudraVigilance DAS: DMEs/TMEs/Medical confirmation/Prioritisation/Impact analysis

- PITFALLS +++

GENERAL PRINCIPLES

3

Before we start … Let’s bet on horse racing …

Presentation title (to edit, click View > Header and Footer)4

Signal detection = horse racing• You might want to bet on the horse which will win the race.• You might want to find the top three / five horses which will win

the race.• You might want to read a specialised newspaper to find out

about each of the horse which will enter the race (pedigree, jockey, owner, previous records, track, form, …).

• You will possibly use the odds to help you to decide (4:1 what is % of bets backing a win of this horse?). An outsider might win the race (more money).

• It is always easier to comment once the race is over than finding the correct combination BEFORE the race starts.

5

Signal detection

6

PRIOR KNOWLEDGE/INFO

STATISTICS LUCK

GUESS (PROSPECTIVE) IS ALWAYS MORE DIFFICULT

77

Measures of disproportionate reporting

Most of the methods routinely used in pharmacovigilance (spontaneous reporting systems) databases are based on measures of disproportionate reporting (i.e. ROR, PRR, BCPNN, MGPS, etc …).Basically: “Observed vs Expected” analysis in a given database i.e. % of reports involving a given reaction for a given medicine compared to the % of reports involving this reaction on the whole database

88

A spontaneous reporting system database

SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N

Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N

Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N

Event 3 n31 n32 … … … … … … n3N

Event 4 n41 n42 … … … … … … n4N

Event 5 n51 n52 … … … … … … n5N

Event 6 n61 n62 … … … … … … n6N

… … … … … … … … … …

Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN

Proportional Reporting Ratio

9

Drug 1 All other medicinal products

Total

Event 1 a c

All other reaction terms

b d

Total N = a + b + c + d

c + d

a + c

a + b

1010

Proportional Reporting Ratio

PRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN IN PRACTICAL TERMS?a/(a+b) = Proportion of reports involving a specific adverse event among all the reports involving DRUG Ac/(c+d) = Proportion of reports involving THE SAME adverse event among all the reports of your database but DRUG A

1111

Proportional Reporting Ratio

If the rate of reporting of AE for drug 1 is similar to the rate of reporting of this AE for all the other products of the database, the PRR will be equal to 1 (same proportion of reports involving the reaction for drug A than for the other drugs) …

BUT … If the reaction is proportionately MORE reported with drug A than for the other products, the PRR will be increased (typically > 1). DIS-PROPORTIONALITY of reporting

1212

Disproportionality analysis (example)

. CNS drug for which the total No of reports is 400, of these 20 reports of diarrhoea. All other products in the database (1 million reports excluding reports involving drug A), of these 50,000 reports of diarrhoea.

PRR = [20/400] / [50,000/1,000,000] = 1 (no SDR)

1313

Disproportionality analysis (example)

CNS drug for which the total No of reports is 400, of these 40 reports of drowsiness. All other products in the database (1 million reports excluding reports involving drug A), of these 25,000 reports of diarrhoea.

PRR = [40/400] / [25,000/1,000,000] = 4 (presence of a SDR)

Strong underlying assumptions

- Association between a true risk and reporting of this risk (not always true i.e. notoriety bias)

- Similar under-reporting for products across the database (not true)

- Role of the confounding (indication, underlying disease)

14

15

Improvements of these methods•Considering possible confounding factors: stratification and log-linear models (ROR – see work from E. Van Puijenbroek)•Trying to circumvent low expected values or low case counts: Bayesian models (A. Bate & W. DuMouchel)•Other regression methods: LASSO and Bayesian logistic regressions (N. Noren, D. Madigan)•Public Health relevance not always clear or demonstrated•Some methods can be computationally demanding

16

Bayesian methods

BCPNN and MGPS rely on the same principle of conjugate prior distributions:•These methods will shrink the value of the measure of disproportionality using a Bayesian approach (prior based on existing dataset)•BCPNN: cell counts ~ Binomial dist., conjugate prior = beta•MGPS: cell counts ~ Poisson, conjugate prior = Gamma (mixture of Gammas).FUNDAMENTALLY SAME PRINCIPLE AS DA +++

1717

Bayesian methods

Assume binomial y=7 successes, 20 trials.Non informative prior = Beta (2,4)

1818

Thresholds - ARBITRARY

All these methods provide a ranking …Thresholds = arbitraryTrade-off between•Reviewing too many drug-event pairs(loss of operational benefit)•Missing some signals

No ADR ADR

Limitations of the quantitative methods

19

The concept of threshold implies that not all the reports will be reviewed and the quantitative methods will not detect all the signals (for which the data have been reported to the database on which the DMA is used)

See Importance of reporting negative findings in data mining – the example of exenatide and pancreatitis Pharm Med 2008; 22(4): 215-219).

2020

Comparison of the methods

Methodological difficultiesNo gold standard / no standardised reference method (in many instances “traditional methods of PhV”)Imprecision of what constitutes a signalRetrospective vs prospective evaluationImportance of clinical judgement. The added value of clinical evaluation is currently unknown (if any).

2121

Comparison of methods

1

2

3

4

7

11

6

9

10

12

5

8

1

2

3

4

7

11

6

9

10

12

5

8

4

2

3

1

7

11

6

9

10

12

5

8

Meth.1

Meth.2

Meth.3

• Threshold 1: Meth. 2 = 5 true signals, meth. 1/3 = 4.• Threshold 1+2: Meth. 2=Meth.3• First 5 signals: Meth. 1 ≠ Meth. 2 = 3.

ADR No ADR

2222

Performances of these methods

Operational benefit (screening of large databases)Anecdotal evidence (in opposition to structured) of signals discovered thanks to the quantitative methods (recent examples incl. D:A:D and MI)Time benefit in some cases (Hochberg & EV study)NND ~ 7/15 (depending whether the study is retrospective or prospective)Idea: Quant. Methods + DMEs/TMEs

2323

New approaches to signal detection

Deviation of Obs. vs Expect. distr. from a fitted distribution (Jim)Modelling of the hazard function of the time to onset (DSRU / François) hazard # mechanismUse of longitudinal databases (record linkage and electronic health records – OMOP / Noren / Callreus) ~ incidence rate ratio• Same patients different time windows (A. Bate)• Hospital records of different patients (T. Callreus)

MODELLING OF TIME TO ONSET

24

25

Hazard fcts of parametric survival dist.

Kalbfleisch and Prentice. The statistical analysis of failure time data. Second ed. Wiley and sons.

Reported hazard of occurrence: a phenomenon involving several mechanisms

26

P(occur.)*P(diag./occur.)*P(rep./diag.)(1)P = prob. failure conditional on survival until time t.Lim f(x)*g(x) = Lim f(x)*Lim g(x)Then when we take Lim t -> 0 (1) becomes.

h(occur.)*h(diag./occur.)*h(rep./diag.)PD

Toxicology profileEfficacy / duration tt

Monitoring and “RM” activities

Awareness

AwarenessReporting mechanisms

27Presentation title (to edit, click View > Header and Footer)

Liver injuries reported with bosentan (KM)

28

Liver injuries reported with bosentan(hazard functions)

Bosentan – liver injuries

29

Logical course of events some occurrences need careful interpretation (blood bilirubin inc. and [hyper]bilirubinemia)Pattern AST/ALT unusual for liver injuries (but not for mitochondrial injuries from hepatocytes) but consistent with clinical safety dataResidual and constant risk of liver failureConsistent with the putative mechanism of toxicity (dose-dpt)Consistent with the safety profile of bosentan (lack of independence)Influence of the risk minimisation activities

Influence of reporting mech.

30

FUNDAMENTAL CONCEPTS OF SDR/SIGNAL

31

The fundamental difference between a SDR and a signal +++

32

•PRR is a measure of disproportionality of reporting in a specific database (observed vs expected value computed on the whole database)•The disproportionality analysis is not an inferential exercise (i.e. the method is not aimed at drawing conclusions about a parent population on the basis of evidence obtained from a random sample from this population).•These “REPORTED statistical associations” detected by the quantitative methods do not imply any kind of causal relationship between the administration of the drug and the occurrence of the adverse event.

Different concepts / different definitions

33

SDR (signal of disproportionate reporting): refer to drug-event pairs highlighted by DMAs. (see EMEA guideline) NOTE: The term SIGNAL in SDR will not be retained by the CIOMS VIII.

Signal: A signal is information on an adverse event that is new or incompletely documented that may have causal relationship to treatment and is recognized as being worthy of further explorations (see CIOMS VIII). The SDRs must be systematically medically confirmed.

(Identified) Risk: An untoward occurrence for which there is adequate evidence of an association with the medicinal product of interest (see Guideline on risk management systems for medicinal products for human use EMEA/CHMP/96268/2005).

34Presentation title (to edit, click View > Header and Footer)

DMA Database (drug-events pairs)

SDRs

SIGNALSSIGNALS(other data sources)

Medical judgement

RISKS

Further evaluation / characterisation

Regulatory action

NO

35

Process flow included in the EMEA guideline on the use of statistical methods implemented in the EV data analysis system (EMEA/106464/06) July 2008.

SIGNAL DETECTION PROCESS

36

37

Signal detection: a

complex multifactorial

process

38

1. Data collectionData capture

(management)Data transmission

Data capture and data management (1)

39

Fundamental but not in the scope of CIOMS VIIIIT infrastructure and softwareThe volume of information hence the data management activities (data coding, entry, recoding, data quality) is extremely resource demanding.Data management will have a critical influence on the signal detection activities incl.• Medicinal product information: creation and

maintenance of dictionaries, lack of international standard, absence of INN or standards in some instances e.g. vaccines

• Medical terminology: criteria for the use of terms, conversion of legacy data encoded with a different terminology, …

• Data quality: FUp, duplicates

40

2. Methods, signal detection & data

analysis

41

42

43

3. Signal management (prioritisation,

evaluation, decision & communication)

44

4. Link with risk management

Signal management

45

• Similarly the CIOMS has identified a signal management step which includes:

• Triage• Prioritisation and impact analysis• Evaluation• Decision• Communication (broad sense)• Follow-up• Link with risk management

FUNDAMENTAL QUESTION OF IMPACT ANALYSIS

NO VALIDATED METHOD. Assess the Public Health impact of the signal:Usually:-Seriousness-Frequency of occurrence (absence of evidence is NOT evidence of absence)-Particular population at risk-“worst case scenario” (what would happen if … ?)-Preventability, reversibility, etc …46

PITFALLS: PRIORITISATION / IMPACT ANALYSIS

47

48

Signal prioritisation and serious medical events: reported rate of fatality as a prioritisation variableAbout the EV-EWG IME list and lists of IMEs in general (e.g. CIOMS V)Useful but purpose not always clear (early signal detection? Focus the detection? Signal prioritisation?)Based on expert’s judgmentHas not been formally “validated” / tested (no standards)Probably situation dependant

49

Concept of seriousness # linked to the outcome # surrogate for grading the severity of the reactions hence prioritisationGrading in seriousness: death >> disability (permanent) >>> life-threatening >>> disability (temp.) >> prolongation hosp.Variable linked to fatal outcome = reported rate of fatalityFor each drug-event pair = No of reported fatal cases / total number of reported casesComputed for the intensively monitored products

Reaction 1 Reaction 2 Reaction 3 Outcome (incl. fatal)

Surrogate to predict the outcome

50

Hazardous identification of serious events a prioriSome examples of reactions not usually considered to be serious per se which can be linked to most dramatic outcomes (e.g. dramatic increases of liver aminotransferases e.g. >100 ULN leading to liver failure, liver transplant and death)

Exhaustion/tiredness

Jaundiceincr. aminotransferases 500ULN

hyperbilirubinemiaLiver transplant Death

Prioritise these events on the associated reported outcome (here death)

51

Reported rate of fatality

Some reactions may be consistently linked to a high reported mortality rateSome reactions are serious but do not lead to a fatal outcomeSome reactions are situation dependent (the reported rate of fatality may be highly variable)For each of the MedDRA PT involved in a DEC in EudraVigilance, the following variables were computed across all the products involved in the reported combinations:• Mean, min., max., range: max. – min., SD

52

How does it relate to IME status?

Reported rate of fatality for IMEs > non-IMEsNumber of events for which the reported rate is high which are non-IMEsVery high number of IMEs for which the reported rate of fatality is zero.IMEs useful for prioritisation?

The figure displays the boxplot of the average reported rate of fatality for non-IMEs (left)compared to IMEs (right) (red and blue line = mean rate for non-IMEs (red) vs IMEs (blue))

53

54

Liver injuries

Clear relation between Reported rate # seriousness of injury and the severity of the outcomeHighest mean rate around 30% (1/3 fatal reports) with a max. at 75% (3/4)Some inconsistencies (bilirubin disorders: hyperbilirubinaemia 18.7%, blood bilirubin increased 16.2%, blood bilirubin unconjugated increased 6.7% and bilirubin conjugated increased 6.3%)Unclear or undefined concepts (liver disorders [?]) linked to a fairly high mean reported rate 18.9%, hepatic function abnormal 8.5% and liver function test (singular) abnormal 9.1%.

56

Data reduction (PCA)

57

DiscussionThree set of events used for signal detection: mild reported rate of mortality, moderate and highReported rate of fatality can be useful (and should be used) for signal prioritisationNeeds to be considered with caution (events with rate of zero include e.g. Torsade de pointes, autism, Breast cancer in situ, Breast cancer stage I, Dermatitis exfoliative)Does not replace DMEsDeath is not the only criterion which could be usedEudraVigilance = only serious reactions(!)Some events are consistently associated either with low rate or conversely with very high rate

PITFALLS: MASKING EFFECT OF DA

58

59

Masking effect of measures of disproportionality (here = PRR)The masking effect has first been described and identified by Gould in spontaneous reporting system databases (pharmacoepidemiology and drug safety in 2003 – 8 years ago).

The masking is a statistical artefact by which true signals are hidden by the presence of information reported with other medicines in the database. Therefore, the masking involves one given reaction and two products (the product for which the DA is conducted) and a possible masking drug.

The masking effect is a potentially important issue for Public Health which is not perfectly understood or perfectly quantified: some signals might be missed or identified with delay because of the presence (or a suspicion on the presence) of masking effect.

60

Masking effect of measures of disproportionalityIn particular, there is no algorithm to identify the potential masking drugs to remove them from subsequent analyses aimed at identifying new signals using the statistical methods of signal detection based on disproportionality analysis.We have developed an algorithm based on the computation of a simple The masking ratio has been developed to be intuitive. The highest masking drugs have the highest masking ratio.From an underlying mathematical framework, we have developed a simple expression of the masking ratio (which can be easily computed on a database incl. No of computations and IT resources) which allow a fairly rapid identification of the main culprits.

61

62

Masking effect of measures of disproportionalityRecent studies have shown effects which were suspected from the article by Gould, that masking products are usually products for which the given reaction is known (i.e. listed in the SPC), therefore likely to have a high PRR (in the database in which the analysis is conducted) for the adverse drug event / reaction which is included in the disproportionality analysis.Unfortunately, the authors could not conclude on any algorithm considering that this association is not systematically present (not all products with high PRR will induce a significant masking even if he masking generally involves products with a high PRR).

63

Masking effect of measures of disproportionality (RRR)Respective proportion of reports in

the database influences the extent

of the masking

The higher the proportion of reports

involving a product for a given

reaction the higher the masking

The lower the proportion involving a

given product over the total number

of reports in the database, the

higher the masking

64

65

Relation between the masking effect and the PRR (of the masking medicinal product for the given event)

MR > 1PRR > 2

66

The highest masking is induced by products known to induce the given reaction (and for which the PRR is likely to be increased)

Products of the same class induce the highest masking for similar reactions (gambling – ropinirole and pathological gambling – cabergoline, Fanconi syndromes, role of drug-drug interactions – rifampicin)

CONCLUSIONS

67

FUNDAMENTAL ISSUES: take home messages

- Image of horse racing-Most of the methods rely on disproportionality analysis: strong underlying assumptions-SDRs: statistical association : needs to be systematically medically confirmed-Process flow: PRIORITISATION & IMPACT ANALYSIS-PITFALLS: METHOD (e.g. masking), prioritisation (e.g. IMEs)-Importance of strategy incl. DMEs / TMEs-PRIOR MEDICAL KNOWLEDGE (Prepared mind)68

69

Mercifrancois.maignen@ema.europa.eu

http://uk.linkedin.com/in/francoismaignen@EMA_News

@FrancoisMaignen