STATISTICS IN MEDICINE, VOL. 12, 2383-2393 (1993)

ALERT SYSTEMS FOR POST-MARKETING SURVEILLANCE OF ADVERSE DRUG REACTIONS

MICHAELA PRAUS, FRITZ SCHINDEL, REINHARD FESCHAREK AND SABINE SCHWARZ Biornetrics Department Therapeutics and Drug Surveillance. Behringwerke AG, P.O. Box I 1400.

35001 Marburg, Germany

SUMMARY

When monitoring spontaneous reports of adverse reactions to registered drugs, it is important to detect any change in the number of reported adverse reactions in the course of time. Sales adjusted adverse drug reaction rates are usually compared in order to be able to take drug exposure into account. Here we review the so-called arithmetic and some statistical procedures which could form the basis for an alert system. The advantages and disadvantages of each of these methods are discussed. The importance of data requirements and the problems which arise when using an alert system are pointed out and then clarified with the help of the example of diphtheria/tetanus vaccine.

1. INTRODUCTION

Both pharmaceutical industry and regulatory authorities are aware of the importance of post- marketing surveillance of therapeutic drugs. In general, it is impossible to include enough patients in clinical trials to detect uncommon adverse reactions. Furthermore, investigational drugs are frequently not administered to risk population groups such as elderly patients or pregnant women. Table I shows examples of required sample sizes to get an impression of the magnitude of clinical trials to detect uncommon adverse drug reactions (ADRs). Based on the level of background incidence A. in an untreated control group and the additional incidence ,I1 of the ADR due to the treatment, sample size estimation methods for the comparison of two propor- tions lo and A. + A1 are applied1V2. Lewis’ points out that owing to the dangers of multiple comparisons these sample sizes even increase if many ADRs are simultaneously monitored in a single trial. It is obvious that such prospective trials are not feasible if the additional incidence of the ADR due to the treatment is small. In this case other methods of investigation are necessary. Case-control studies are adequate if there is already a suspicion that a drug causes a certain uncommon undesired effect. But only thorough monitoring of spontaneous reports of ADRs to marketed drugs can lead to such suspicions.

Spontaneous reports of ADRs are systematically collected by the pharmaceutical company and reported to regulatory authorities. In addition to single case discussions, further summarizing analyses are necessary to detect risks as early as possible. For this reason one tries to detect any changes in the ADR rates in the course of time.

* Presented at the Society for Clinical Trials/International Society for Clinical Biostatistics Joint Meeting, Brussels, Belgium, July 1991.

0277-671 51931242383-1 1$10.50 0 1993 by John Wiley & Sons, Ltd.

2384 M. PRAUS ET A L

Table I. Number of patients required per group:' a = 0.05 (two-sided), fl = 0.2, equal sized groups

Background incidence of Additional incidence of ADR on drug ADR in control group 00 1 0.005 0.00 1

0.1 15,000 58,000 1,420,000 0.05 8,200 32,000 760,000 0.0 1 2,400 7,800 170,000

0.18

0.16

0.14

0.12 ? - ._

0 1 0 2 9

0.08 g n

0 .- L

0.06 c%

0.04

0.02

0.00

Alerts based on ( 0 ) arithmetic method (*) FDA-method (.) a11 methods ~ Injection site - - - - Serious

Figure 1 . Rates of injection site and serious ADRs to the DT vaccine, West Germany, 1986-1989

Figure 1 shows the local injection site and serious ADR rates for a combined diph- theria/tetanus vaccine frequently used for children from 1986 to 1989 in West Germany. One litre of vaccine corresponds to 2000 single doses and the number of doses delivered is obtained from the factory sales of the vaccine. A high percentage of the reported ADRs are injection site reactions. Apparently, injection site reactions, but not serious ADRs, markedly increased in the second and fourth quarter of 1987. The problem is how such increases can be assessed. There are three aspects which influence the evaluation of such observations:

(a) factors influencing the number of ADRs reported by the medical community (numerator

(b) quality of surrogates for the population exposed, for example the sales volume of the

(c) adequate statistical methodology to detect changes in ADR rates.

bias)

factory (denominator bias)

SURVEILLANCE OF ADVERSE DRUG REACTIONS 2385

Although statistical methodology should be used and will sometimes lead to an alert of a changing situation, it cannot be the sole basis for further decision such as the with- drawal of a batch from the market, unless the data quality has been properly taken into account. Especially in biologicals, for example vaccines or clotting factor products, where either the starting material or the production process is of biological nature, batch to batch variation can affect the incidence of associated adverse events. This can lead to time- dependent changes in the safety of a drug.

The United States Food and Drug Administration (FDA) requires a report in situations where an increase in the sales adjusted ADR rates per quarter is observed for certain categories of ADRs. As a consequence, the implementation and systematic use of alert systems for post- marketing surveillance have become more important. Any algorithm used as a basis for an alert system should be applied to the data not merely to fulfil statutory requirements, but also to detect a so far unrecognized new risk as early as possible.

2. DATA REQUIREMENTS

In order to detect an increase in the frequency of ADRs, the number and type of reported ADRs are required as well as sales data which reflect the drug usage in the corresponding periods.

With regard to the safety data, Norwood and Sampson3 suggested counting the number of adverse reactions reported to the producing company in the time intervals in which they were reported. In the following, this will be referred to as the number of reported adverse reactions. This is preferable to counting the number of reactions which actually occurred in the time intervals concerned, since ADR rates of the current period are usually compared with those of a given historical period. When using the number of reactions which occurred, the comparison of the ADR rates would be biased by the different durations of the follow-up periods of the current and the historical time intervals. This means that in comparison to the historical ADR rate, the current ADR rate would be deflated by the shorter follow-up period.

Country-specific differences in methods of reporting ADRs, legislation and drug usage make separate analyses necessary. According to the FDA? separate increased frequency detection should be made for domestic (US) and foreign ADR reports.

The ADRs reported to a pharmaceutical company are very heterogeneous. In the case of a vaccine, for example, a high percentage of the ADRs are injection site reactions such as injection site oedema, injection site hypersensitivity or pain. But, additionally, reports on serious ADRs are received which may lead to death, for example sudden infant death. Owing to these qualitative differences in the adverse reactions, it does not make sense to do a pooled analysis of all reported reactions. A classification of the various ADRs and separate analyses are necessary. The FDA requires a so-called increased frequency detection for the classes of ADRs summarized in Table 11.

In general, no direct measure of drug usage exists. For that reason, the size of the respective population at risk is estimated with the help of the manufacturer’s sales volume which reflects the drug consumption.

A quarterly analysis is proposed by the FDA, and the preceding quarter is generally taken as an adequate comparison period. In some circumstances it makes sense to choose an alternative comparison period especially if there are seasonal differences in the drug application. Thus influenza vaccines, for example, are mainly given in the period from September to December in Europe. In this case, an appropriate comparison would be of the ADR rates of the current period and of the same period in the preceding year.

2386 M. PRAUS ET AL.

Table I1 Classes of ADRs for which increased frequency detection is required by the FDA

All ADRs labelled serious According to the FDA a serious adverse event is any event that suggests a significant hazard, contramedi- cation, side effect, or precaution, including:

0 any event that is fatal or life-threatening 0 any event that is permanently disabling + any event that requires or prolongs hospitilization + any event that involves cancer or congenital anomaly, or occurs as a result of overdose.

All cases of death

ADRs occurring as a result of drug overdose

Failure of effect In case of the DT vaccine, a failure of effect is observed if a person falls ill with diphtheria or tetanus though he has been vaccinated.

Failure of effect which leads to serious ADRs

3. METHODS

In this section, methods to detect an increase in the sales adjusted ADR rate are reviewed. One should keep in mind that the statistical tests have only an alerting function and cannot be regarded as formal significance tests. The purpose of the alert is to draw attention to a possible problem. Further investigations are required to clarify if the suspicion is or is not founded. The proposed analyses should be carried out separately for each drug as well as for certain categories of ADRs.

For a period i let x i denote the number of reported adverse reactions and ci the corresponding sales or any other estimate of drug usage for this period. We denote the probability of a reported adverse reaction per sales unit in period i as pi. The marketing data ratio is M i = c i /c i - where ci- is sales in the historical (previous) period.

The statistical methods introduced in the next section will be illustrated by data on a diph- theria/tetanus vaccine (DT vaccine) produced by Behringwerke AG and marketed in West Germany for more than 30 years. The ADR data and the sales data presented in Figure 2 refer to West Germany in the period from 1986 to 1989. The sales data show seasonally dependent variations with peaks in the second or third quarter of each year. This pattern, however, is not reflected in the spontaneous ADRs.

The proposed analyses are carried out for two categories of ADRs: all serious ADRs, and all injection site reactions.

3.1. Arithmetic method

One of the proposals made by the FDA to determine an increase in frequency of ADRs is the ‘arithmetic’ method. This method postulates that a doubling of the sales adjusted ADR rates is a relevant increase and, whenever detected, a report has to be made to the FDA. Of course, this method can only be applied if the number of reported ADRs in the historical period is at least 1.

SURVEILLANCE OF ADVERSE DRUG REACTIONS 2381

2.6 1- 11 .4

0.2 1 0 0 0.0

1 I I I I I I I I I I I I I I I

1/86 II 1 1 1 IV 1/87 II Ill IV 1/88 II Ill IV 1/89 II Ill IV

Calendar Deriod

I Boseline volue: arithmetic mean of quarterly soles volume / ADR reports in 1986 is set t o 1 Index of ADR reports - - - - - Soles Index

Figure 2. DT vaccine: sales volume and ADRs, West Germany, 1986-1989

3.2. Poisson method

The F D A does not require a report if an increase from either 0 to 1 or 0 to 2 ADRs is observed. To detect increases from 0 to 3 or more reactions, they propose an asymptotic method termed the statistical (Poisson) method.

As adverse reactions are usually rare, the number of reported ADRs xi can be assumed to have a Poisson distribution with parameter cipi . Thus xi and and xi- can be modelled by independent Poisson random variables Xi and X i - and the following test specification could be the basis for an alert system:

H o : p i < p i - , versus H,: p i > pi-1.

An increase in ADR frequency is detected if the null hypothesis Ho is rejected.

given Xi + Xi- 1 , For p i = pi- Norwood and Sampson3 propose an exact test based on the conditional distribution of Xi for

this conditional distribution is binomially distributed in the form

Pr(Xi = x i [ Xi + Xi-, = y) = RXi(1 - R)y-xt, (:i )

where R = + ci) = MJ(1 + M i ) is a function of the sales ratio. The null hypothesis is rejected if the p-value satisfies

P-value = (;) Rj(1 - R)Y-j -= a for a prespecified a. j = x ,

2388 M. PRAUS ET A L

A value of a = 0.05 has been used because the FDA suggests this implicitly in proposing an asymptotic test which is described below. When the calculated P-value is smaller than 5 per cent an alarm is signalled and a report has to be submitted to the FDA.

The normal approximation to the above binomial distribution leads to the following asymp- totic version of the exact test:3

xi - (xi + xi-1)R [(xi + xi-l)R(l - R)]”” ZNS =

which is compared with za, the lOO(1 - a) percentile of the standard normal distribution. The FDA4 proposes a similar asymptotic procedure called the statistical (Poisson) method. It is

also based on the Poisson method. Increased frequency will be detected with the help of the following test:

xi - (Xi + xi- l)R [(l - R)’xi + R 2 ~ i - 1 ] 1 / 2 ’ ZFDA =

Simple algebraic transformation shows the following relationship between the ZNS test statistic proposed by Norwood and Sampson3 and the Z F D A test statistic proposed by the FDA:4

(1 - R)’xi + R2xi-l ‘I2 c R(l - R)(x~ + xi- 1 ) 1 ZNS = ZFDA

If no adverse reactions are reported in the historical period, xi - = 0, the sales data are not taken into account in the FDA method since Z F D A = ,/xi (in contrast ZNS = (Jxi)[J(l - R)/JR]). As the FDA suggests a = 005, an increased frequency is always detected if the number of ADRs in the current period equals or is higher than 3 following a comparison period with 0 events reported. In this case, the respective P - value for xi = 3 and x i - l = 0 is P = 0.04.

Norwood and Sampson3 compared the two asymptotic methods. The methods are the same mif R = 05, that is if the sales volumes of the current period and the comparison period are the same. If R < 05, which implies ci < ci- 1 , the FDA proposal is slightly liberal, and if R > 05, which implies ci > ci- 1 , it is slightly conservative.

For the DT vaccine, the resulting alerts based on the methods presented are given in Figure 1 . In the case of injection site reactions all methods lead to an alert in the second and fourth quarters of 1987. The asymptotic method proposed by the FDA leads to an additional alert in the first quarter of 1989.

For the serious ADRs the results of the different methods do not agree so well. The arithmetic method leads to frequent alerts, whereas neither method proposed by Norwood and Sampson3 gives rise to an alert. The asymptotic test suggested by the FDA leads to an alert in the second quarter of 1989. In this case no serious ADRs were reported in the comparison period of the previous quarter. Three serious ADRs reported here were therefore signalled without taking the sales data into account.

The main reason for the large discrepancy between the Poisson based methods and the arithmetic method is that in each quarter only a few serious ADRs are observed and the arithmetic method does not take variability into account.

SURVEILLANCE OF ADVERSE DRUG REACTIONS 2389

When comparing the arithmetic method and the tests based on the Poisson method, it is obvious that drug usage is always taken into consideration with the help of the ‘sales ratio’ or the ‘marketing data ratio’. No absolute sales data but only relative figures are used. But only the exact test based on the Poisson method and its asymptotic version take the sales data into account even if the number of adverse reactions in the comparison period is 0. All these methods fail to detect slowly increasing trends in the sales adjusted ADR rates. This is due to the principle on which these methods are based: the comparison of only 2 fixed periods.

3.3. Cumulative sum scores

The cumulative sum (cusum) chart, initially introduced by Page’ for industrial process control, later proved to be an efficient method for the monitoring of birth defects6 or other rare disease^.^ In these applications, the cusum statistic is the cumulative sum over time of the differences between observed case counts, such as birth defects, and a reference value, except that whenever the sum falls below zero, the statistic is reset to zero. An alert is signalled if the statistic exceeds a detection boundary. Usually the population at risk, for example the number of newborn children per month or the living population of a region, is assumed to be constant.

In the following, this procedure is adapted to a varying population at risk, represented by the sales volume. In general, a growing or decreasing market share of a pharmaceutical product may lead to this situation.

We assume that a background incidence level ko of reported ADRs per unit of sales volume is known, together with its standard deviation. This information may, for example, be taken from a run-in series of earlier (historical) periods. Let k l > ko be an increased incidence level which is considered to be the alert or rejection level. The cusum score Si for the period i is then defined in relation to its value at i - 1 by

Si = max 0, Si- + - - k, for i > 0, where So = 0.

Here k,, where ko < k, < kl , is called the reference level and is usually taken as the mean of ko and k l . In the definition xi and ci are the number of ADRs and the sales volume of period i as indicated earlier.

Whenever Si exceeds a detection boundary h, an increased incidence level is suspected (alert case); the average run length ARLO under the background incidence ko, the standard deviation of ko and the difference k l - k, will determine h and, at the same time, the average run length ARLl under an increased incidence level k l . The equations defining h and ARLl cannot be solved explicitly, but tables and nomograms for practical use are available.**

For the DT vaccine, cusum scores for injection site and serious adverse reactions are presented in Table 111. Detection boundaries were determined according to k l = 1.5 k o , k , = 1.25 ko and ARLO = 500; the rejection level therefore corresponds to a 50 per cent increase in the background incidence level ko which was derived from the year 1986. The false alarm rate l/ARLo was taken as 1 : 500, that is once in 125 years. The standard deviation of ko was derived from the 1986 periods.

According to Table 111, an alert would be signalled for injection site reactions in quarters IV of 1987, I1 of 1988 and IV of 1989. Serious ADRs were frequently lower than the reference level; the corresponding detection boundary was not exceeded between 1987 and 1989.

The main advantage of the cusum method is that it allows one to detect trends over time. Compared with the arithmetic and Poisson methods, even more data are required to apply the cusum method. Besides the number of adverse reactions and the sales volume for each period, the

( x i Ci 1

2390 M. PRAUS ET A L .

Table 111. Cusum scores for injection site and serious ADRs DT vaccine, 1986-1989

Baseline periods* (1/86-IV/86)

Injection site Serious ADRs ADRs

ko f SD (ADRs/litre)t 0.0574 f 0.0163 0.0200 f 0.0092 k , (ADRs/litre) 0.07 16 0.0250 k , (= 1.5ko) 0.0861 0.0300 ARLO 500 500 ARL, 3.8 8.1 h (detection boundary) 0.0435 0.0382

Observation Injection site ADRs Serious ADRs periods* Xi lc i Si XilCi Si

1/87 00058 0 0.0234 0 11/87 0.0954 00238 0.0045 0 III/87 0.0731 0.0252 0.0146 0 IV/87 0.1667 0.1203$ 0.0185 0

11/88 0.0995 00479$ 0.0043 0 III/88 0.1133 0.0417 0.0168 0

1/89 0.04 1 8 0 o.oOO0 0 11/89 0.0781 0.0065 0.0123 0 III/89 0.1047 0.0395 0.0050 0 IV/89 0.0809 0.0489$ 0.0062 0

1/88 0.0916 0.0200 0.0431 0.0181

IV/88 0.0329 0.0030 0.0274 0.0024

* Periods given in quarters (I-IV) and years. t 1 litre of DT vaccine corresponds to 2000 vaccinations. $ Detection boundary exceeded, score reset to 0.

background incidence and its standard deviation, an increased incidence level and a reference level must be given. Absolute figures of the sales volumes are needed, not relative ones.

Together with other methods mentioned in this paper, the cusum method takes the sales volume as a surrogate for the population exposed to the pharmaceutical product in question; whether this is appropriate is further discussed in the next section. There are many other statistical methods available that can serve as a basis for alert systems, including Poisson regression as proposed by Parker," scan functions" and time series.

4. DATA QUALITY PROBLEMS

In practice, an alert system may be affected either by bad data quality regarding the ADRs (numerator bias) or by the sales volume (denominator bias) or by both. In the literature, much attention is paid to the numerator bias. In a review, Sachs and Bortnichak'* identify the following biases which influence the spontaneous reporting of ADRs:

Length of time a drug has been on the market In general an increase of ADR reporting can be observed in the first few years that a drug has been on the market. Thereafter a decrease is detectable regardless of the sales volume.

SURVEILLANCE OF ADVERSE DRUG REACTIONS 239 1

Country-specijic differences Country-specific differences in the drug usage frequently exist. For that reason the respective populations at risk are not comparable. Furthermore, spontaneous reporting of ADRs is affected by differences in the reporting procedures and differences in the compliance to a reporting system.

Reporting environment Several extraneous factors including scientific and other publications concerning a certain drug or ADR lead to an increased level of publicity and increase the spontaneous reports of ADRs. Additionally, changes in the reporting system itself can affect the number of ADR reports.

Extent of interaction between physicians and manufacturer’s$eld sales force The field sales force comprises the manufacturer’s representatives who have closest contact with the physicians. A high percentage of ADRs are directly reported to them. The amount of time they spend with physicians or in giving special training can lead to increases of ADR reports.

These numerator biases can inflate or deflate the reported ADRs, so it is difficult to determine the effect of extraneous factors.

Far less attention is paid to the problem of denominator bias, which means that the sales volume may be a poor surrogate for the population at risk. No recommendations are given in the literature concerning to which sales data the number of adverse reactions reported in a given period i should be referred. Usually the figures for sales from the works are available but not the figures supplied by the wholesale trade or pharmacy. By choosing the sales figures of the period i as the basis for calculation, one implicitly expects that this quantity has been applied in the respective period. This assumption is not justified owing to warehousing on several levels (for example, wholesale trade or pharmacy) which causes a delay between the selling and application of the drug. Owing to this, increased frequency analyses can give biased results if the time lag between selling and application cannot be adequately estimated, that is if ADR data are related to sales volumes which do not reflect the drug usage of the corresponding period.

For the case of the DT vaccine we discuss if an estimate of the delay between selling and application is possible when using batch-specific adverse reaction and batch-specific sales data. Thus from 1 January 1986 to 31 December 1989,357 cases of ADRs to this drug were reported to the Drug Surveillance Department of Behringwerke AG. Batch numbers were available in more than 50 per cent of the reports. In Figure 3, the shelf life of a batch is compared with the periods in which ADRs of this batch were reported. Here the shelf life is defined as the period between sales from the works and the expiry date. The average shelf life per batch is 2 years. Several batches are available on the market at the same time and the periods of reporting ADRs vary. The delay between beginning of sales from works and the first reported ADR of a batch varies between 87 and 330 days. Thus no systematic constant delay is observed.

A11 in all, this shows that the sales volume of a period may be an inaccurate surrogate for the population at risk in this period. Accurate estimation of the population at risk seems to be extremely difficult. Consequently, alerts may be affected by this denominator bias.

One further problem is the delay between the occurrence of an adverse reaction and the report, and the delay between the exposure and the report (see Table IV). For the DT Vaccine, the median delay between the occurrence and the reporting of a reaction is 25 days, while the median delay between exposure and reporting is 30 days. As a consequence, 40 per cent of the ADRs reported in one quarter occurred in a previous quarter. It should be noted that 45 per cent of the adverse reactions reported in one quarter correspond to vaccinations given in a previous quarter.

2392

1 4 ~

1 3 -

1 2 -

11 - 10-

9 .

0 8 -

7 - m

6 -

5 -

4 -

3 -

2 -

1 -

r

M. PRAUS ET AL.

1 I I I I I I I

l 5 f

........... ADR report period __ Shelflife (beginning: sales ex works)

Figure 3. DT vaccine: ADR report periods versus shelf life per batch

Table IV. DT vaccine, domestic, quarter I of 1986 to quarter IV of 1989

Days of delay between Occurrence and Exposure and

ADR report ADR report (N = 209)* ( N = 238)*

Minimum Lower quartile Median Upper quartile Maximum

0 12 25 66

2766

0 14 30 86

8285

* N = number of ADR reports with the respective information.

5. DISCUSSION

Statistical methods which can serve as a basis for alert systems are available even if all of them have certain disadvantages. The main problem is that increased frequency reports may be affected by the quality of the safety and sales data. Spontaneous reporting of ADRs can be influenced by many extraneous factors which can result in a numerator bias.

Furthermore, the sales volume may be an inaccurate surrogate for the size of the population at risk (denominator bias). This bias can go both ways. On the one hand, it may result in a false alert. As a consequence of frequent false alerts, the system will not be accepted. On the other hand, the inadequate data quality may prevent the raising of an alarm although a change in the true ADR rate has occurred. This is one reason why all tests used as a basis for an alert system cannot be regarded as a formal significance test, but according to FinneyI3 only ‘as an index of suspicion’.

SURVEILLANCE OF ADVERSE DRUG REACTIONS 2393

Despite these problems, it is important t o look for changes in the ADR rates or in the safety profile of a drug. After the raising of an alarm one must investigate if a possible foundation for this suspicion exists or if it can be ruled out. In any event it may give important hints for the risk assessment of a drug. In cases, where the answer provided by an alert system is insufficient, case-control or prospective phase IV studies may become necessary.

REFERENCES 1. Machin, D. and Campbell, M. J. Statistical Tables for the Design of Clinical Trials, Blackwell Scientific

2. Lewis, J. A. ‘Post-marketing surveillance: how many patients? TIPS, 93-94 (April 1981). 3. Norwood, P. K. and Sampson, A. R. ‘A statistical methodology for postmarketing surveillance of

4. Drug Information Association. “Postmarketing Reporting of Adverse Reactions, FDA Division of

5. Page, E. S. ‘Continuous inspection schemes’, Biometrika, 41, 100-1 15 (1954). 6. Barbujani, G. and Calzdari, E. ‘Comparison of two statistical techniques for the surveillance of birth

7. Raubertas, R. F. ‘An analysis of disease surveillance data that uses the geographic locations of the

8. Kemp, K. W. ‘The use of cumulative sums for sampling inspection schemes’, Applied Statistics, 11, 16-31

9. Oakland, J. S. Statistical Process Control, Heinemann, London, 1986.

Publications, Oxford, 1987.

adverse drug reaction reports’, Statistics in Medicine, 7 , 1023-1036.

Epidemiology and Surveillance Workshop, Rockville, ,Maryland, 16th May 1988.

defects through a Monte Carlo simulation’, Statistics in Medicine 3, 239-247 (1984).

reporting units’, Statistics in Medicine, 8, 267-271 (1989).

(1962).

10. Parker, R. A. ‘Analysis of surveillance data with Poisson regression: a case study’, Statistics in Medicine,

11. Shore, D. L. and Quade, D. ‘A surveillance system based on a short memory scheme’, Statistics in

12. Sachs, R. M. and Bortnichak, E. A. ‘An evaluation of spontaneous adverse drug reaction monitoring

13. Finney, D. J. ‘The detection of adverse reactions to therapeutic drugs’, Statistics in Medicine, 1, 153-161

8, 285-294 (1989).

Medicine, 8, 31 1-322 (1989).

systems’, The American Journal of Medicine, 81 (suppl. SB), 49-55 (1986).

(1982).