1

Validation of Qualitative Microbiological Test Methods

NCS Conference

Brugge, October 2014

Pieta IJzerman-Boon (MSD)Edwin van den Heuvel (TUe, UMCG/RUG)

2

Contents

• Introduction

• Statistical Detection Mechanisms

• Validation Issues

• Likelihood-Based Inference

• Conclusions

3

Introduction

Validation parameters Qualitative tests

Microbiological guidelines

Analytical guideline

Accuracy and precision EP

Repeatability USP

Specificity EP/USP ICH

Detection Limit EP/USP ICH

Ruggedness USP

Robustness EP/USP

• Guidelines on validation do not agree

4

Introduction

• In this presentation we will show an optimal validation strategy:–Compare methods–Two dilutions–Optimal densities for the two dilutions–Required number of samples

• Optimal validation strategy differs substantially from the guidelines

5

Statistical Detection Mechanisms

• Suppose a test sample is tested with a qualitative test

• The sample contains X organisms–X=0: sample is sterile–X>0: sample is contaminated

• The outcome of the test is Z –Z=1: positive test result–Z=0: negative test result

6

Statistical Detection Mechanisms

• Classification of test result

• So we need to look at the conditional probabilities

• The function describing this detection probability is referred to as the detection mechanism

Okay

False Positive

False Negative

Okay

Number of Organisms

X=0 X>0

Tes

t R

esul

t

Z=

0Z

=1

xX|ZPx 1

7

Statistical Detection Mechanisms

• Zero-deflated binomial mechanism:

– is the false positive rate: (0)=–p is the detection proportion: if =0 then it is the

probability to detect just one organism: (1)=p–If =0 and p=1 the test method is perfect– and p are related to specificity and accuracy–The binomial mechanism (=0) was introduced in

Van den Heuvel and IJzerman-Boon (2013)

0if111

0if

xp

xx x

8

Statistical Detection Mechanisms

=0, p=0.85 =0.10, p=0.70

9

Validation Issues

• Estimate detection mechanism via experiments–Exact low spikes of X cannot be generated–Hence the detection probability (x) cannot be

estimated, only the average proportion over samples

• Expected proportion of positive test results:–Assume that the number of organisms X ~ Poi(

peX 11E

10

Validation Issues

• The detection proportion p cannot be estimated–Without knowledge on the average number of

organisms in the test samples–With serial dilution experiments

• The false positive rate can always be estimated using samples from a blank dilution (=0)

Compare alternate with compendial method–Using the same for both methods–Likelihood ratio test (LRT)

11

Likelihood-Based InferenceExperimental Design

• Suppose we test samples from the same dilution with two methods–Alternate method: i=1–Compendial method: i=2–Dilution has on average organisms per sample

–Number of samples tested per method: n

• Expected proportion of positive results now depends on method i (i=1,2):

ipii e 11

12

Likelihood-Based InferenceExperimental Design

• Asymptotic distribution of LRT for comparing these proportions converges to -distribution with

• Hence, power can be optimized by maximizing–Bacterial density can be optimized independently

from sample size n–There is a single optimal density

)nc(21

))((

)(nnc

2121

221

2

2

nc

13

Likelihood-Based InferenceExperimental Design

Compendial: 2=0.01p2=0.95

14

Likelihood-Based InferenceSimulations

Simulation Results: Single dilution • Average density • Detection proportions pAL=0.7 and pCM=1

• Power (%) of likelihood ratio test LRT for differences in detection probabilities for various false positive rates

AL=CM=0 AL=0.05, CM=0 AL=CM=0.05

n=150 n=200 n=250 n=150 n=200 n=250 n=150 n=200 n=2501.01.52.02.53.0

65.369.270.167.564.8

74.680.481.880.476.8

85.888.589.689.284.4

49.157.360.460.257.8

56.168.572.371.971.9

68.579.682.782.879.8

61.067.267.363.961.3

71.777.379.077.375.0

83.286.187.885.982.9

15

Conclusions

Optimal strategy when parameters are unknown–Compare alternate with compendial method–Two dilutions are needed

1. Blank dilution

2. Dilution with on average ~2 organisms–Sample size should be at least n=200

• False positive rates can be tested with LRT

• Accuracy pAL/pCM can be tested with appropriate CIs as an alternative for the LRT for the ratio pAL/pCM

(IJzerman-Boon and Van den Heuvel, 2014)

16

Conclusions

• Differences with guidelines–Only specificity and accuracy need to be considered–Two dilutions are needed, using five 10-fold dilutions

is a loss of power–The optimal density is ~2 CFU/unit, ~5 CFU/unit is

much too high–Use 200 instead of 5 samples per method and dilution

to detect a 30% drop in accuracy with 80% power

17

References

• IJzerman-Boon PC, Van den Heuvel ER, Validation of Qualitative Microbiological Test Methods, Submitted, 2014.

• Van den Heuvel ER, IJzerman-Boon PC, A Comparison of Test Statistics for the Recovery of Rapid Growth-Based Enumeration Tests, Pharmaceutical Statistics, 2013; 12(5): 291-299.

• EP 5.1.6 Alternative Methods for Control of Microbiological Quality

• USP <1223> Validation of Alternative Microbiological Methods

• ICH Q2 (R1) Validation of Analytical Procedures