Validation of screening methods (2002/657/EC) N. Van Wouwe IPH AFSCA-FAVV

Validation of screening methods (2002/657/EC)

N. Van Wouwe

IPHAFSCA-FAVV

Definition (2002/657/EC)

Screening method : used to detect the presence of a substance

or class of substances at the level of interest. have the capability for a high sample

throughput

=> are used to sift large numbers of samples for potential non-compliant results.

Exemple: ELISA, plate test, biosensor, receptor test,…

Definition (2002/657/EC)

Minimum criteria to use an analytical method as screening method:

must be validated (traceability)

must have a false compliant rate of <5% (β-error) at the level of interest

Performance characteristics for method validation (screening)

Detection Capability

CCß

Decision Limit CC

Trueness/ Recovery

Precision Selectivity/ Specificity

Applicability/ Ruggedness/

Stability

Qualitative methods

S + - - - + +

Quantitative methods

S + - - + + +

+ = determination is mandatory

Qualitative method: identifies a substance on basis of its chemical, biological or physical propriety (binary response: +/-, absence/presence)

Quantitative method: determines the amount or mass fraction of a substance (response: numerical value of appropriate unit)

Validation of screening test

Definition of the scope of the method Analyte of group of analytes Range of concentration List of matrices

Initial validation with the most often used matrice in national monitoring program

Detection capacity (CCβ) Selectivity/Specificity Applicability/ Ruggedness/Stability Precision (only for semi-quantitative method)

If possible: different sources of blank material, different technicians, different days on the same spiked sample

Validation of screening test

Targeted test: for 1 compound validation for this compound

Targeted test: for a family of compounds validation for 1 representative molecule of the family (antibody)

Wide range test: for more than 50 different molecules Validation for at least a list of representative compounds

Common pattern of activity on a specific bacteria? Common way of action (acting target)? Published reference data on validation available?

Proposition of the CRL for antimicrobials (in milk)

Representative Compound Antimicrobial classCefalonium/Cephapirin/Cefquinome CEPHALOSPORINS

Penicillin G/Cloxacillin PENICILLINS

Tetracycline/Doxcycline TETRACYCLINS

Gentamicin/Streptomycin/Spectinomycin AMINOGLYCOSIDES

Enrofloxacin/Flumequine QUINOLONES

Sulfathiazole/sulfaguanidine/Sulfamerazine SULFONAMIDES

Erythromycin/Tylosin MACROLIDES

Lincomycin LINCOSAMIDES

Thiamphenicol PHENICOLATED

Trimethoprim/colistine MISCELLANEOUS

Performance characteristics

Detection capacity Selectivity/Specificity Applicability/ Ruggedness/Stability Precision (only for semi-quantitative

method)

Detection capability (CCβ)

The smallest content of the substance that may be detected, identified and/or quantified in a sample with an error probability of β In case of MRPL, CCβ= lowest concentration at which

the method is able to detect truly contaminated sample with a statistical certainty of 1-β

In case of MRL, CCβ= concentration at which the method is able to detect the MRL concentrations with a statistical certainty of 1-β


No permitted limit Analyse 20 blank materials => CCα = 3x signal/noise

Analyse 20 blank materials fortified at CCα

=> CCβ = CCα + 1.64 x SDRW

Calibration curve procedure (ISO 11843) Analyse of blank material fortified at 0 MRLP, 0.5 MRLP,

1 MRLP, 1.5 MRLP and 2 MRLP Plot analytical results (y-axis) vs concentration(x-axis) CCα = y-intercept (blank) + 2.33 x SDRW

CCβ = CCα + 1.64 x SDRW


No permitted limit If no quantitative results

Analyse fortified blank samples at and above CCα

(n ≥ 20 / concentration level) CCβ = concentration level where only ≤5% false compliant

results remain


CC

Signal orConcentration

CC+2.33xSDblank

+1.64xSDRW

Blank

α=1% β=5%


Permitted limit (MRL) Analyse 20 blank materials fortified at MRL

=> CCα = MRL + 1.64 x SDRW

Analyse 20 blank materials fortified at CCα

=> CCβ = CCα + 1.64 x SDRW

Calibration curve procedure (ISO 11843) Analyse of blank materials fortified at 0.5 MRL, 1 MRL, 1.5

MRL and 2 MRL Plot analytical results (y-axis) vs concentration(x-axis) CCα = MRL + 1.64 x SDRW

CCβ = CCα + 1.64 x SDRW


CC


CCMRL

+1.64xSDMRL +1.64xSDRW

β=5%α=5%



method)

Selectivity/specificity

Specificity: ability of a method to distinguish between analyte being measured and other substances

problem of interference?

F(measuring technique, class of compounds, matrices,…)


How to test specificity for qualitative screening method? Analyse 20 different blank samples and 20 positive samples

(blind study, same or different days/technicians)

True positive (N+)

True negative (N-)

test result positive

Positive agreement

(PA)

False positive (FP)

test result negative

False negative (FN)

Negative agreement (NA)

Specificity= 100* NA/N-

Other parameters:

Accuracy= 100* (PA+NA)/(N- + N+)

Sensitivity= 100* PA/N+

False positive= 100* FP/(N- + N+)

False negative= 100* FN/(N- + N+)


How to test specificity for semi-quantitative screening methods? Select potentially interfering substances (metabolites,

derivatives,…) Analyse relevant blank samples (n ≥ 20) Analyse fortified blank samples with interfering substances at a

relevant concentration Estimate the effect of the interferences

False identification? Influence in quantification? Identification of the target analyte is hindered?



method)

Applicability

Scope of the method must be define in term of : Matrix (solid/liquid matrix, type of tissue) Animal species

To introduce a new matrix Analyse at least 10 different blank material fortified at level

of interest for the new matrix (CCβ) + test of interferences If 10 positive results => method applicable for the new matrix If 1 negative result => 10 additional analyses

If 1 negative result=> CCβ must be recalculated for the new matrix

Ruggedness

Ruggedness: the susceptibility of an analytical method to changes in experimental conditions

sample material analytes storage condition environmental condition sample preparation condition

Ruggedness

How to test ruggedness? (during development) Identify possible factor that could influence the results

(the analyst, solvents, pH, T°, rate of heating,…) Vary each factor slightly If one factor is found to influence results of the

representative molecule, conduct further experiments

=> acceptability limits for this factor

(in the method protocol)

Recommendation of CRL: analyses of 10 blank and 10 spiked samples at the same concentration and with minor change of factor to detect influence on results

Stability Test are not necessary if stability data already

exist (from other lab or from publication) To include in the validation report

Stability test: the analyte in solution the analyte in matrix

Aliquots of a fresh solution or sample stored under different conditions (T° and/or storing time)



method)

Precision (for quantitative screening) Precision: the closseness of agreement

between independent test results obtained under predetermined conditions

Expressed in terms of imprecision / standard deviation of test results

How to test precision? Repeatability test within-laboratory reproducibility test(or intermediate

precision) Reproducibility test (between laboratories: interlaboratory

studies)

determination of RSD (%) < Precision criteria

Precision (for quantitative screening)

Precision (for quantitative screening) Repeatability

3 concentrations: 1x; 1,5x; 2x MRPL 0,5; 1x; 1,5x MRL

6 replicates/level 3 times same conditions

Within-laboratory Reproducibility 3 concentrations:

1x; 1,5x; 2x MRPL 0,5; 1x; 1,5x MRL

6 replicates/level 3 times different conditions

(analyst, env. condition,…)

Precision (for quantitative screening) ANOVA treatment of data => RSDr & RSDRW

Comparison with precision criteria: Horwitz equation: RSDR(%) = 2(1-0.5logC)

Criteria for repeatability: RSDr = 1/2 to 2/3 RSDR

Criteria for within-lab reproducibility: RSDRW = 2/3 to 1 RSDR

! For concentration < 100 µg/kg, RSDR becomes too high!

Other recommendations False negative rate <5%: Analyses of 20 negative and 20

positive samples in order to test the screening method (see selectivity).

One QC sample must be added in routine and results must be added to the validation file

Method transfer/Commercial test Bibliographical survey to compil the evaluation of performance of the

test Collection of data from supplier on validation study Experimental plan to test skillness of technician to perform the test Use of QC sample Participation to proficiency test

Exemple: analyse of PCDD/F by CALUX bioassay

PCDD/F: 17 toxic congeners to analyse in various matrices (TCDD=most toxic dioxin)

Results expressed in TEQ (=Sum (CCixTEFi)i=1-17) MRL for each matrix (milk, meat, egg, fish oil,…) MRL expressed in pg TEQ/g fat or ng TEQ/ kg Reference method: GC-HRMS Screening method: immunoassay, bioassay,…

Exemple: analyse of PCDD/F by CALUX bioassay

Gene expression

LIGHT

All substances fixing the Ah receptor

CALUX bioassay= genetically modified cell-based bioassay (luciferase)

Amount of light produced is proportional to the toxicity (TEQ) of extracts

Analyse of PCDD/F by CALUX bioassay Advantage:

RapidCheaper than GC-HRMSTime for analyses

Disadvantage:Various compounds can fix the Ah receptor

(PAH, PCB, PHDD/F,…)

specificity!!!!

Analyse of PCDD/F by CALUX bioassay : protocol

Extraction of fat

Clean-up on silica acid + carbon columns

Fraction with PCBs

Fraction with PCDD/F

Evaporation

Dosing plateReading plate

Fraction with interfering compounds

Analyse of PCDD/F by CALUX bioassay : validation

Selectivity/specificity Ruggedness/Stability Precision Detection capability

Analyse of PCDD/F by CALUX bioassay : selectivity

Possible interfering compounds? PAH : mostly in environmental sample PCB: fractionation during clean-up Other compounds? (PHDD/F): dependant of the matrix? (matrix

effect?)

Results of the selectivity test: No interferences for feedstuff, milk, egg, fat Interferences for fish oil

CALUX results = 2 x GC-HRMS results

Analyse of PCDD/F by CALUX bioassay : selectivity Matrix effect for fish oil

y = 1,9161x + 1,999

R2 = 0,9287

0

5

10

15

20

25

0 2 4 6 8 10 12

GC-HRMS TEQ value (pg TEQ/g oil)

CA

LU

X T

EQ

va

lue

(p

g T

EQ

/g o

il)

Analyse of PCDD/F by CALUX bioassay : ruggedness What are the critical point in the protocol?

Carbon column (interferences)Solvent (interferences)Curve (results)Evaporation time (recovery)Age of CALUX cell line (RSD)

Analyse of PCDD/F by CALUX bioassay : ruggedness Carbon column: amount of carbon used

0,7cc XCARB

0

10

20

30

40

50

60

70

80

90

100

PCB81

PCB77

PCB123

PCB118

PCB114

PCB105

PCB126

PCB167

PCB156

PCB157

PCB169

PCB189

% r

eco

very

1cc XCARB

0

10

20

30

40

50

60

70

80

90

100

PCB81

PCB77

PCB123

PCB118

PCB114

PCB105

PCB126

PCB167

PCB156

PCB157

PCB169

PCB189

% r

eco

very

1,4cc XCARB

0

10

20

30

40

50

60

70

80

90

100

PCB81

PCB77

PCB123

PCB118

PCB114

PCB105

PCB126

PCB167

PCB156

PCB157

PCB169

PCB189

% r

eco

very

1,9cc XCARB

0

10

20

30

40

50

60

70

80

90

100

PCB81

PCB77

PCB123

PCB118

PCB114

PCB105

PCB126

PCB167

PCB156

PCB157

PCB169

PCB189

% r

eco

very

DX fraction

PCB fraction

Not collected fraction

(Rdt PCDD/F= 60%) (Rdt PCDD/F= 80%)

(Rdt PCDD/F= 80%)

Analyse of PCDD/F by CALUX bioassay : ruggedness Evaporation time

0

1

2

3

4

5

6

7

8

0 5 10 15 20 25 30

evaporation time with dry extract

CA

LU

X r

esp

on

se

Analyse of PCDD/F by CALUX bioassay : ruggedness

Solvent: tested before use on a TCDD solution (antagonist/agonist effect)

Curve: tested with an independant TCDD solution

Age of CALUX cells: new cell every 2 months

Analyse of PCDD/F by CALUX bioassay : precision

Validation protocol

Day 1 Day 2 Day 3 Day 4 Day 5 Day 6

Blank solvent 1 1 1 1 1 1

Blank sample 1 1 1 1

Sample at MRL/2 3 3 2

Sample at MRL 6 2 3 3

Sample at 2MRL 3 2 3

Quality sample 1 1 1 1 10 10

Analyse of PCDD/F by CALUX bioassay : precision

ANOVA results for the TEQ determination of PCDD/F in

feedstuff by CALUX bioassay At MRL (0.75ng TEQ/kg) : XMRL= 0.751 ng TEQ/kg

Sr= 0.063 => RSDr= 8.4%

SRW=0.073 =>RSDRW= 9.7%

At MRL/2 (0.376ng TEQ/kg) : XMRL/2= 0.464 ng TEQ/kg

Sr= 0.051 => RSDr= 11%

SRW=0.051 =>RSDRW= 11%

At 2MRL (1.5ng TEQ/kg): X2MRL= 1.571 ng TEQ/kg

Sr= 0.107 => RSDr= 6.8%

SRW=0.115 =>RSDRW= 7.3%

RSD < 30%

(2002/70/EC)

Analyse of PCDD/F by CALUX bioassay : detection capacity CCβ for the TEQ determination of PCDD/F in feedstuff

by CALUX bioassay

CCα = MRL + 1.64 x SRW

CCα = 0.75 + 1.64 x 0.073 = 0.87 ng TEQ/kg

CCβ = CCα + 2.33 x SRW

CCβ = 0.87 + 2.33 x 0.073 = 1.04 ng TEQ/kg2002/70/EC: false negative rate < 1% !

=> At a concentration of 1.04ng TEQ/kg, we are sure that the sample is a positive sample with 99% certainty

Analyse of PCDD/F by CALUX bioassay :confirmatory range

MRL CC


CC

NON COMPLIANTCOMPLIANT SUSPICIOUS

CC*+1.64MRL +2.33sample

-2.33MRL

?

α=5%β=5% * =1%

Analyse of PCDD/F by CALUX bioassay :confirmatory range

Lower limit of the confirmatory range for the TEQ determination of PCDD/F in feedstuff by CALUX bioassay CC*= MRL-2.33 x SDRW

CC*= 0.75- 2.33 x 0.073 = 0.58 ng TEQ/kg

Conclusion Sample lower than 0.58 ng TEQ/kg are negative with 99%

certainty (false negative rate < 1%) Sample above 0.58 ng TEQ/kg must be confirmed by GC-HRMS

Documents

Validation of screening methods (2002/657/EC) N. Van Wouwe IPH AFSCA-FAVV