Validation of diagnostic assays OIE general principles and OIE … · 2010-06-22 · Theory to Practice Melbourne 2003 OIE validation template and “Quality Standard” 2005 OIE

CSIRO.

Axel CollingCSIRO, Australian Animal Health Laboratory3219 Geelong, VictoriaAustralia [email protected]

Second Global Conference of OIE Reference Laboratories and Collaborating CentresOIE Headquarters, Paris, France, 21-23 June 2010

Validation of diagnostic assays OIE general principles and OIE procedure

CSIRO.

< 1990

Sensitivity

Specificity

Repeatability

ReproducibilityPrec

isio

n

Acc

urac

y

1996 2000 2001 2007R. JacobsonPrinciplesof test validation(serology)

M. Greiner & I. Gardner“Fit for purpose”and epidemiological considerations forassay validation

2010

Validation??? WAVLDValidationFromTheory to PracticeMelbourne

2003OIE validation templateand “QualityStandard”

2005

OIE ManualPrinciples & methods& best practice docs

Scotland Design & analysis of diagnostic test evaluationstudies (STARD)

Diagnosis ~ Validation ~ Epidemiology

FAO/IAEAmeetings2002/2003“fit for purpose”standardized a) validationb) recognition c) certification processes

2002

3 OIE ad hoc groupmeetings2008/9 & 2010

2008-9

CSIRO.

Strategy presented at WAVLD 2007, Melbourne….

- The “One-Set-of-Validation-Principles” concept is applicable to all types of diagnostic assays (Ab-Ag, NAD, TSE, etc.)

- One set of Validation Criteria applicable to all assay types (what has to be done?)

- A unified Validation Chapter

- Best Practice documents or annexes, which describe the processes for each assay type (how to do it?)

- A unified template would be accompanied by Unique Guidelines for each assay type

- OIE endorsed this proposal and supported its development through an ad hoc group for test validation

CSIRO.

1) One chapter on test validation, which includes Nucleic Acid Detection Tests (NADs). Before there were two chapters, e.g. one with a focus on serological tests and one for NADs.

2) The chapter includes principles and methods for test validation

3) The chapter has 7 annexes as best practice documents:(The first 3 focus on the development and optimization of diagnostic tests)

- 3.1) Development and optimization of antibody detection assays (Ab) - 3.2) Development and optimization of antigen detection tests (Ag) - 3.3) Development and optimization of nucleic acid detection tests (NAD)

- 3.4) Statistical approaches to test validation including (Latent Class Models) - 3.5) Method comparability (Equivalence testing)- 3.6) Measurement Uncertainty (MU) - 3.7) Selection and use of reference panels

Essential changes in the new approach to international test validation

CSIRO.

4) Guidelines for the use of the OIE validation template

5) “Provisional recognition”Example: Assays developed and used in emergency or outbreak situations. To be assessed: ASe, ASp, repeatability and an estimate for reproducibility

6) “Adjunct tests or procedures”Example: VNT to type an isolated virus or molecular sequencing to confirm a real time PCR result. To be assessed: ASe and ASp.

Essential changes in the new approach to international test validation (cont.)

CSIRO.

Assay Validation’s 3 components

• Assay Development Pathway

• Assay Validation Pathway with its 4 stages

• Validation Retention Pathway

Charting these components…

CSIRO.

OIE test development and validation pathway

Diagnostic Specificity

Study Designand

Protocol

Samples from experimentalanimals (where used)

Samples fromreference animals

Candidate test compared withStandard Test Method

Reagents and Controls

Ruggedness

Reproducibility

Repeatability and preliminaryReproducibility

STAGE 2Diagnostic

characteristics

Assay Validation Pathway

STAGE 1

Analyticalcharacteristics

Provisional Recognition

STAGE 4Implementation

Interpretation oftest results

International recognition (OIE)

Reference standards selected

Deployment to other Labs

Validationstatus

retention

Monitoring andmaintenanceof validation

criteria

Monitor precisionand accuracy

Daily in-house QC

Proficiency testing

Replacement of depleted reagents

Assay modifications and re-validation

Equivalency assessments

AssayDevelopment

Pathway Optimization, Robustness, Calibration to Standards

STAGE 3ReproducibilityDefine evaluation panel

Select collaborating Labs

Analytical Sensitivity

Analytical Specificity

Diagnostic Sensitivity


Cut-off determination

Essential Prerequisites

Fitness of Assay for its Intended Purpose

Study Designand

Protocol

Study Designand

Protocol





Candidate test compared withCandidate test compared with

Reagents and ControlsReagents and Controls

RuggednessRuggednessRuggedness

ReproducibilityReproducibilityReproducibility




STAGE 2Diagnostic

characteristics


STAGE 1


Provisional RecognitionProvisional Recognition




International recognition (OIE)International recognition (OIE)

Reference standards selectedReference standards selected

Deployment to other LabsDeployment to other Labs

Validationstatus

retention


criteria



Daily in-house QCDaily in-house QC

Proficiency testingProficiency testing

Replacement of depleted reagentsReplacement of

depleted reagents



Equivalency assessmentsComparability assessments

AssayDevelopment


Optimization, Robustness, Calibration to Standards


STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel

Select collaborating LabsSelect collaborating Labs

Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity

Analytical SpecificityAnalytical SpecificityAnalytical Specificity

Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity

Analytical SpecificityAnalytical SpecificityDiagnostic Specificity

Cut-off determinationCut-off determinationCut-off determination

Essential PrerequisitesEssential Prerequisites

Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents



Equivalency assessmentsEquivalency assessments

AssayDevelopment













Defining the purpose of the assay guides all subsequent step in the validation process.

Examples of purposes for which an assay may be used:

- Documenting freedom from disease in a country/region- Preventing spread of disease through trade- Eradication of an infection from a region or country- Confirming diagnosis of a clinical case or in a population of animals - Estimating prevalence to facilitate risk analysis- Classifying animals for herd health or immune/vaccination status

Fitness for intended purpose

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment













Essential prerequisites:

- Reagents availability & costs- Throughput capacity- Turn around time- Technical sophistication- Interpretation skills - QA/QC capability- Requirement for automation?

Essential prerequisites

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment













Considerable thought and planning is required for study design and protocol (biological, physical and chemical underlaying factors).

- reagents and controls (acquire reagents & instrument testing)- proof of concept experiments or feasibility studies- optimization and standardization (statistician)

(Provided in Best Practices appendices to the Chapter)

Study Design and Protocol

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment













Optimization is the process by which the most important physical, chemical, and biological parameters of an assay are evaluated and adjusted to ensure that the performance characteristics are best suited to the intended application.

Robustness is a measure of an assay’s capacity to remain unaffected by small changes or variations in test conditions as anticipated in routine laboratory operation.

Lack of robustness may be observed during optimization when critical reagent concentrations are not yet achieved.

(Examples provided in Best Practices appendices to the Chapter)

Optimization & Robustness

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment














CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment














Repeatability is the level of agreement between results of replicates of a sample both within and between runs of the same test method in a given laboratory.

Analytical specificity is the degree to which the assay distinguishes between the target analyte and other components in the sample matrix.

Analytical sensitivity is the smallest detectable amount of analyte in a sample that can be measured with a defined certainty.

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment














Diagnostic specificity is the proportion of known uninfected reference animals that test negative.

Diagnostic sensitivity is the proportion of known infected reference animals that test positive in an assay.

Cut-off and threshold are considered to be synonymous. Intermediate, indeterminate, suspicious or equivocal are terms used for a zone of test values between the positive and negative cut-offs.

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment














Reproducibility is the ability of a test method to provide consistent results when applied to aliquots of the same samples tested at different laboratories.

Ruggedness is a measure of the assay’s capacity to remain unaffected by substantial changes or substitutions in test conditions anticipated in multi-laboratory utilisation, part of fitness studies and reproducibility assessments (e.g. impact of shipping conditions, technology transfer, reagents batches, equipment, testing platforms and/or environments).

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment














http://www.oie.int/vcda/eng/VCDA_dossier.dot

Application Form for the Certification of Diagnostic Tests as validated fit for specific purposes

CSIRO.


Study Designand

Protocol





Ruggedness

Reproducibility


STAGE 2Diagnostic

characteristics


STAGE 1








Validationstatus

retention


criteria


Daily in-house QC

Proficiency testing




AssayDevelopment











Study Designand

Protocol

Study Designand

Protocol












STAGE 2Diagnostic

characteristics


STAGE 1









Validationstatus

retention


criteria






depleted reagents




AssayDevelopment













- Proper maintenance and upgrading of the assay requires an “on going validation”

Validation Status Retention

CSIRO.

1) Fitness for intended purpose(s)2) Optimisation3) Standardisation4) Robustness5) Repeatability 6) Analytical sensitivity7) Analytical specificity8) Thresholds (cut-offs)9) Diagnostic sensitivity10) Diagnostic specificity11) Reproducibility12) Ruggedness

12 criteria that must be addressed in the development and validation of diagnostic tests

CSIRO.

The new proposed OIE validation process

OIE validation navigation flowchart

Chapter 1.1.4. Principles and methods of validation of diagnostic assays for infectious diseases

OIE validation template

1.1.4.1.Development andoptimizationof Ab detection tests

1.1.4.2.Development andoptimizationof Ag detection tests by direct+ immunolo-gical means

1.1.4.3.Development andoptimizationof NucleicAcid Detectiontests (NAD)

1.1.4.4.Statisticalapproaches to validation

1.1.4.5.Methodcomparability

1.1.4.6.MeasurementofUncertainty(MU)

1.1.4.7.Selection and use of referencepanels

7 best practice documents or annexes

Guidelinesfor Immunological based assays, e.g. ELISA, CFT, Luminex,Ag, Ab.

Guidelinesfor NADs

Guidelinesfor TSEagentdetectionassays

3 guidelines documents

CSIRO.

- We have come a long way toward a strong science based assay development and validation model.

- Validation is a process that determines the fitness of an assay, which has been properly developed, optimised and standardised, for an intended purpose. Validation includes estimates of the analytical and diagnostic performance characteristics of a test.

- We are close to implementing detailed processes that will harmonize validation efforts and drive assay validation science toward excellence.

- FAO/IAEA/OIE have given green light for a consultants meeting in Vienna, 6-9 September 2010 to develop a roadmap for the implementation of modern OIE principles and methods for diagnostic test validation.

Conclusions

CSIRO.

- OIE ad hoc group for Test Validation and OIE

- Working group for test development and validationof Subcommittee for Animal Health Laboratory Standards Australia and New Zealand (SCAHLS)Joseph O’Keefe

- National Animal Health Laboratory Network, NAHLNBarbara Martin

- FAO/IAEA, Gerrit Viljoen, Adama Diallo

- Australian Animal Health Laboratory, AAHL John White, Ross Lunt Wilna Voslo, Jianning Wang Martyn Jeggo, Peter Daniels

Rear f.l.t.r. Peter Wright, Canada, Richard Jacobson, USAFront f.l.t.r.: Axel Colling, Australia, Ian Gardner, USA, Francois Diaz, OIEApologies: Katherine Webster, UK, Sandor Belak, Sweden

Acknowledgements

CSIRO.

Thank you for your attention!

Documents

Validation of diagnostic assays OIE general principles and OIE … · 2010-06-22 · Theory to Practice Melbourne 2003 OIE validation template and “Quality Standard” 2005 OIE