OIE principles and methods for validation of … principles and methods for validation of diagnostic tests Overview of presentation Manual chapter on validation and guidelines Stages

Ian Gardner E-mail: [email protected]

OIE principles and methods for validation of diagnostic tests

Overview of presentation

Manual chapter on validation and guidelines Stages in the validation pathway

Statistical methods guidelines (stage 2)

Modifications for wildlife diseases (stages 1 and 2)

Coherence between terrestrial and aquatics manuals

Further guidance is needed

Experimental challenge studies to estimate DSe/DSp

Principles & Methods of Validation of Diagnostic Assays for Infectious Diseases

Updated chapter – approved May 2013

Identical chapter in both manuals because principles and methods are same

Chapter 1.1.5 – Terrestrial Manual Chapter 1.1.2 – Aquatics Manual

Work in progress!

Does not provide explicit guidance on multiplex assays or new technologies

Study Design

and

Protocol

Assay Validation Pathway

Study Design

and

Protocol

Intended Purpose(s) for assay

Essential Prerequisites

Study Design

and

Protocol


Study Design

and

Protocol

STAGE 1

Analytical characteristics

Repeatability and preliminary Reproducibility



Analytical Specificity

Analytical Sensitivity

Study Design

and

Protocol

STAGE 2

Diagnostic characteristics


Study Design

and

Protocol

STAGE 2

Diagnostic

characteristics

STAGE 1







Diagnostic Specificity

Diagnostic Sensitivity

Cut-off Determination Latent class analysis to deal with imperfect reference standard

Traditional approach assuming a perfect reference standard

Study Design

and

Protocol

STAGE 2



STAGE 3

Study Design

and

Protocol

STAGE 2

Diagnostic

characteristics

STAGE 1


Reproducibility

STAGE 3








Cut-off Determination

Study Design

and

Protocol

STAGE 2



STAGE 4

Implementation

STAGE 3

Study Design

and

Protocol

STAGE 2

Diagnostic

characteristics

STAGE 4

STAGE 1


Reproducibility

STAGE 3

Implementation

STAGE 4









Deployment to other Labs

Study Design

and

Protocol

STAGE 2



STAGE 4

Implementation

STAGE 3

Study Design

and

Protocol

STAGE 2

Diagnostic

characteristics

STAGE 4

STAGE 1


Reproducibility

STAGE 3

Implementation

STAGE 4










International recognition (OIE)

Provisional recognition

Validated for original intended purpose(s)

Adjunct tests validated

Study Design

and

Protocol

STAGE 2



STAGE 4

Implementation

STAGE 3

Study Design

and

Protocol

STAGE 2

Diagnostic

characteristics

STAGE 4

STAGE 1


Reproducibility

STAGE 3

Implementation

STAGE 4

Proficiency testing Daily in-house QC

Monitoring and maintenance of validation

criteria










Guidelines to complement Manual chapter (7 completed, 1 under study)

Antibody detection assays

Antigen detection assays

Nucleic acid detection assays

Method uncertainty

Statistical approaches to validation

Reference samples and panels

Wildlife (modification of chapter )

Comparability of assays after minor changes in a validated test method (under study) Axel Colling and colleagues at AAHL, Geelong

Yes No

Perfect reference standard

Single binary

candidate test

Single continuous

candidate test

DSe and DSp (95%

exact CI)

DSe and DSp (95% exact

CI) and Area under ROC

curve (95% CI)

Binary candidate and

reference test

DSe and DSp (95% CI or PI)

by latent class methods

DSe and DSp (95% exact

CI) for relevant

subpopulations

DSe and DSp (95%

exact CI) for relevant

subpopulations

Guideline on statistical approaches to validation Stage 2: diagnostic sensitivity (DSe) & specificity (DSp)

Guideline on statistical approaches to validation Stage 2: ROC analysis - cutoff independent method

Latent class analysis (LCA) for estimation of diagnostic sensitivity & specificity

Challenges in PCR validation

Traditional statistical methods can’t demonstrate

PCR to be more sensitive than an imperfect reference

test (e.g. virus isolation [VI])

LCA solves the problem; allows calculation of

probability that sensitivity of PCR > sensitivity of VI

(and similarly can show specificities are comparable)

Example: African Horse Sickness PCR

(Horse diseases -- parallel session)

Guideline on validation of tests for wildlife diseases

Challenges:

Stage 2: difficulty in obtaining sufficient samples for estimation of DSe and DSp

Experimental infections may be only source of reference samples

Regulations limiting or prohibiting possession and international shipment of samples

Poor sample quality

Limited knowledge of pathogenesis/epidemiology of many diseases

“Sample size credit” for a validated test in a taxonomically-related wildlife species

Provisional recognition (national authorities) at stage 2a for an intended purpose(s)

Validation of tests for wildlife diseases

Effect of reduced sample size

Note: samples must be representative of the target condition of interest (e.g. infected vs diseased) and appropriate for designated purpose

Disease-specific Manual chapters

Questions

Are the approach and terminology similar in reporting of diagnostic characteristics in aquatics and terrestrial manual?

Methods

Manual chapters approved May 2014

Aquatics (all 4)

IHHN (shrimp), infectious salmon anemia and salmon alphavirus (fish), herpesvirus-1 microvariant (oysters)

Multispecies (3/7)

Paratuberculosis, Leishmanosis, Bluetongue

Bluetongue

+++ = recommended method; ++ = suitable method; + = may be used in some situations, but cost,

reliability, or other factors severely limits its application; – = not appropriate for this purpose. Although not all of the tests listed as category +++ or ++ have undergone formal validation, their routine nature and the fact that they have been used widely without dubious results, makes them acceptable.

Infectious salmon anemia virus

a = recommended for reasons of availability, utility, DSe and DSp b = standard method with good DSe and DSp c = method has application in some situations (cost, accuracy and other factors limit its application d = method not recommended for this purpose These are somewhat subjective………. Although not all the tests listed as category a or b have undergone formal standardisation and validation, their routine nature and the fact that they have been used widely without dubious results makes them acceptable

Experimental studies to estimate DSe/DSp

Aquatic animals and wildlife, in particular

Key questions

Is the design appropriate and does it adequately mimic natural exposure/infection?

What are the key elements that should be reported in a peer-reviewed publication to allow that assessment by readers/end-users?

Scenarios needing additional guidance for stage 2 validation

c) Experimentally infected or vaccinated reference animals

“ Samples obtained sequentially from experimentally infected or vaccinated animals are useful for determining kinetics of antibody responses ……. Single time-point sampling of individual experimental animals can be acceptable [for estimation of DSe] (e.g. one sample randomly chosen from each animal). Nevertheless it should be noted that for indirect methods of analyte detection, exposure to organisms under experimental conditions, or vaccination, may elicit antibody responses that are not quantitatively and qualitatively typical of natural infection in the target population (Jacobson, 1998). The strain of organism, dose, and route of administration to experimental animals are examples of variables that may introduce error when extrapolating DSe and DSp estimates to the target population. In cases when the near-impossibility of obtaining suitable reference samples from naturally exposed animals necessitates the use of samples from experimental animals for validation studies, the resulting DSe and DSp measures should be considered as less than ideal estimates of the true DSp and DSe”

Thank you

Documents

OIE principles and methods for validation of … principles and methods for validation of diagnostic tests Overview of presentation Manual chapter on validation and guidelines Stages