3B-Using CLSI Guidelines to

8/12/2019 3B-Using CLSI Guidelines to

1/53

USING CLSI GUIDELINES TO

PERFORM METHOD EVALUATIONSTUDIES IN YOUR LABORATORY

James Blackwood , MS, CLSI

David D. Koch, PhD, FACB, DABCC, Pathology & LaboratoryMedicine, Emory University School of Medicine

Breakout Session 3BTuesday, May 1

8:30 10 am


2/53

Outline

Learning Objectives Identify the seven performance characteristics that should

be evaluated prior to reporting results from a new test

method.

Verify the claims of manufacturers regarding analyticalperformance by following CLSI guidelines.

Demonstrate ongoing compliance with the method

evaluation criteria contained in accreditation guidelines.

Who is CLSI and what are these guidelines?

Method evaluation basic definitions and experiments

Use of the CLSI Evaluation Protocols

Use of StatisPro software in method evaluation


3/53

Who is CLSI? Clinical and Laboratory Standards Institute

ANSI-accredited, global, nonprofit standards development organization

CLSI has over 2,000 members organizations such as IVDmanufacturers, hospital laboratories, reference laboratories, universities,

professional associations, and government agencies

We promote the development and use of voluntary consensus standardsand guidelines within the health care community.

Our documents help health care organizations meet their responsibilitieswith efficiency, effectiveness, and global acceptance.


4/53

We Make the Blue Books


5/53

Standards in the Clinical Laboratory

Goal of standardization in thelaboratory:

The right laboratory testat the right time with

the right result leads toquality diagnostics andimproved patient care, andimproved public health

around the world.

Standardized test

Standardized

procedure

Standardizedreporting

Improved outcomes


6/53

CLSI documents are developed by volunteer experts from threedistinct constituencies: professions, government, and industry.

Under the supervision of a consensus committee, these volunteerswork on:

Document Development Committees

or

Standing subcommittees and Working groups

How are CLSI Documents developed?


7/53

CLSI Consensus Process

Balance

Government

Industry

Professions


8/53

Why are CLSI GuidelinesImportant?

The US Food and Drug Administration(FDA) recognizes over 100 CLSIdocuments.

The College of American Pathologists (CAP) recognizes 80 CLSIdocuments.

The Joint Commission recognizes over 144 CLSI documents.

All Evaluation Protocol guidelines in this presentation are recognized

by all three groups.


9/53

Why are Evaluation ProtocolsImportant?

Provide clear and thorough guidance.

Evaluation protocols are guidelines for clinical laboratories andmanufacturers to characterize the performance of their analyticalsystems.

Ensure consistency with good laboratory practice.

Good laboratory practice requires clinical laboratories to verifyperformance claims before reporting results used for decisions aboutpatient care.

Help you to comply with the law!

Evaluation of performance characteristics is required by regulatory andaccreditation bodies in the United States.

See http://www.cms.hhs.gov/clia (493.1255).


10/53

CLSI and Evaluation ProtocolsCLSI has over 25 Evaluation Protocol Guidelines.

These include:

EP05 Evaluation of Precision

EP06 Evaluation of Linearity

EP09 Evaluation of Bias and Comparability Using Patient SamplesEP10 Preliminary Evaluation (Bias, Carryover, Drift, Linearity)

EP15 Verification of Precision and Trueness

EP17 Limits of Detection and Limits of Quantitation

C28 Defining, Establishing, and Verifying Reference Intervals


11/53

Performance CharacteristicsThe seven performance characteristics that should be evaluated before

reporting results of a new test method include:

1. Precision

2. Accuracy (measured bias) or comparability (measured differences)

3. Linearity over the measuring interval or analytical measurementrange (AMR)

4. Limit of detection (LoD) and limit of quantitation (LoQ or analyticalsensitivity)

5. Specificity or interference

6. Reagent or sample (analyte) carryover

7. Reference interval or decision value (interpretive information)


12/53

Precision & Accuracy


13/53

Apply a clinical perspective; set a target, an analyticalgoal, before you begin

Perform experiments that gather representative data

about a methods analytical performance

Convert data into statistical estimates of errors

Compare error estimates to specifications for medically

allowable error for an objective assessment of theerrors


14/53

Introducing a New MethodEstablisha Need

MethodSelection

MethodEvaluation

MethodDevelopment

Define the

Quality Goal

Implementation

RoutineAnalysis

SubmitSpecimen

Quality ControlPractices

PreventiveMaintenance

ReportResult


15/53

APPROACH IN METHOD EVALUATION:Evaluate imprecision and inaccuracy

IMPRECISION Refers to Random Analytic Error(Lack of repeatability, reproducibility)

INACCURACY Refers to Systematic Analytic Error(Lack of trueness)

1. Constant2. Proportional

TOTAL ERROR Combined error for a single result


16/53

RELIABLE DECISIONS ABOUT PERFORMANCEREQUIRE:

1. Standards for acceptable performance

2. Experimental protocols to estimate

performance reliably

3. Criteria for comparing estimatedperformance with performance standards


17/53

PERFORMANCE STANDARD (PS)PERFORMANCE STANDARD (PS)

Specify:

EA . . . Allowable error

XC . . . Decision level

Format:

PS = EA at XC


18/53

ALLOWABLE ERROR (EA)ALLOWABLE ERROR (EA)

The amount of error that can be tolerated without

invalidating the medical usefulness of the result

or

causing the test to fail a proficiency testing event


19/53

DECISION LEVEL (XC)DECISION LEVEL (XC)

Any concentration of the analyte that is critical formedical interpretation whether for

diagnosis,

monitoring, or

therapeutic decisions.

Laboratory data are most carefully interpreted at

these decision level concentrations.


20/53

plasma glucose, mg/dL

1 2 3

0 20 50 80 126 160 200 260 300 340

DECISION LEVELS FOR GLUCOSEDECISION LEVELS FOR GLUCOSE


21/53

Performance standards for Glucose Medical Decision

PS1 = 6.0 mg/dL @ 50 mg/dLHypoglycemia

PS2 = 10% = 12.6 mg/dL @ 126 mg/dL Impaired glucose control

PS3 = 30 mg/dL @ 300 mg/dL Poorly controlled diabetes

DECISION LEVELS FOR GLUCOSEDECISION LEVELS FOR GLUCOSE


22/53

Sources of Allowable ErrorsSources of Allowable Errors

1. Proficiency testing requirements for acceptable performance

2. Literature guidelinesa. based on physician surveys

e.g.: Karon, Boyd & Klee, Glucose Meter Performance Criteria for TightGlycemic Control Estimated by Simulation Modeling, Clin Chem, 2010; 56:

1091-97

b. based on intra-individual biological variation of analyte Ricos C et al., Scand J Clin Lab Invest,1999; 59: 491-500

Fraser C, Biological Variation: From Principles to Practice, AACC, 2001

Internet at http://www.westgard.com/biodatabase1.htm

3. Input from clinicians and/or professional judgment


23/53


24/53

Performance Characteristics: Precision

CLSI Guidelines for Precision

EP15: a five-day procedure to verify that imprecisionmeets the claims of a measurement procedure

(EP15 is most frequently used by clinical laboratories for

method evaluation.)

EP05: a 20-day procedure to establish the imprecision fora measurement procedure


25/53

Replication Experiment

1. Time period: within-run

within-dayday-to-day

2. Number of samples: minimum of 20

3. Sample matrix: simulate patient sample

4. Analyte concentration: medical decision limit

5. Calculations: mean, standard deviation (SD),coefficient of variation (CV)


26/53

Performance Characteristics: Accuracy

Accuracy [Trueness] (Measured as Bias)

Bias: Estimate of a systematic measurement error; a quantitativemeasure of the average difference between results from ameasurement procedure and results from an accepted referencemeasurement procedure.

When a reference measurement procedure is not available for ananalyte, a best-available comparative method may be used tomeasure bias.

Frequently, clinical laboratories perform a comparison of patientsample results between a new and an existing measurementprocedure.

(correlation studies)


27/53

CLSI Guidelines for Trueness (Measured as Bias)

EP15: a method comparison to verify that a new methodconforms to a manufacturers claim for comparability toanother procedure.

(minimum of 20 patient samples) EP09: a method comparison to establish a claim for

method comparability.

(minimum of 40 patient samples)

Performance Characteristics: Accuracy


28/53

Comparison of Methods Experiment

CLSI EP9-A:

User Comparison of Quantitative Clinical Laboratory Methods

Using Patient Samples

1. Choice of comparative method:

critical for the conclusions which can be made

2. Number of test samples:

minimum N = 40

uniform distribution (EP9-A includes a table) a bin-box approach


29/53


30/53


31/53

Three Approaches to AnalyzingComparison of Methods Data

1. correlation coefficient

2. t-test statistics

3. regression statistics


32/53

Sensitivity of Statistical Parametersto Errors

Parameter Random Constant Proportional

LEAST SQUARES

SLOPE no no yes

Y-INTERCEPT no yes noSTD. ERROR yes no no

T-TEST

BIAS no yes yes

sd yes no yes

CORRELATION COEFFICIENTr yes no no


33/53

Effect of range on the correlation coefficient

Range 0 to 300 70 to 110

Random Error 10 units 10 units

Corr. Coef., r 0.986 0.764

H

H


34/53


35/53

Linear regression statistics

Subject to certain limitations:

Data must be linear

Outliers must be carefully examined

Range of data must be wide:

a. r > 0.99 (Waakers et al.)

b. r > 0.975 (CLSI EP9-A)


36/53

Recommendations for MethodComparison

Summary

Present graph of data

Present slope, y-intercept, and Sy/x

Present mean and standard deviation of X data Present correlation coefficient ONLY to show that least

squares regression is applicable; if not, use Deming or

Passing-Bablock regression statistics


37/53

Performance Characteristics:LinearityLinearity Measuring Interval or Analytical Measurement Range (AMR)

A linearity study is used to establish or verify the measuring interval for ameasurement method.

Measuring Interval: the interval between lower and upper numerical

values for which a method can produce quantitative results suitable forthe intended clinical use.

The measuring interval is verified by demonstrating a linear relationship

between the measured and expected concentration relationships.

CLSI Guideline for Linearity Measuring Interval

EP06: procedures to verify or establish the linear measuring interval of ameasurement procedure.


38/53

Performance Characteristics:

LoD/LoQLimit of Detection (LoD) & Limit of Quantitation (LoQ)(sometimes referred to as Analytical Sensitivity)

LoD: the lowest amount of analyte (measurand) in a sample that can be

detected with a stated probability.

LoQ: the lowest amount of analyte (measurand) in a sample that can be

quantified with acceptable precision and bias under stated experimentalconditions.

Usually, laboratories review and accept the manufacturers claims for LoDand LoQ.

But these characteristics can be tested by laboratories using:

CLSI Guideline for LoD and LoQ

EP17: procedures for verifying or establishing the LoD and the LoQ


39/53


Interference Interference: an artifactual increase or decrease in the apparent quantity

of an analyte due to the presence of a substance that reactsnonspecifically with the measuring system.

Most manufacturers evaluate a large number of substances known orsuspected to be potential interferents. They report this information in the

Instructions For Use (IFU). It is not practical for most clinical laboratories to repeat such an

investigation and inspection of the manufacturers information isfrequently sufficient.

But these characteristics can be tested by laboratories using:

CLSI Guideline for Interference

EP7: procedures for testing constant error due to interference


40/53

1. See CLSI EP7-A2

2. What to test:

Literature review

Always test hemolysis, lipemia, bilirubin

Tube additives

3. Concentrations to test: Interferent: highest compatible with life

Analyte: at medical decision levels

4. Volume of interferent


41/53

Interference Experiment: N=?Number of Measurements / Replicates:

at least several samples per interferent

at least duplicates per sample

EP7 lists a table of N as a function of bias/stm (EA,I/Stm ), with whichone can determine how many replicates are necessary to reach95% probability of observing a certain magnitude of error:

EA,I/Stm No. Replicates EA,I/Stm No. Replicates

0.8 41 1.5 121.0 26 1.6 10

1.1 22 1.8 81.2 18 2.0 71.3 16 2.5 61.4 14 3.0 3


42/53


Carryover Carryover: the discrete amount of reagent or analyte carried by the

measuring system from one test into subsequent test(s), thereby

erroneously affecting test results.

Reagent carryover among different measurement procedures on

multichannel automated analyzers is an evaluation that is usuallyconducted by measuring system manufacturers.

But this characteristic can be tested by laboratories using:

CLSI Guideline for Carryover

EP10: includes an assessment of sample carryover along with other

parameters.

NOTE: EP10 is intended to determine if a device has unacceptableperformance. It is recognized in the CAP Chemistry Checklist as anacceptable way to measure carryover.


43/53

Performance Characteristics:Reference Intervals

Reference Interval: interpretive information for laboratory test results thatis frequently provided as the central 95% interval of results for a group ofwell-defined reference individuals.

Laboratories can produce reference intervals in a variety of ways,

including testing procedures found in

CLSI Guideline for Reference Intervals or Decision Value

C28: procedures for establishing a reference interval or verifying thesuitability of a manufacturer-proposed reference interval


44/53

Transference of established reference intervals to an individual laboratory ora new method may be accomplished in a variety of ways:

1. Subjective assessment by a responsible individual; the Medical Director (sometimes called by divine judgment)

2. Donor testing

a. Verify with ~ 20 donor samples

b. Validate/Estimate using ~ 60 donor samplesc. Establish using ~ 120 donor samples

3. Calculation

use regression statistics from a comparison of methods study to calculate

reference limits for the new method (Y) that correspond to the referenceinterval limits of the former method (X).

Reference Interval Determination

Y = a + b X


45/53

CLSI Makes Life Easier with

StatisPro

In October 2010, CLSI released

StatisPro software:

Direct, faithful implementation ofCLSI Evaluation Protocol Guidelines

Study Advisor step-by-step help foreach study

Four steps to complete a study:

Definition, Data Input, Analysis, and

Signoff


46/53


47/53

StatisPro Study Design

Performance Claimto be Verified

Study Goal

Identifying

Information

Details of

the Study

Description of

Materials Used


48/53

StatisPro Data Entry

Copying and Pasting from any spreadsheet application or Windowsapplication with clipboard support is easy.


49/53


50/53

StatisPro Study Advisor


51/53

StatisPro Demonstration Demonstrate EP15 (method comparison) and EP06 (linearity).


52/53

User Experience with StatisPro StatisPro is useful when introducing new methods into your laboratory.

StatisPro is useful when performing six-month linearity or calibration verificationstudies.

By using StatisPro:

You are demonstrating compliance with regulatory and accreditation bodies.

You are ensuring that your laboratory delivers accurate results.


53/53

Thank YouQuestions?