Basic Statistics for Laboratory Proficiency Testing

PRAKIT BOONPORNPRASERT DVM, M.A.

Virology Laboratory

National Institute of Animal Health

OUTLINE

• 1) Type of PTA Proficiency Testing

Programmed

• 2) Composition of the proficiency

panel and statistical design

• 3) Data Preparation

• 4) summary statistics and example

Continue

• 5) Robust Z-scores and Outliers

• 6) Graphical Displays

• 7) Robust Z-scores interpretation

• 8) Youden diagram interpretation

Type of PTA Proficiency Testing Programme

1. Testing Interlaboratory

Comparisons

2. Calibration Interlaboratory

Comparisons

Testing Interlaboratory Comparisons

* testing interlaboratory comparisons ,

which involve concurrent testing of

samples by two or more laboratories

and calculation of consensus values from all participants’ results.

Testing Interlaboratory Comparisons

Calibration Interlaboratory Comparisons

• calibration interlaboratory

comparisons in which one test item

is distributed sequentially among two

or more participating laboratories

and each laboratory’s results are compared to reference values.

Calibration Interlaboratory Comparisons

Assign Value

• consensus value

- an assigned value obtained from the

results submitted by participants (e.g.

for most testing programs the median

is used as the assigned value)

Assign Value

• reference value

- an assigned value which is provided by a Reference Laboratory

Consensus Value

• The advantages of participant consensus include

low cost, because the assigned value does not re-quire additional analytical work.

• No one member or group is accorded higher

status.

• Calculation of the value is usually straightforward.

Composition of the proficiency panel

Statistical design

• Negative sample

• Strong positive Sample

• Weak positive Sample

Consensus Value

• The principal disadvantages of participant

consensus values are, first, that they are not

independent of the participant results.

• if the majority of results are biased,

participants whose results are unbiased may

unfairly receive extreme z-scores.

Composition of the proficiency panel

Statistical design

• Between-laboratories variation

• Within-laboratory variation

• participants must perform the same

testing more than once (e.g. Twice or

paired samples).

Example of AI Type A PT Panel

Type of paired sample

1. Uniform pairs

identical blind duplicates

(where the results are expected

to be the same)

Type of paired sample

2. Split pairs

slightly different blind duplicates

(where the results should be slightly

different. )

Analysis of paired sample

• The statistical analysis of the

results is the same for both

types of pairs (uniform or split),

but the interpretation is slightly different .

Data Preparation

• Prior to commencing the statistical

analysis, a number of steps are

undertaken to ensure that the data

collected is accurate and appropriate

for analysis.

Continue

• It is during this checking phase

that gross errors and potential

problems with the data in general may

be identified.

• In some cases the results are then transformed.

for example

• For microbiological count data the

statistical analysis is usually carried

out on the log 10 of the results,

rather than the raw counts.

• For HI test the statistical is usually

carried out the Log titer i.e. dilution 1/32 = 25

Quantitative Data

Qualitative Data

The Normal Distribution

Robust statistics

• Robust statistics are based on the

assumption that the data are a sample

from an essentially normal distribution

contaminated with heavy tails and a small proportion of outliers.

Robust statistics

• Insensitive to the presence of outliers

and heavy tails to avoid undue

influence from poor results, and this is

why the median or a robust mean is

valuable.

Robust statistics

• The median, however, is more robust

when the frequency distribution is strongly skewed.

summary statistics • No. of results

• Median

• Normalised IQR

• Robust CV

• Minimum

• Maximum

• Range

summary statistics

summary statistics

summary statistics

summary statistics

summary statistics

• The no. of results is simply the

total number of results received

for a particular test/sample, and is denoted by N.

Example

• จากการทดสอบ AI Type A โดยวิธี Realtime

PCR ได้ค่า Ct value ของห้องปฏิบัติการที่เข้าร่วมทดสอบได้ผลดังนี ้36.97, 35.88, 36.04 ,35.46 ,35.74, 36.22, 36.64 ,38.20, 37.20 ,38.30

• ค่า N =

summary statistics

• The median is the middle value of

the group, i.e. half of the results

are higher than it and half are

lower. If N is an odd number the

median is the single central value,

i.e. X[(N+1)/2].

Continue

• If N is even, the median is the average

of the two central values, i.e. (X[N/2]

+X[(N/2)+1])/2.

• For example if N is 9 the median is

the 5th sorted value and if N is 10 the

median is the average of the 5th and 6th values.

Example

• จากการทดสอบ AI Type A โดยวิธี Realtime

PCR ได้ค่า Ct value ของห้องปฏิบัติการที่เข้าร่วมทดสอบได้ผลดังนี ้36.97, 35.88, 36.04 ,35.46 ,35.74, 36.22, 36.64 ,38.20, 37.20 ,38.30

• ค่า Median =

summary statistics • The normalised IQR is a measure of

the variability of the results.

• It is equal to the interquartile

range (IQR) multiplied by a factor

†(0.7413), which makes it

comparable to a standard deviation.

Example

• จากการทดสอบ AI Type A โดยวิธี Realtime

PCR ได้ค่า Ct value ของห้องปฏิบัติการที่เข้าร่วมทดสอบได้ผลดังนี ้36.97, 35.88, 36.04 ,35.46 ,35.74, 36.22, 36.64 ,38.20, 37.20 ,38.30

• ค่า Interquatile Range =

• ค่า The normalised IQR =

Continue

• In most cases Q1 and Q3 are obtained

by interpolating between the data

values.

• The IQR = Q3 – Q1 Where

• Q3=3*(N+1)/4

• Q1=(N+1)/4

The IQR = (Q3 – Q1)=(37.2 - 35.88)

The normalised IQR = IQR × 0.7413

summary statistics • The robust CV is a coefficient of

variation (which allows for the

variability in different samples/tests to

be compared) and is equal to the

normalised IQR divided by the median,

expressed as a percentage - i.e. robust CV = 100 × normalised IQR ÷ median.

Example

• จากการทดสอบ AI Type A โดยวิธี Realtime

PCR ได้ค่า Ct value ของห้องปฏิบัติการที่เข้าร่วมทดสอบได้ผลดังนี ้36.97, 35.88, 36.04 ,35.46 ,35.74, 36.22, 36.64 ,38.20, 37.20 ,38.30

• ค่า The robust CV =

summary statistics • The minimum is the lowest value (i.e.

X[1]).

• The maximum is the highest value

(X[N]).

• The range is the difference between them (X[N]–X[1]).

Example

• จากการทดสอบ AI Type A โดยวิธี Realtime

PCR ได้ค่า Ct value ของห้องปฏิบัติการที่เข้าร่วมทดสอบได้ผล ดังนี ้36.97, 35.88, 36.04 ,35.46 ,35.74, 36.22, 36.64 ,38.20, 37.20 ,38.30

• คา่ The Minimum =

• คา่ The Maximum =

• คา่ Range =

Robust Z-scores and Outliers

• Z-scores based on robust summary

statistics (the median and normalised

IQR).

• Where pairs of results have been

obtained, two z-scores are calculated

a between-laboratories z-score and a within-laboratory z-score.

Robust Z-scores

• Suppose the pair of results are from

two samples called A and B.

• The median and normalised IQR of all

the sample A results are denoted by

median(A) and normIQR(A), respectively (similarly for sample B).

Uniform pair sample

Z-scores • The z-score is a measure of the

deviation of the result from the

assigned value.

• Z score= (X-Xbar)/SD

• X= result reported by participant

• Xbar= assigned value

• SD= standard deviation for proficiency

assessment

Robust Z-scores A simple robust z-score (denoted by Z) for

a laboratory’s sample A result would then be:

The standardised sum (denoted by S) and standardised difference (D)

• The standardised sum (denoted by S)

and standardised difference (D) for the

pair of results are:

Between-laboratories z-score

• The between-laboratories z-score

(denoted by ZB) is then calculated as

the robust z-score for S

Within-laboratory z-score

• The within-laboratory z-score (ZW) is

the robust z-score for D

Outlier

• An outlier is defined as any

result/pair of results with an

absolute z-score greater than or

equal to three, i.e. Z ≥ 3.0 or

Z ≤ -3.0.

• Outliers are identified in the table by a marker (§) beside the z-score.

Outlier

• This outlier criteria, | Z | ≥ 3.0, has

a confidence level of about 99%

(related to the normal distribution)

• A confidence level of approximately

95% z-score in this region

(i.e. 2.0 < | Z | < 3.0)

Outlier of ZB uniform and split pairs

• A positive between-laboratories outlier

(i.e. ZB ≥ 3.0) indicates that both

results for that pair are too high.

• A negative between-laboratories

outlier (i.e. ZB ≤-3.0) indicates that the results are too low.

Outlier of ZW uniform and split pairs

• For uniform pairs, where the results

are on identical samples, a within-

laboratory outlier of either sign (i.e.

| ZW| ≥ 3.0) indicates that the

difference between the results is too

large.

Continue

• For split pairs, where the analyte is

at different levels in the two

samples, a positive within-laboratory

outlier (i.e. ZW ≥ 3.0) indicates that

the difference between the two results is too large.

Continue

• For split pairs, a negative within-

laboratory outlier (i.e. ZW ≤ -3.0)

indicates that the difference is too

small or in the ‘opposite direction’ to the medians.

Z-score of single results

• A single result on one sample (X),

a simple robust z-score is calculated

as

• Z = {X-median(X)}/normIQR(X)

Continue

• The sign of the z-score indicates

whether the result is too high (positive

z-score) or too low (negative z-

score).

• But whether this is due to

between-laboratories or within-

laboratory variation, or both, is unknown.

Graphical Displays

• The z-score bar-chart

• The Youden diagram

• Very useful to participants - especially

those participants with outliers

because they can see how their results

differ from those submitted by other

laboratories.

The z-score bar-chart

The z-score bar-chart

• The advantages of these charts are that

each laboratory is identified and the

outliers are clearly indicated.

• They are not graphs of the actual results.

The Youden diagram

• The advantages of these diagrams

are that they are plots of the

actual data.

• So the laboratories with results

outside the ellipse can see how their results differ from the others.

The Youden diagram

As a guide to the interpretation of the Youden diagrams:

• laboratories with significant systematic

error components (i.e. between-

laboratories variation) will be outside the

ellipse in either the upper right hand

quadrant (as formed by the median lines)

or the lower left hand quadrant, i.e.

inordinately high or low results for both samples;

As a guide to the interpretation of the Youden diagrams:

• laboratories with random error

components (i.e. within-laboratory

variation) significantly greater than other

participants will be outside the ellipse

and (usually) in either the upper left or

lower right quadrants, i.e. an inordinately

high result for one sample and low for the other.

The Youden diagram

• It is important to note however that

Youden diagrams are an illustration of

the data only, and are not used to

assess the results (this is done by the z-scores).

Summary table

Between-laboratories z-score (ZB)

Within-laboratories z-score (ZW)

Youden diagram of Type A

•Statistic Significant but Diagnostics not significant

Example of AI Type A PT Panel

ปกติ

Between Lab Variation (ZB)

Within Lab Variation (ZW)

Youden plot

ระดบัสูง,เท่ากนั

Between Lab Variation (ZB)

Within Lab Variation (ZW)

Youden plot

ระดบัต า่,เท่ากนั

Between Lab Variation (ZB)

Within Lab Variation (ZW)

Youden plot

ระดบัสูง,ไม่เท่ากนั

Between Lab Variation (ZB)

Within Lab Variation (ZW)

Youden plot

ระดบัต า่,ไม่เท่ากนั

Between Lab Variation (ZB)

Within Lab Variation (ZW)

Youden plot

THANK YOU

• 1. Paul Selleck (AAHL) for HI - PT testing

• 2. Brian Mehan (AAHL) for PCR - PT testing

• 3. Gemma Carlile (AAHL) for Statistics for PT

panel

• 4. Dr. Vimol Jirathanawat : Director of NIAH

• 5. Dr. Sujira Pachariyanon : Head of Virology Laboratory