How to use the best of your QA data

Dubrovnik Course in Zagreb 2015

Gunnar Nordin

The topics

Evaluate your QC results! Be interested. Discuss them at the lab and with friends.

Why are my, and not others, results out of limits?

“The EQA material is not commutable”“The assigned target value is not correct”“The EQA material is not stable”

There are too many ways to express “accuracy”

Standard for the evaluation of a participant in an EQA scheme

Standard for assessment of IVD procedures

A flowshart for deviating EQA-results

A flowshart for deviating EQA-results

A flowshart for deviating EQA-results

EQA material should be commutable

Zhang et al, Commutability of Possible External Quality AssessmentMaterials for Cardiac Troponin Measurement, 2014

EQA material should be stable

ISO 13528:2015

41

42

43

44

45

46

day 1 day 3

resu

lt, g

/L

test day

Stability tested by repeated measurments at one site at the day of distribution and at the last day for the round.

� Standard error

<0.3σPT

The target value for a EQA material should be a true value

1. Target value known by formulation

2. Target value from by reference measurement procedure

3. Target value by ’expert laboratory methods’

4. Target value by consensus mean

The consensus value is often not a true value

Consensus value

”Mean of method mean” (MOMM)

Mean of the major and homogenous method group mean values

Deviating method groups are excluded

Small method groups are excluded

14

”Mean of method mean” (MOMM)

50 55 60 65 70 75 80 85 90 95 100 105 110 115 120

P-Creatin ine, µmol/L

Num

ber

of o

bser

vatio

ns

5M AD

5% of observations

Do you think MOMM is a more realiable target value than the consensus mean?

Yes ?

No ?

15

”Mean of method mean” (MOMM)

Uncertainty is clearly defined

What is accuracy?

”more” or ”less” are results on an ordinal scale

Is it possible to measure accuracy?

Yes

No

One type of measure of accuracy

Fraction of results within quality specifications

A pragmatic approach for POCT HbA1c in Sweden:

Max 2 of 10 results outside the limits

Accuracy and bias reported to participants in Equalis HbA1c scheme

Accuracy as ”fraction of values within” denoted ”P30” for accuracy of eGFR

Stevens, 2008

Equalis accuracy graphs

Accuracy described as ”total error”the simple model TEa = bias(%) + 1.65 x CV

0

2

4

6

8

10

12

-20 -10 0 10 20

Rand

om e

rror

(CV%

)

bias (%)

Total error; 10, 15 and 20% TEa

Contour NEXT USB (n=6)

Contour XT (n=5)

MedCore Care (n=21)

fig 2

Accuracy described on the ”six sigma” scale

0

2

4

6

8

10

12

-20 -10 0 10 20

Rand

om e

rror

(CV%

)

bias (%)

Total error 15 % and sigma scale

Contour NEXT USB (n=6)

Contour XT (n=5)

MedCore Care (n=21)

sigma 2

sigma 3

sigma 4

Accuracy (or total error?) with the root mean square concept (RiliBÄK)

Accuracy described as ”long term uncertainty measurement” (LTUM)

Matar et al, 2015

mean absolute relative deviation (MARD) – a new accuracy metric

Much used to describe performance of continous glucose measurment (CGM)

In case of no bias MARD ≈ 0,8 x CV

No additional informationcompared with “p15”

0%

2%

4%

6%

8%

10%

12%

14%

60% 65% 70% 75% 80% 85% 90% 95% 100%

MA

RD

p15

bias 0

bias 0.5

bias 1

Alternative expressions for accuracy

1. Something that can only be good or bad?

2. A fraction of results within performance specification?

3. The “total error”

4. The root mean square of bias and precision

5. The long term uncertainty measurement (LTUM)

6. The mean absolute relative deviation (MARD)

Thank you for listening

30

Do you think MOMM is a more realiable target value than the consensus mean?

A. Yes

B. No

YesNo

0%0%

Is it possible to measure accuracy?

A. Yes

B. No

YesNo

50%50%

Which expression for accuracy do you prefer?

A. Something that can only be good or bad?

B. A fraction of results within performance specification?

C. The “total error”

D. The root mean square of bias and precision

E. The long term uncertainty measurement (LTUM)

F. The mean absolute relative deviation (MARD)

G. The question is wrong, beacuse these terms do cover different concepts and should be given different names

A. B. C. D. E. F. G.

14% 14% 14% 14%14%14%14%

How should target values be assigned to commutable EQA materials?

A. Reference method values

B. Value transfer from certified reference materials

C. Consensus mean from all participants

D. Mean of method mean (MOMM)

E. Method group mean values should be used

A. B. C. D. E.

20% 20% 20%20%20%

Some words on results that are not quantitative

36

1. Semi quantitative test

2. Classification test

3. Binary test

4. Test with results ”yes/no”

5. Test with results on an ordinal scale

37

What is a ”qualitative test” ?

38

Performance characterstics for pregnancy tests

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 10 100 1 000 10 000 100 000

frac

tion

of p

ositi

v res

ults

hCG(IU/L)

Performance in EQA for two pregnancy tests

Test A 20 IU/L

Test B 40 IU/L

c5

c95

c50

Unreliability zone

39

How to draw the line….

Hyltoft Petersen, P., et al. (2008) Scand J Clin Lab Invest 68(4): 298-311.

40

How to draw the line….

-3

-2

-1

0

1

2

3

0,1 1 10 100 1000 10000 100000

Rank

it (z

)

hCG(IU/L)

Performance in EQA for two pregnancy tests

Test A 20 IU/L

Test B 40 IU/L

41

Proposal 2: c5, c50 an c95

For ordinal binary test with a quantitative back ground scale assays should be characterized

with the three quantities:

“c5”, ”c50” and ”c95”.

The manufacturer should declare the c50 value for the assay, and describe the metrological traceability

<c5 c5 – c95 >c95

False pos - True pos

True neg - False neg

42

Internal quality assessment for ordinal tests

<c5 c5 – c95 >c95

False pos True pos ? True pos

True neg True neg ? False neg

43

External quality assessment for ordnal tests

44

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 000 10 000 100 000 1 000 000 10 000 000 100 000 000

And

el p

ositi

va re

sulta

t

Koncentration grupp A streptokocker (CFU/mL)

QuickVue Dipstick Strep ATest A

Empirical, ”state of the art” limits for

performance

45

How do we report EQA results on ordinal scale

46

Tests with deviating performance can be revealed

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 000 10 000 100 000 1 000 000 10 000 000 100 000 000

And

el p

ositi

va re

sulta

t

Koncentration grupp A streptokocker (CFU/mL)

Analyz Strep ATest B

Take a home message

All QC results must be evaluated and discussed.

If deviating QC results are not related the performance of the method, it might be due to properties of the material used. Some examples are:

Non commutable materialIncorrect target value for the materialUnstable material

The measure of method performance varies. Unfortunately there are today many different measures for “accuracy”. We need to agree on a common use of the concept accuracy.