Planning appropriate PT activity for a calibration laboratory – dispelling the myths

Eddie Tarnow

NLA T&M 2014

Introduction

“PT activity shouldn’t be implemented to satisfy the accreditor – it should form an

integral part of assuring the quality of measurement results to mitigate risk.”

It follows therefore that it should be a properly planned activity with a specific

objective.

Planning PT activity • Define the objective

• Appoint a Pilot Laboratory

• Define the parameter to be measured

• Define the measurement ranges or type of equipment to be covered

• Determine the Target Uncertainty of Measurement

• Define the Reference Value & how it will be determined

• Identify suitable participants

• Define analysis of results and performance evaluation

• Report the results

Objective of PT activity • To assure the quality of the results

• To validate a new or revised method

• To prove competence of lab personnel

• To prove validity of results at the extremes of an extended range

• To benchmark your lab capability against your peers

• To verify a new instrument’s performance

• Etc.

The parameter to be measured • Could be a specific type of instrument

• Demonstrate “proficiency” in the calibration of contact Tachometers

• Demonstrate “proficiency” in the calibration of an External Micrometer in the range 100 mm to 200 mm

ITEM TYPE OF INSTRUMENT RANGE OR MEASURED

QUANTITY CMC (±)

3.15 External Micrometers Up to 100 mm 2,0 µm

Above 100 mm to 200 mm 2,0 µm

Above 200 mm to 300 mm 3,0 µm

The parameter to be measured (2) • Could be a measurement range or parameter

• Demonstrate “proficiency” in measuring to within CMC in the range 10 V to 100 V

ITEM MEASURED QUANTITY RANGE CMC (±)

1 dc Voltage 1,018 V and 10 V 1•10-6•U

0 V to 10 V 1•10-6•U + 0,3 µV

10 V to 100 V 3•10-6•U

100 V to 1 kV 5•10-6•U

Adequately covering the range • Impossible to participate in PT activity covering all

measurement parameters and ranges • Group parameters and ranges using the same method

• Often useful to select points at range extremities where measurements can be more complex

• If several methods are used to cover a range, PT activity should be carefully selected to cover each method

• Impossible to participate in PT activity covering all types of instruments

• Proficiency on one instrument not proficiency on all

• 3,5 digit DMM ǂ 8,5 digit DMM (same method????)

Target uncertainty of measurement

• Currently the biggest problem with calibration lab PT activity

• Depends heavily on objective:-

• Instrument based proficiency • Target Uncertainty of Measurement (TUM) should be as

the minimum the accuracy spec of the artefact instrument

• This will adequately demonstrate a cal lab’s proficiency to calibrate such an instrument to within the manufacturer’s accuracy spec – common customer requirement

• Instrument unlikely to perform much better the spec

Target uncertainty of measurement (2)

• Parameter and range based proficiency • TUM should preferably be smaller than the CMC of the

best participant

• Usually requires a “special” artefact which has been very well characterised.

• The PT activity TUM is a critical consideration since it has a significant influence on the selection of an appropriate artefact and the cost of its characterisation.

Determination of the Reference Value

• Contrary to popular belief this must be defined at the beginning of the PT activity but may be determined at the end of the PT activity

• There must be agreement up-front amongst all participants as to what will constitute the Reference Value (Defining)

Determination of the Ref Value (2)

• Several methods of determining the Reference Value

• Determined from the calibrated value obtained from a higher order laboratory.

• Determined as the manufacturer’s accuracy specification as proven through appropriate calibrations

• Determined statistically from the results of all the participants

Determination of the Ref Value (3)

0,049790

0,049792

0,049794

0,049796

0,049798

0,049800

0,049802

0,049804

0,049806

0,049808

0,049810

NLA1 NLA2 8r56 5ht9 8r56

Measu

red

Vo

ltd

rop

(V

)

Participant Codes

Deviation results for 5 A dc

Identifying suitable participants

• Heavily dependent on several factors, especially the Target Uncertainty of Measurement

• For PT activity where the Reference Value is either a calibrated value, or instrument accuracy specification

• All participants must have a larger TUM than the Reference Value Uncertainty

• All participants must be able to perform measurements at the exact measurement points

• There should be at least three participants (Reference Lab plus two participants)

Identifying suitable participants (2)

• For PT activity where the Reference Value is determined statistically

• All participants should have similar reported uncertainties unless a weighted mean is used

• All participants must have similar technical ability to perform the measurements (skills, equipment etc)

• All participants should preferably use the same or similar methods to perform the measurements to reduce the effect of bias

• The number of participants must be large enough to make the statistics valid

Identification/selection of a Pilot Lab

• The “Pilot Lab” is not necessarily the source of the PT activity Reference Value

• It is the laboratory tasked with the responsibility of planning, executing the PT activity and reporting the results in a formal report.

• This responsibility must be assigned at the beginning of the PT activity

• The Pilot Lab must have the necessary competence to take on this responsibility

• The Pilot Lab does not necessarily have to be a participant.

Drawing up the “Protocol”

• Sometimes referred to as the “Audit Instructions”

• Is generated before the PT activity is implemented

• Must describe the objective of the PT activity

• Circulated to all potential participants beforehand to provide an opportunity for them to determine if they are technically suitable participants

• A protocol must be in place even for “informal” PT activity and be available for Technical Assessor scrutiny

Reporting the results

• All PT activity must produce results which must be formally reported

• The report should contain the following as a minimum:

• An indication of the evaluated performance of each participant

• Minimum acceptance criteria

• Conclusions and recommendations

• A conclusion needs to be drawn at the end of every PT activity

• This should indicate whether the objectives set out in the beginning, were achieved

Identifying/implementing C/As

• Participants in PT activity have an obligation to respond appropriately to the results they obtain.

• Where results are unsatisfactory, a root cause into their cause must be conducted and appropriate corrective actions implemented to rectify the problem.

• The root cause analysis must include investigation into the potential impact erroneously reported results may have had for customers.

• Customers should then be immediately informed of any potential impact

Avoiding the pitfalls

• Unsatisfactory PT activity plans • Inadequate detail,

• Plans do not adequately cover entire accreditation scope

• Plans do not continuously describe a “rolling window”

• Lack of documented protocols, inadequate detail • Can result in different participants performing the

measurements differently

• Participants may use different, incompatible, methods

• The artefact may be setup differently by different participants (different filter settings, sampling rates, settling times etc.)

Avoiding the pitfalls (2)

• Poor selection of artefact or instrument • In most cases the artefact selected is not technically

capable of performing in line with the PT activity objective

• In most cases the artefact has not been adequately characterised prior to the PT activity for stability, robustness to travel/couriering etc.

• Unclear definition of the Reference Value • Often, no Reference Value is defined or determined

• Participants just calculate the En value between each other

• In many cases, En ≤ |1| merely indicates “equivalence” not necessarily “proficiency” (could both be equally wrong!)

Avoiding the pitfalls (3)

• Unclear definition of PT activity Target Uncertainty of Measurement (TUM)

• There is seldom a TUM defined before the start of the PT activity

• This often results in two participants having vastly different reported measurement uncertainties

• This provides no benefit to the participant with the smaller reported uncertainty

• In most cases, this is the main reason why the demonstration of proficiency to measure to within CMC, is unsuccessful.

Avoiding the pitfalls (4)

• Poor determination of the PT activity Reference Value Uncertainty

• There is a general lack of understanding regarding the UoM associated with PT activity

• The Reference Value UoM is NOT the smallest participant uncertainty but rather the Root Sum Square of the participant uncertainties

• For PT activity with a CMC proficiency objective, this must be carefully considered since the Reference Value UoM must be smaller than the participant with the smallest CMC

Avoiding the pitfalls (5)

• Bilateral comparison PT activity with incompatible measurement uncertainties

97,00

98,00

99,00

100,00

101,00

102,00

103,00

NLA1 7gv2 7gv2 7gv2 NLA2 5ht9

Devia

tio

ns f

rom

No

min

al (A

)

Participant Codes

Deviation results for 100 A dc

Avoiding the pitfalls (6)

• Lack of formal report • In most cases, the PT activity report presented to

Technical Assessors, is grossly inadequate

• Contains too little detail

• Does not comply with the minimum SANAS requirements

Summary

• PT activity has to be a planned activity with a specific objective

• Whilst it is obligatory to meet certain SANAS requirements, PT activity should be enthusiastically embraced to manage the risk of a lab reporting technically invalid results

• The cost – often used as an excuse of not participating, should be balanced against the costs which could incurred if the impact of reporting invalid results was severe