Proficiency testing (PT) scheme for Shellfish · 2018-02-23 · Nita Patel –Food and Environmental Proficiency Testing Unit 10 November 2016. Contents • Public Health England’s

Proficiency testing (PT) scheme

for Shellfish

ISO 17043 what does compliance as a PT provider mean?

Nita Patel – Food and Environmental Proficiency Testing Unit10 November 2016

Contents• Public Health England’s mission for National Infection Service

• About Shellfish testing

• About proficiency testing

• Why laboratories do PT

• ISO 17043 and compliance

• Summary of the benefits

Proficiency testing scheme for Shellfish: ISO 17043 what does compliance as a PT provider mean?

Mission – National Infection Service (NIS)


The mission and core purpose of PHE for NIS is to protect the population of

England from infectious disease. It does this by:

a) Detecting outbreaks and monitoring communicable disease;

b) Analysing data collected on communicable diseases and their causes,

providing outbreak detection tracking and national surveillance;

c) Researching and effecting interventions, both immediately (emergency

response to outbreaks and incidents as a Category 1 responder, defined

under the 2004 Civil Contingencies Act) and through longer-term prevention;

d) Providing expert scientific advice and guidance to the public, government

and international agencies

IntroductionThe consumption of sewage polluted bivalve molluscan shellfish (BMS) may

cause illness in the consumer through ingestion of faecally borne pathogens

(such as Salmonella, norovirus and hepatitis A virus)

The risks of exposure to these infectious agents are compounded by the

traditional consumption of raw, or only lightly cooked BMS. Enteric bacteria, such

as Escherichia coli are used as indicator organisms to assess the sanitary

quality of shellfish and their growing waters and to predict the risk of exposure to

enteric pathogenic microorganisms


Public Health concernsShellfish areas often impacted by sewage

Bivalves can filter 10-24 litres per hour

Pathogens from sewage are concentrated (~100x)

Shellfish are eaten raw or lightly cooked

Outbreaks of disease can occur on large scale


European Union Regulation 854/2004

In the European Union (EU), the criteria

for laying down the microbiological

standards for BMS are set out in

Regulation (EC) 854/2004 and Regulation

(EC) 2073/2005. This stipulates the

conditions for the production and placing

on the market of live BMS

Across the EU E. coli in shellfish flesh is

used as an indicator of faecal

contamination of BMS

The EU reference method for the

determination of E. coli in BMS is ISO

16649‐3*


*ISO 16649‐3. ‘Microbiology of food and animal feeding stuffs ‐ Horizontal method for the enumeration of β‐glucuronidase positive

Escherichia coli Part 3: Most probable number technique using 5‐bromo‐4‐chloro‐3‐indolyl‐β‐D glucuronide’

Roles and Responsibilities – England and

Wales


Food Standards Agency

• ‘Competent Authority’

•Makes final classification decisions

• Issues classification list

•Determines policy

Facts and Figures - UKScale of programme:

• 75 production areas

• 375 beds classified: oysters, mussels, cockles and various clams

• Approximately 90,000 results


Facts and Figures - UKScale of programme:

• 75 production areas

• 375 beds classified: oysters, mussels, cockles and various clams

• Approximately 90,000 results

There is a need for examination of samples using

approved method/s and accurate reporting of results

to ensure that public health occurrences

can be effectively managed


Food and Environmental Proficiency

Testing Unit (FEPTU)


External quality assessment (EQA) /

Proficiency Testing (PT) explained

• Samples of known but undisclosed content are introduced into a

laboratory’s routine testing procedure


Freeze-dried LENTICULE® discs

Proficiency Testing


PARTICIPANTSORGANISING LABORATORY

Prepare samples Examine samples

Analyse results Report results

Prepare report Evaluate

Overview - Why laboratories do PT• To demonstrate competence as part of your accreditation requirement –

ISO/IEC 17025:2005 - General requirements for the competence of testing

and calibration laboratories

• Helps to provide assurance of the results obtained provided they are treated

and processed the same as other samples

• Helps improve laboratory processes and understanding of

regulation/legislation

• To remain up to date with new and emerging organisms - educational

• To challenge processes/media/training with difficult or atypical organisms

• Inter-laboratory comparison of performance

• To support work tendered for as an accredited laboratory

• Because you enjoy the challenge that participating in PT brings!


Impact of incorrect results – food microbiology

14 Proficiency testing scheme for Shellfish: ISO 17043 what does compliance as a PT provider mean?

E. coli levels too high or false positive

Salmonella result:

Incorrect classification

Unnecessary disruption

Restriction or ban on the export of the

product

Loss of confidence in harvesting area

Financial cost

E.coli levels too low or false negative

Salmonella result:

Unsafe product released for sale

People become ill/die

Negative publicity

Loss of reputation

Financial cost

Laboratory mistakesIncorrect results may arise from:

• Sample handling errors (mixed up, incorrectly labelled etc.)

• Calculation errors and misreading the most probable numbers (MPN) tables

• Cross-contamination

• Inadequate staff training

• Equipment and culture media failures

• Reporting errors


Shellfish SchemeThis scheme provides external quality assessment samples for laboratories that

examine raw bivalve molluscs from harvesting sites in accordance with Regulation

(EC) No. 854/2004 and from the production chain between harvest and

consumption, in accordance with Regulation (EC) 2073/2005 as amended.

The Scheme is organised in collaboration with the Centre for Environment,

Fisheries and Aquaculture Science (Cefas), Weymouth, United Kingdom (UK).

The scheme focuses on raising awareness:

• of interpreting tube combination results and the associated MPN value per 100g

for Escherichia coli for classification of harvest beds

• of problems with Salmonella detection (included in the scheme for

end-product testing, not for classification purposes)

• of updates in the ISO method


Shellfish Scheme•Three distributions a year with two samples in each (2 labs in Peru)

•A total of 63 participants in 30 different countries

•Mandatory participation for the National Reference Laboratories (NRL) in each

member state reporting to the European Union Reference Laboratory (EURL -

Cefas)

•Permission from the NRLs is obtained to release performance date to the

EURL


Most Probable Numbers (MPN)The method used to determine the MPN has greater inherent variability than

colony count methods

However, it is sometimes required for the detection and estimation of levels

when low levels of micro-organisms are expected, especially when the micro-

organisms may be stressed (e.g. as a result of processing, freezing etc.)

The MPN method is stipulated in European Union criteria for the

microbiological examination of certain products such as dairy products and live

bivalve molluscs and other shellfish

Most results are reported as the number of positive tubes out of 5 at 3/4

dilutions and the corresponding most probable number (MPN)

http://standards.iso.org/iso/7218/ (electronic calculator)


http://standards.iso.org/iso/7218/

How do you measure performance?


Statistics used • Participating in PT is the only way a laboratory can obtain an ongoing

independent assessment of its performance

• To get the maximum benefit from participating in PT it is helpful to know

how the statistics that are used are applied

• FEPTU uses statistics to determine whether a participants results are within

an expected range


Statistics used • The expected range comes from the distribution (average and variability) of

all the participants’ results – the median value

• ISO 13528:2005 - Statistical methods for use in proficiency testing by inter-

laboratory comparisons is followed

• Certain criteria must be followed when analysing participants’ results


Setting the criteria• The statistical design for PT schemes for microbiology are influenced by:

Homogeneity level in a sample (distribution of organisms)

Whether the test is for the detection or the enumeration of a target organism

• Number of participants in a scheme

• Target level of organism

• Method of analysis

For example, results for an examination using colony count methods will be

analysed in a different way to those where determination of the most probable

number (MPN) is required


Scheme Reports• Summary of all participants’ results

• Assessment of every result using a scoring system

• Assessment of each participant’s performance over time

• Scatter graphs to help participants compare performance with other

laboratories

• Advice and guidance


Summary of all participants’ results






laboratories



PHE scores appliedE. coli MPN result is determined from the median result reported by

participants’± 2.68 standard deviations (SD). The SD values used to

determine the expected range take into account the expected inherent

variability of the three by five tube MPN method, which on a log10 scale has a

value of 0.26

Results are defined as follows:

A) Expected range = median ± 2.68 SD

B) Outlying result 1 = median ± >2.68 ± ≤4 SD

C) Outlying result 2 = median ± >4 SD

Marks are deducted if tube combination is inconsistent with MPN reported





laboratories




Scatter graph of results


Scatter graph of results





laboratories




On going performance over time


• Is designed to alert participants with on-going problems

• Is only useful if a laboratory takes more than one distribution a year in a

scheme

• Allows laboratories to identify errors in the system

• Is an effective tool for on-going monitoring of quality control processes

• Outlying results are allocated a low score

• On single occasion reporting an outlying result participants must not be

alarmed but they should access the reasons why

• Laboratories that achieve less than 70% of the maximum possible score are

likely to be experiencing significant problems

On-going performance data - table


On-going performance data - graph


Trend analysisTrend analysis tools are available on our web-site, these help participants to plot their own

results to identify performance trends against the participants’ median. See example

below:

Trend analysis tools are available under each scheme heading:

https://www.gov.uk/government/collections/external-quality-assessment-eqa-and-proficiency-testing-pt-

for-food-water-and-environmental-microbiology


ISO 17043 compliance


ISO 17043 versus

ISO 17025


ISO 17043 versus

ISO 17025


ISO 17043 versus

ISO 17025


ISO 17043 introduction


ISO 17043 – some issues• It is a very generic document written mainly by chemists

• The language used does not always make sense for microbiological criteria

• For microbiology we need to take into account stability, variability and

homogeneity, behaviour of the organisms etc.

• To address specific requirements for microbiology ISO/TS 22117:2010 was

published (Microbiology of food and animal feeding stuffs - Specific

requirements and guidance for proficiency testing by interlaboratory

comparison)


ISO 17043 – critical aspects


ISO 17043 – Planning (4.4.1)‘The proficiency testing provider shall not subcontract the planning of the

proficiency testing scheme’

‘The proficiency testing provider shall document a plan before commencement

of the proficiency testing scheme’


ISO 17043 – Homogeneity and stability (4.4.3)

‘Criteria for suitable homogeneity and stability shall be established and shall be

based on the effect that inhomogeneity and instability will have on the evaluation of

the participants' performance’.

Homogeneity – organisms are distributed in a sample

Stability – the levels of the organisms are similar over a defined period of time

For the Shellfish Scheme difficult to prove homogeneity due the inherent variability of

the MPN method. Method for assessing participants’ results assumes that the

sample is well-mixed prior to testing and therefore allows for this variability (wider

standard deviation used)


ISO 17043 – Homogeneity and stability (4.4.3)

FEPTU Quality Control: To demonstrate stability of the sample, a minimum of ten

LENTICULE® discs, selected randomly from a batch, are examined throughout

the distribution period for E. coli enumeration and for Salmonella detection


ISO 17043 – Statistical design (4.4.4)

‘Statistical designs shall be developed to meet the objectives of the scheme,

based on the nature of the data (quantitative or qualitative, including ordinal

and categorical), statistical assumptions, the nature of errors, and the expected

number of results’

‘The proficiency testing provider shall document the statistical design and

data analysis methods to be used to identify the assigned value and

evaluate participant results, and shall provide a description of the reasons for

their selection and assumptions upon which they are based’

‘In designing a statistical analysis, the proficiency testing provider shall give

careful consideration to the following: accuracy, number of participants, identify

and how to handle outliers, scoring criteria etc.’


ISO 17043 – Statistical design (4.4.4)


PHE statistician is consulted on all unusual results reported by participants

ISO 17043 – Packaging, labelling and

distribution of PT items (4.6.3)‘The proficiency testing provider shall control packaging and labelling

processes to the extent necessary to ensure conformity with relevant national,

regional, or international safety and transport requirements’

‘The proficiency testing provider shall specify relevant environmental

conditions for the transport of proficiency test items. Where relevant, the

proficiency testing provider shall monitor the pertinent environmental

conditions of the proficiency test item during transport and assess the impact

of environmental influences on the proficiency test item’


http://www.un3373.com/un3373-packaging/set-p650-un3373/8712963012106/

http://www.un3373.com/un3373-packaging/set-p650-un3373/8712963012106/

ISO 17043 – Packaging, labelling and

distribution of PT items (4.6.3)This ensures that everybody receives samples that are the same across the

world

We therefore examine samples as part of our QC process that have gone

through the shipment process to identify impact on the sample contents and

levels


ISO 17043 – Evaluation of performance (4.7.2)

‘The proficiency testing provider shall use valid methods of evaluation which

meet the purpose of the proficiency testing scheme. The methods shall be

documented and include a description of the basis for the evaluation. The

evaluation of performance shall not be subcontracted’

‘Where appropriate for the purpose of the proficiency testing scheme, the

proficiency testing provider shall provide expert commentary on the

performance of participants’


ISO 17043 – Evaluation of performance (4.7.2)


ISO 17043 – Reports (4.8.1)‘Proficiency test reports shall be clear and comprehensive and include data

covering the results of all participants, together with an indication of the

performance of individual participants. The authorization of the final report shall

not be subcontracted’

‘Reports shall be made available to participants within planned timescales - this

allows for early investigation of possible error’


ISO 17043 – Reports (4.8.1)


ISO 17043 – Reports (4.8.1)


ISO 17043 – Reports (4.8.1)


ISO 17043 – Reports (4.8.1)


ISO 17043 – Reports (4.8.1)


ISO 17043 – Reports (4.8.1)


ISO 17043 – Confidentiality (4.10)‘The identity of participants in a proficiency testing scheme shall be confidential

and known only to persons involved in the operation of the proficiency testing

scheme, unless the participant waives confidentiality’


Why participate? PHE PT samples are designed to challenge your testing procedures therefore

will include challenging organisms – so beware

We extensively test the samples using ISO methods – so your results should

align with our results including confirmatory test results

Process PT samples the same as other routine samples. Otherwise nothing will

be learnt about your quality system

We are not here to trick you but to:

• raise awareness of the limitation/s of your procedure or method

• provide an insight into staffs’ knowledge and experience

• endorse the requirement to carry out confirmatory tests

• encourage the use of approved methods

• give you an opportunity to examine samples containing organisms less frequently

encountered that are of public health concern


Summary of the benefits to you

Raises awareness of new and emerging organisms of public health concern

Provides an opportunity to improve staffs’ knowledge and experience with organisms

not frequently encountered

Better understanding of your performance in relation to the method/s you use

Greater understanding of the limitation/s of your method especially with

atypical/emerging organisms

Highlights impact on results when approved methods are not followed

Highlights impact on results with unusual, atypical strains or less commonly

encountered organisms

Gives you an independent assessment of your laboratory’s overall performance

Allows you to identify gaps in your process where quality improvements are required


Other schemes available are


Water PT

• Legionella Isolation Scheme

Including one specific for molecular

• Drinking Water Scheme

• Recreational and Surface Water Scheme

Bathing beach (marine)

Swimming pool

River, lakes and streams

• Bottle and Mineral Water Scheme

• Endoscope Rinse Water Scheme

• Dialysis Water Scheme

• Hospital Tap Water Scheme

Food PT

• Standard Scheme

• European Food Microbiology Legislation Scheme

• Non-Pathogen Scheme

• Shellfish Scheme

• Pathogenic Vibrio Scheme

• Staphylococcus aureusEnterotoxin Scheme

• Environmental Swab Scheme

Hygiene indicator

Pathogen detection

• Shiga toxin E. coli molecular

Acknowledgements


• PHE Steering Group members

• eQAD clinical team

• Andy Younger (Cefas)

• Louise Stockley and Cefas team

• Angela Appea

• Richard Borrill

• Kermin Daruwalla

• Joanna Donn

• Carmen Gomes

• Thomas Harper

• Salah Himedan

• Zak Prior

• Manchari Rajkumar

• Aneta Stranc

• Lili Tsegaye

• Vanessa Waite

Thank you for listening


Documents

Proficiency testing (PT) scheme for Shellfish · 2018-02-23 · Nita Patel –Food and Environmental Proficiency Testing Unit 10 November 2016. Contents • Public Health England’s