Guerrino Macori
National Reference Laboratory for Coagulase Positive Staphylococci including S.aureus – TorinoSC Controllo Alimenti e Igiene delle ProduzioniFood Control Laboratory and Hygiene of the Production
Ring Trial in Italy:Screening Method for Detection of staphylococcal enterotoxins in Cheese, Milk and Meat
25 - 27 May 2016
ANSES Laboratory for Food Safety, Maisons-Alfort, France
10th workshop of the NRLs for Coagulase Positive Staphylococci
NRL for
Coagulase Positive Staphylococci including
Staphylococcus aureus
10th workshop of the NRLs for Coagulase Positive Staphylococci
Outline
1. Introduction
2. Aim of the study
3. Material and Methods
4. Results
5. Conclusion
NRL for
Coagulase Positive
Staphylococci
including S.aureus
10th workshop of the NRLs for Coagulase Positive Staphylococci
Background
To protect consumer health and reduce the number of SE-contaminated foods
entering the world market, EC Regulation 2073/2005 on microbiological criteria
for foodstuffs includes guidelines for testing for SEs in raw or heat-treated milk
cheese when CPS levels exceed 10^5 CFU/g or CFU/ml. Samples are to be taken
during the manufacturing process when CPS levels are expected to be the highest.
NRL for
Coagulase Positive
Staphylococci
including S.aureus
10th workshop of the NRLs for Coagulase Positive Staphylococci
Background
According to EU Regulation 882/2004, the implementation of comparative tests
between official national laboratories is among the duties of the NRL.
Such interlaboratory ring trials are conducted to ensure high technical and analytical
proficiency in the national network of official laboratories that analyze samples and
to better coordinate analyses at the national level.
NRL for
Coagulase Positive
Staphylococci
including S.aureus
10th workshop of the NRLs for Coagulase Positive Staphylococci
Introduction
NRL for
Coagulase Positive
Staphylococci
including S.aureus
10th workshop of the NRLs for Coagulase Positive Staphylococci
Aim of the study
NRL for
Coagulase Positive
Staphylococci
including S.aureus
• Presentation of the results of an interlaboratory ring trial for the detection of SEs in dairy products. The study
involved 24 laboratories: 23 belonging to the network of Italian official food laboratories plus the EU-RL
• The purpose of the interlaboratory trial was to assess the reproducibility of the results obtained by the
official laboratories for detection of SEs according to the official European screening method.
• Cohen’s k index was used to assess the real agreement between laboratories. The trial was conducted to
determine the accuracy (sensitivity and specificity) of the results obtained by each laboratory according to
the International Organization for Standardization (ISO) 16140: 2003 definition.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Asti
Brescia
Campobasso
Caserta
Cosenza
Cuneo
Florence
Foggia
Lugo di Ravenna
Novara
Padua
Palermo
24 labsPerugia
Pesaro
Portici
Roma
Roma D.O. Latte
Salerno
Savona
Sassari
Siena
Teramo
Turin
EU-RL
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
12 milk
samples
12 cheese
samples
Codes L01 through L24 and was randomly assigned to each laboratory, and all information about the
participating laboratories and the results of analytical tests was processed in a blinded fashion.
24 samples
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Positive samples contaminated with Staphylococcal
Enterotoxin type D solution
Positive and Negative samples tested by NRL
before the shipment
Sending samples to labs
Sample analyses
Sending Results to NRL
• 3 McFarland units solution of strain
in 3 mL of 0.85% NaCl solution was
prepared starting from a culture:
• 3 mL into 1 L of pasteurized
commercial bovine milk (negative
for SEs);
• Milk incubated at 37°C for 24 h;
• CPS (10^5 CFU/ml);
• milk (25 mL) verified ESMv5;
• Milk (75 ml) to EU RL [SED] = 11.5
ng/g
milk samples
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Positive samples contaminated with Staphylococcal
Enterotoxin type D solution
Positive and Negative samples tested by NRL
before the shipment
Sending samples to labs
Sample analyses
Sending Results to NRL
• Semisoft, pasteurized bovine cheese
was produced at the pilot dairy plant at
the NRL with about 20 L of SE-positive
milk obtained;
• Warmed in a caldron at 37°C, liquid
rennet and cheese starter culture were
added at 10 mL/ 100 L and 1.5 g/100
liters;
• The curd was then broken into
fragments and whey was removed;
• 5 days 10°C ripening.
• Cheese (25 g) verified ESMv5;
• Cheese (75 g) to EU RL [SED] = 8.2 ng/g
Cheese samples
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Cheese samples Milk samples
These different concentrations were obtained by mixing positive and negative food
matrices.
6.0
ng/mL
[SED]
3.0
ng/mL1.5
ng/mL
2.3
ng/g
4.6
ng/g[SED]
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Each laboratory recorded a single result for each sample: detection or no detection of SEA through SEE.
Laboratory analysis
The laboratories were instructed to analyze the samples according to the European screening method
version 5.
After a single dialysis concentration step, detection can be performed with either an enzyme-linked
immunofluorescent assay (VIDAS SET 2) or an enzyme-linked immunosorbent assay (RIDASCREEN SET
Total). The laboratories were free to choose either of these two commercial kits for analysis of the ring
trial samples.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Statistical analysis
1. Data base
• Ad hoc database creation;
• Stata Statistical Software release 11.2 (StataCorp LP, College Station, TX).
2. Descriptive analysis about general information
• T°C receipt of the samples (Position index and variability)
• T°C storage of the samples
• Evaluation RIDAS/VIDAS analysis
• Dates (concordance receiving/start analysis/batches/expiration)
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Statistical analysis
3. Reproducibility (Aim of the PT)
• Results (positive/Negative)
• Evaluation by calculating Cohen’s k (+IC 95%) for any possible combination of methods (RIDAS and
VIDAS) and food matrices (cheese and milk). This index allows calculation of the real agreement
between two raters independently evaluating the same observations and removes the random
component of the results.
The calculation is based on the difference between how much agreement is actually present (observed
agreement) and the agreement expected to be present by chance alone (expected agreement).
Agreement is considered good at k > 0.6 and excellent at k > 0.92. Cohen’s k evaluates only the
agreement between results of same observations without taking into account their diagnostic
correctness.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Materials and Methods
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Statistical analysis
4. Statistical accuracy
• Sensitivity and Specificity (Se and Sp) and their respective 95% CIs;
• Each possible combination of the two methods (RIDAS and VIDAS) and the food matrices (cheese
and milk.
• Accuracy was considered satisfactory for laboratories when sensitivity and specificity were both >
or equal to 80%.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Descriptive analysis about general information
0
5
10
15
20
25
Results return Methods Results
RID
AS
VID
AS
19
/24
Re
turn
of
resu
lts
1 l
ab
23
/24
sa
mp
les
20
/24
co
rre
ctre
sult
s
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Descriptive analysis about general information
• L03 had three false-positive results for milk samples;
• L04 had false-positive results for three milk samples and one cheese sample and one false-negative
result (cheese sample 7 ng/g)
• L11 obtained one false-positive result for one milk sample;
• L23 obtained two false-positive results for a negative milk sample and one false-negative (cheese
sample 4.6 ng/g.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Reproducibility
High level of reproducibility for the laboratory network: k- combined values ranged from 1.00 (RIDAS
method) to 0.88 (VIDAS method) and to 0.90 (both methods combined).
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Sensitivity and Specificity
agreement was not
perfect for four
laboratories
reported three,
five, one, and two
discrepancies
22 labs max values
20 labs max
values
10th workshop of the NRLs for Coagulase Positive Staphylococci
Discussion
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Cohen’s k index has some limitations and disadvantages:
• It depends on the prevalence of the condition being examined.
• When marginal totals are highly symmetrically unbalanced, a low k value will result despite a high
value of agreement, whereas the opposite will occur when marginal totals are highly asymmetrically
unbalanced.
• This study, the number of negative samples was higher than the number of positive samples, and
the comparisons with imperfect agreement were nearer to a highly symmetrically unbalanced than a
highly asymmetrically unbalanced situation.
• We assume that our k values were conservative estimations of the agreement among laboratories.
At the spiking levels used in the study, sensitivity and reproducibility for both the VIDAS and RIDAS
methods are expected to be 100%.
10th workshop of the NRLs for Coagulase Positive Staphylococci
Discussion
NRL for
Coagulase Positive
Staphylococci
including S.aureus
PT second yearsame 23 labs
+ 2 new labs
↑ labs agreements in results (4 to 8)
↑ discrepancies (da 8 a 17)
VIDAS increase use despite of RIDAS (2011 and 2013)
↓ labs RIDAS (21% in 2011; 4 % in 2013)
• Elevato grado di riproducibility
K-combined 2010: 0.96 (0.93-1.00)
2011: 0.90 (0.88-0.93)
2013: 0.88 (0.85-0.90)
• Follow-up:
7/8 labs expected results
1 lab training
PT second yearsame 23 labs
+ 2 new labs
↑ labs agreements in results (4 to 8)
↑ discrepancies (da 8 a 17)
VIDAS increase use despite of RIDAS (2011 and 2013)
↓ labs RIDAS (21% in 2011; 4 % in 2013)
• Elevato grado di riproducibility
K-combined 2010: 0.96 (0.93-1.00)
2011: 0.90 (0.88-0.93)
2013: 0.88 (0.85-0.90)
• Follow-up:
7/8 labs expected results
1 lab training Riproducibility:
MEAT: 6/24
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
Riproducibility IIZZSS net
• CHEESE: k-combined = 0,94 (IC95% 0,89-0,99)
• MILK: k-combined = 0,97 (IC95% 0,92-1,00)
• MEAT: k-combined = 0,79 (IC95% 0,75-0,82)
• Withouth differences between methods and matrix:
k-combined = 0,88 (IC95% 0,85-0,90)
very high (due to increased number of participants)
...but discrepancies in MEAT
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
The results of RT: today and the past
0,88
0,89
0,90
0,91
0,92
0,93
0,94
0,95
0,96
0,97
0,98
0,99
1,00
2010 2011 2012 2013 2014 2015
K-combined: agreement among the net
10th workshop of the NRLs for Coagulase Positive Staphylococci
Results
NRL for
Coagulase Positive
Staphylococci
including S.aureus
0,79
0,810,820,830,84
0,860,870,880,89
0,910,920,930,94
0,960,970,980,99
0,80
0,85
0,90
0,95
1,00
2010 2011 2012 2013 2014 2015
ComplessivoLatteFormaggioCarne
1234
6789
11121314
16171819
21222324
0
5
10
15
20
25
2010 2011 2012 2013 2014 2015
Numero di partecipanti nei RT
carne formaggio latte
The results of RT: today and the past
# of participants