International Journal of Food Microbiology 141 (2010) S180–S187
Contents lists available at ScienceDirect
International Journal of Food Microbiology
j ourna l homepage: www.e lsev ie r.com/ locate / i j foodmicro
Food safety performance indicators to benchmark food safety output of food safetymanagement systems
L. Jacxsens a,⁎, M. Uyttendaele a, F. Devlieghere a, J. Rovira c, S. Oses Gomez c, P.A. Luning b
a Department of Food Safety and Food Quality, Laboratory of Food Preservation and Food Microbiology, Faculty of Bioscience Engineering, University of Ghent,Coupure Links, 653, 9000 Ghent, Belgiumb Product Design and Quality Management Group, Department of Agrotechnology and Food Sciences, Wageningen University, P.O. Box 8129, NL-6700 EV Wageningen, The Netherlandsc Department of Biotechnology and Food Science, University of Burgos, Pza. Misael Bañuelos s/n 09001, Burgos, Spain
⁎ Corresponding author. Department of Food Safety aFood Microbiology and Food Preservation, Coupure linTel.: +32 9 264 60 85; fax +32 9 225 55 10.
E-mail address: [email protected] (L. Jacxs
0168-1605/$ – see front matter © 2010 Elsevier B.V. Adoi:10.1016/j.ijfoodmicro.2010.05.003
a b s t r a c t
a r t i c l e i n f oKeywords:
Performance indicatorFood safetyFood safety management systemThere is a need to measure the food safety performance in the agri-food chain without performing actualmicrobiological analysis. A food safety performance diagnosis, based on seven indicators and correspondingassessment grids have been developed and validated in nine European food businesses. Validation wasconducted on the basis of an extensive microbiological assessment scheme (MAS). The assumption behindthe food safety performance diagnosis is that food businesses which evaluate the performance of their foodsafety management system in a more structured way and according to very strict and specific criteria willhave a better insight in their actual microbiological food safety performance, because food safety problemswill be more systematically detected. The diagnosis can be a useful tool to have a first indication about themicrobiological performance of a food safety management system present in a food business. Moreover, thediagnosis can be used in quantitative studies to get insight in the effect of interventions on sector orgovernmental level.
nd Food Quality, Laboratory ofks 653, 9000 Ghent, Belgium.
ens).
ll rights reserved.
© 2010 Elsevier B.V. All rights reserved.
1. Introduction
Information of the food safety performance in the agri-food chainis desired by food business operators, government/food safetyauthorities and/or sector organizations in the frame of the evaluationof implemented interventions to improve the (microbiological) foodsafety output. Major financial, technological and managerial invest-ments are conducted in the last 10 years in Europe in order to imple-ment food safety management systems (FSMS) along the agri-foodchain. The ‘white paper on food safety’ of the European Commissionstated a decenniumof new and harmonised legislation in the period of2000–2010 (Anonymous, 2000). The ‘general food law’ (Albersmeieret al., 2009; Anonymous, 2002) lays down new principles in thecontrol of food safety, e.g. the responsibility for food safety with thefood business operator from farm to fork and the introduction ofan FSMS along the agri-food chain based on Good ManufacturingPractices andHazardAnalysis andCritical Control Point principles. As aresponse to the above challenges, Quality Assurance (QA) standardsand guidelines are required by various stakeholders (e.g. food safetyauthorities, retailers and sector organizations) to control and assurequality and safety of food (Fulponi, 2006; Gorris, 2005; Jacxsens et al.,
2009a; Luning and Marcelis, 2009). The wide range of legislativedocuments, QA standards and guidelines leads to difficulties for foodbusinesses and specific small andmediumenterprises (SMEs) to selectand implement these into their company specific FSMS.
Questions arising now are “how effective are these FSMS?” and“by implementation of a good performing FSMS, is the food safetyoutput then better?” Food business operators want to see their returnon investment and need to have insight in which aspects of theirFSMS they should further improve (Cormier et al., 2007; Fraser andMonteiro, 2009; Luning et al., submitted for publication; Tsalo et al.,2007). Also stakeholders as the government, food safety agencies and/or sector organizations are interested in these questions. Measuringperformance of implemented interventions is necessary to see theeffectiveness of the interventions and to identify bottlenecks for fur-ther improvement (Fraser and Monteiro, 2009). How far are we after10 years of investment in food safety? However, regarding perfor-mance measurement of food safety not many studies are yet pub-lished (e.g. Fazil et al., 2009; IFT, Institute of Food Technologists, 2004;Serapiglia et al., 2007). Recent papers are found in other domainssuch as in the environmental and waste management (e.g. Bechiniand Castoldi, 2009; Block et al., 2007; Haughton et al., 2009; OECD,Organisation for Economic co-operation and development, 1991;Van Gerven et al., 2007;) or in health care (e.g. Copnell et al., 2009;McDonald, 2009) or economical sustainability studies (e.g. Erol et al.,2009). In these studies, an attempt is made to define indicators andto measure performance.
S181L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
Recently, a food safety management system diagnostic instru-ment (FSMS-DI) (Luning et al., 2008; 2009) and a microbial assess-ment scheme (MAS) (Jacxsens et al., 2009b) have been developed tosystematically assess the performance of a company specific FSMSindependent of QA standards and guidelines that have been im-plemented. The FSMS-DI encompasses a systematic analysis of whichcore control and assurance activities are addressed in the companyspecific FSMS and an assessment of the levels at which these controland assurance activities are executed. Moreover, it includes an assess-ment of the contextual situation wherein the FSMS has to operate(Luning et al., 2008; 2009).
The MAS protocol enables a methodical analysis of actualmicrobiological performance of a FSMS as an indication of their foodsafety output. The basic principle behind MAS is that low numbersof microorganisms and small variations in microbiological countsindicate an effective and well-functioning FSMS. It contains differentsteps 1) selection of critical sampling locations (CSL, i.e. food products,environmental samples and hands of the personnel), 2) selectionof microbiological parameters (pathogens as Listeria monocytogenes,Salmonella spp., Campylobacter spp. (in poultry processing), hygieneindicators as E. coli and Enterobacteriaceae and utility parametertotal viable counts), 3) sampling frequency, 4) selection of samplingand analytical method, and 5) data processing and interpretation(Jacxsens et al., 2009b). Both tools can be used separately and providedifferent insights, i.e. system versus actual microbiological perfor-mance analysis. The tools can be also applied as combined assessment(e.g. Luning et al., Submitted for publication; Sampers et al., 2010).
The MAS protocol is, however, not always possible to performbecause 1) the translation of the protocol to the company specificsituation requires in-depth knowledge of the behaviour of micro-organisms and the production process, 2) it is expensive and timeconsuming to perform the microbiological analysis, and 3) expertknowledge is necessary for the interpretation of the obtained resultsand to make the link towards the FSMS (Jacxsens et al., 2009b). Thereis a need to be able to measure the food safety performance in theagri-food chain without performing actual microbiological analysis.The FSMS-DI, developed by Luning et al. (2008; 2009), offers apossibility to add performance indicators to diagnose the microbio-logical performance of an FSMS without the requirement of actualmicrobiological analysis. The diagnosis may provide a first indicationof the microbiological performance of a current FSMS and measuringthe food safety output.
The objective of this paper is to develop food safety performanceindicators (FSPI), which can be introduced in the FSMS-DI, in order toobtain a first indication of the microbiological food safety perfor-mance of an implemented FSMS. The selection of the FSPIs is basedon literature analysis and expert discussions, and validated valida-tion was conducted using actual microbiological analysis according tothe MAS protocol of Jacxsens et al. (2009b) in nine European foodbusinesses.
2. Materials and methods
2.1. Food businesses
This study was conducted in nine European food businesses(poultry 1, poultry 2 and poultry 3, beef, lamb, dairy 1, dairy 2, pork 1and pork 2); both industrial business and SMEs, most of themparticipating in a European project Pathogen Combat. Poultry 1 and 2are poultry slaughterhouses in Belgium producing carcasses andpoultry meat cuts. Poultry 3 has the same activities but is located inSpain. Pork 1 is a pork meat processing business in Belgium producingcooked ham. Pork 2 is a pork meat processing business in Spainproducing dry cured ham. Beef is a beef meat processing business inSpain producing beef meat cuts from carcasses, similar for lamb. Dairy
1 is a dairy business in Italy producing cheese and dairy 2 similar butis located in Cyprus.
2.2. Microbiological assessment scheme (MAS) principle
The MAS protocol of Jacxsens et al. (2009b) was adapted for thespecific production process of the nine food businesses. The detailedinformation of these MAS analyses have been described by Jacxsenset al. (2009b) for pork 1, Sampers et al. (2010) for poultry 1 and 2,and Luning et al. (Submitted for publication) for poultry 3, beef andlamb. The samples of theMASwere taken by independent researchersand were analysed in research laboratories in Belgium, Spain andGreece.
To judge the microbiological data of the different critical samplinglocations, levels have been defined (Jacxsens et al., 2009b): for eachanalytical microbiological parameter that has been analysed over allcritical sampling locations (CSLs), a level 0, 1, 2, or 3 can be received(Fig. 1). Level 3 represents a good safety performance, i.e. legal criteriaor guidelines are respected; no improvement is required and thecurrent level of the FSMS seems sufficient to cover respective hazard.Level 2 indicates a moderate safety performance, i.e. legal criteria orguidelines are exceeded; improvements can be made on a specificcontrol activity in the FSMS. Level 1 reveals a poor safety performance,i.e. legal criteria or guidelines are exceeded; improvements need to bemade on several control activities in the FSMS. Level 0 is applied whenthe parameter is not applied for that specific case e.g. Campylobacterwhich is only applied in poultry processing (Jacxsens et al., 2009b).
Based on the levels, a microbiological safety level profile can bemade by summation of the levels for the individual microbiologicalparameters up to a total of 18 or 21 depending on the total numberof parameters that have been selected (e.g. in the case of poultryproduction seven parameters have been analysed). To obtain an overallindication of the actual food safety performance, assigned scores can beattributed as follows (Table 3). If the average level is between 0 and 1.2,then an assigned score of 1 is attributed. If the average level is between1.3 and 1.7, then an assigned score of 1_2 is attributed. If the averagelevel is between 1.8 and 2.2, then an assigned score of 2 is defined. Ifthe average level is between 2.3 and 2.7, then an assigned score of 2_3is given. Finally, if the average level is between 2.8 and 3.0, then anassigned score of 3 is attributed (Sampers et al., 2010).
2.3. Development of the food safety performance diagnosis
To develop the food safety performance diagnosis, indicatorshave been selected and grids have been developed with descriptionsof different levels for each indicator. This is in alignment with thediagnosis of the FSMS activities (Luning et al., 2008; 2009) and usingthe principles behind the MAS levels as described by Jacxsens et al.(2009b). The selection of indicators and description of grids werebased on literature study and expert discussions.
In total seven food safety performance indicators have beenselected, and four different levels (0, 1, 2, and 3) have been describedin the grids (Tables 1 and 2 and Sections 3.1.1, 3.1.2 and 3.1.3). Thelevels in the food safety performance grids were based on the criteriaused to define the levels of core control and assurance activities, andthe criteria used to define the FS performance levels in MAS (Jacxsenset al., 2009b; Luning et al., 2008; 2009). Level 0 (i.e. no indicationof food safety performance) is referring to absent, not present, notconducted. It means that e.g. the FSMS evaluation is not done, and orthat the specific food safety performance information is not known.Level 1 (i.e. poor performance) is associated with aspects like ad-hocsampling, minimal criteria used for FSMS evaluation, and havingvarious food safety problems due to different problems in the FSMS.Level 2 (i.e. moderate performance) is referring to regular sampling,several criteria used for FSMS evaluation, and having restricted foodsafety problems mainly due to one (restricted) type of problem in
Fig. 1. Microbiological safety level profiles according to Jacxsens et al. (2009b) and Sampers et al. (2010) for nine European food businesses A level 0, 1, 2 or 3 is attributed for eachmicrobiological parameter (Level 3 represents a good safety performance, i.e. legal criteria or guidelines are respected; no improvement is required and the current level of the FSMS seemssufficient to cover respective hazard. Level 2 indicates amoderate safety performance, i.e. legal criteria or guidelines are exceeded; improvements can bemade on a specific control activityin the FSMS. Level 1 reveals a poor safety performance, i.e. legal criteria or guidelines are exceeded; improvements need to be made on several control activities in the FSMS. Level 0is applied when the parameters is not applied for that specific case e.g. Campylobacterwhich is only applied in poultry processing (Jacxsens et al., 2009b)). (TVC: total viable counts, Stap:S. aureus, ENTE: Enterobacteriaceae, ECOL: E. coli, CAMP: Campylobacter spp., SALM: Salmonella spp., LIST: Listeria monocytogenes). A maximum of 21 can be obtained for poultry 1, 2 and3 (7 parameters are analysed) and maximum of 18 for the others (6 parameters are analysed). The overall MAS score is given for each food business in bold.
S182 L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
the FSMS. Level 3 (i.e. good performance) is referring to a systematicevaluation of the FSMS using specific criteria and having no safetyproblems. Based on the scores for the individual indicators, an overall
Table 1Selection of external food safety performance indicators and their grids.
Indicator Motivation Level 0— no indicaof food safety perfo
Comprehensiveness ofexternal evaluation: Howwould you typify yourFood Safety ManagementSystem evaluation?
A certification audit by a third party oran inspection by the national foodsafety agency gives an external andindependent evaluation of the currentFood Safety Management System.
An inspection or anof the Food SafetyManagement Systewas never perform
Seriousness of remarks:How would you indicateseriousness of remarksof the FSMS evaluation?
A positive evaluation (without seriousremarks) of the FSMS by a national foodsafety agency and/or accredited thirdparty indicates a good safety performance(i.e. all requirements of the stakeholdersare met)
Not appropriate sinnever an inspectionan audit of the FSMwas performed
Type of microbiologicalfood safety complaints:How would you typifythe microbiologicalfood safety complaintsof customers?
The presence of a good functioningsystem for complaint registration andevaluation of complaints is an importantaspect in a FSMS. Low number or nocomplaints regarding themicrobiological food safety of finalproducts indicates a good performanceof food safety. Also when the complaintscan be dedicated to one specific aspect ofthe FSMS or one type of pathogen a wellperforming FSMS and a good food safetyoutput can be expected.
Not known becausecomplaint registrat
Type of hygiene-relatedcomplaints: How wouldyou typify the hygienerelated complaints bycustomers?
The presence of a good functioningsystem for complaint registration andevaluation of complaints is animportant aspect in a FSMS. Lownumber or no complaints of hygieneindicates a good performance of foodsafety. Alsowhen the complaints can bededicated to one specific aspect of theFSMS or one group of hygiene indicatora well performing FSMS and a goodfood safety output can be expected.
Not known since ncomplaint registra
score and an assigned score can be defined to give an overall judgmentof the food safety performance of the current FSMS, similar as de-scribed for MAS (Section 2.2).
tionrmance
Level 1 — poor foodsafety performance
Level 2 — moderatedfood safety performance
Level 3 — good foodsafety performance
audit
med
Inspection of the FSMSperformed by nationalfood safety agency
Audit of the FSMSperformed by oneaccredited third party
Audits and/or inspectionsof the FSMSperformedbyseveral accredited thirdparties and/or nationalfood safety agency
ceorS
Major remarks onvarious aspects ofthe FSMS
Major remark only on onespecific aspect of FSMS(eventually additionalminor remarks on otheraspects of the FSMS)
No major remarks and/oronly minor remarks onspecific or various aspectsof the FSMS
noion
Various complaintswhich canbededicatedtowards multipleproblems in thefunctioning of theFSMS
Restricted complaintswhich can be dedicatedto one specific problemin the functioning ofthe FSMS
No complaints regardingmicrobiological foodsafety
otion
Various complaintswhich can bededicated towardsmultiple problems inthe functioning of theFSMS
Restricted complaintswhich can be dedicated toone specific problem inthe functioning of theFSMS
No complaints regardingmicrobiological hygieneindicators
Table 2Selection of internal food safety performance indicators and their grids.
Indicator Motivation Level 0 — no indicationof food safety performance
Level 1 — poor foodsafety performance
Level 2 —moderatedfood safety performance
Level 3 — good foodsafety performance
Advancedness ofproduct sampling:How would you typifyyour product samplingto confirm microbiologicalperformance?
Regular sampling and differenttypes of samples gives a morecomprehensive and accurateindication of the actualmicrobiological performanceof your FSMS
No samples are taken andnomicrobiological analysesare performed
Ad hoc sampling(on the demandingof customers orlegislation) and onlyon final food product
Regular samplingconducted on bothfinal food productand raw material(s)
Structured sampling(with fixed frequencyand company ownsampling plan is present)and conducted on finalfood product, raw material(s)and environmental samples
Comprehensiveness ofmicrobiological criteria:Which judgment criteriaare used to interpretmicrobiological results?
Using more criteria to criticallyinterpret obtained results ofmicrobiological analyses gives amore accurate indication of themicrobiological performance of theFSMS.
No criteria known sincemicrobiological analysesare not performed
Only legal criteria used(restricted numberof combinationpathogen/hygieneindicator and foodmatrix)
Combination oflegal criteria andrequirements andor specifications(set by externalparties) is used
Combination of legalcriteria, requirements andor specifications by externalparties and additionalcompany specificspecifications establishedin internal guidelines
Type of hygieneand pathogennon-conformities:How would youtypify your hygieneand pathogennon-conformities?
The presence of a good system fornon-conformities registration andevaluation gives a good indication ofthe performance of a FSMS. Lownumber or no non-conformitiesindicates a good food safetyperformance. Also when the non-conformities can be dedicated to onespecific aspect of the FSMS or onegroup of hygiene indicator/pathogena well performing FSMS and a goodfood safety performance can beexpected.
Not known since nointernal product analyse,and no non-conformitiesregistration
Several non-conformities whichcan be dedicatedtowards multipleproblems of thefunctioning of theFSMS
Restricted numberof non-conformitieswhich can bededicated to onespecific problem inthe functioning ofthe FSMS
No non-conformitiesregarding microbiologicalfood safety/hygiene indicators
S183L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
3. Results and discussion
3.1. Food safety performance diagnosis
The term performance indicator usually refers to a measurableparameter that gives an indication about the performance of a certain(or set of) activity (Block et al., 2007; Carruthers, 2009; PEST Analysis,2009; Van Gerven et al., 2007). Typical for a performance indicator isthat it includes a certain judgment (is it good or moderate or bad). Anindicator can provide information about or can be indicative for anoverall situation (Luning et al., 2008). We adapted the name towardsfood safety performance indicator (FSPI) because the focus of thisresearch is microbiological food safety.
3.1.1. Types of indicatorsThe development of the FSMS diagnosis consists of the selection of
indicators and development of grids with descriptions of differentlevels.
In order to obtain an indication of the food safety performance wehave selected indicators to judge how a FSMS has been appreciated byindependent experts (i.e. qualitative indicators or descriptive indica-tors), and indicators to get insight in the microbiological performanceof the FSMS (i.e. quantitative indicators or indicators based onmicrobiological analysis). The assumption behind the food safetyperformance diagnosis is that food businesses which evaluate theirFSMS performance in a more structured way and according to verystrict and specific criteria will have a better insight in their actualmicrobiological food safety performance, because food safety pro-blems will be more systematically detected. Therefore, food busi-nesses following such strict FSMS evaluations, which have no foodsafety problems, are considered as revealing a good food safety per-formance. In fact, only judging the information about the microbio-logical performance by quantitative indicators might ignore theimportance of the appreciation of FSMS by independent experts,measured by qualitative indicators. Besides the distinction betweenqualitative and quantitative indicators, we have made a distinctionbetween the external and internal evaluation of the FSMS. Both ways
of FSMS evaluation are important, because similar findings by internaland external system evaluations support the reliability of the FSMSperformance judgement (Block et al., 2007; Van Gerven et al., 2007).
3.1.2. External food safety performance assessmentIn total four FSPIs have been selected to analyse the external
judgment of the FSMS performance (Table 1). The first indicator is the‘comprehensiveness of external evaluation’. It refers to the type andextent of external evaluation of the food business' FSMS. Various waysof external judgement of an FSMS performance can be distinguished.More in detail, an inspection is conducted by the national food safetyagency in which typical legislative aspects are checked on the basisof a checklist (check to compliance). An audit is conducted by anaccredited third party against an acknowledged Quality Assurancestandard as Global Standard for Food Safety (BRC, British RetailConsortium, 2008) or Standard for Auditing, Retailer and Wholesalerbranded food products (IFS, International Featured Standards, 2007).A certification audit should give confidence to compliance to therequirements of a certain QA standard based on a valid examination(Albersmeier et al., 2009; Block et al., 2007; Tuominen and Maijlala,2009; Van Gerven et al., 2007). A certification audit by a third party oran inspection by the national food safety agency gives an external andindependent evaluation of the FSMS. It should be noted, that onlythe external checks are considered as third party audits, i.e. conductedby food safety authorities or by accredited bodies, but not secondparty audits (e.g. conducted by customers). At level 3 (i.e. good per-formance) audits and or inspections of the FSMS have been performedby several accredited third parties and or the national food safetyagency, or multiple QA standards are audited by the same third party(e.g. combined audits of BRC and IFS). Crucial for level 3 is that morethan one audit is performed by accredited third party(s) or in com-bination with an inspection by the national food safety agency (e.g. aBRC audit by accredited third party and inspection of the national foodsafety agency). In this way different stakeholders have been reviewingthe FSMS resulting in a more comprehensive assessment of the FSMSperformance. The difference between a poor and moderate level liesin the fact that an inspection is conducted by the national food agency
S184 L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
(level 1) or an audit by a third party (level 2), in both cases only onestakeholder was checking the FSMS. A level 0 corresponds with thesituation that a food business never had an external check of theirsystem.
In order to obtain an indication of the microbiological perfor-mance, as judged by the external evaluation, the indicator ‘serious-ness of remarks’ has been defined. The increasing levels in the gridcorrespond with increasing food safety performance. Table 1 showsthat level 3 means that all requirements of the stakeholders are met. Apositive evaluation i.e. without major remarks of the FSMS indicates agood food safety performance. Major remarks are given when duringan audit or an inspection it is clear that requirements are not fulfilled,or are not completely met with an implication on the performanceof an FSMS (Albersmeier et al., 2009; Block et al., 2007; BRC, BritishRetail Consortium, 2008; ISO International Standard Organisation22000, 2005; Van Gerven et al., 2007). Minor remarks are giventowards small incompleteness in fulfilling requirements. The differ-ence between a poor and moderate level lies in the fact that majorremarks are given on various aspects of the FSMS (level 1) or on onespecific aspect of the FSMS (level 2). If the remarks are clearlyattributed to one specific aspect of the FSMS (e.g. a specific preventivemeasure), then it is level 2 or 3. Crucial for level 3 is that the remarksare only minor remarks. Level 1 refers to a situation where majorremarks are given, whereas level 0 corresponds with not appropriatesince never an audit or inspection was performed.
The presence of a good functioning system for complaint regis-tration and evaluation of complaints is also an important aspect ina FSMS (Block et al., 2007; Luning and Marcelis, 2009; Van Gervenet al., 2007). Complaints' registration is also an important source ofknowledge (Dubrawski et al., 2007). We have selected therefore asthird indicator, the ‘type of microbiological food safety complaints’(Table 1). Here we consider only the complaints from customers onthe level of the final food product. A customer can be a final consumeror can be another actor in the agri-food chain in case of business-to-business activities. Since, the diagnosis is aimed at obtaining anindication of the food safety performance, we focused on majorpathogens like, L. monocytogenes, Salmonella spp., E. coli O157:H7,Campylobacter spp. No complaints regarding the microbiological foodsafety of final products provide an indication of a good performanceof food safety (level 3). When the complaints can be dedicated to onespecific aspect of the FSMS (e.g. control of raw materials, cross con-tamination and control of critical control point) or one pathogen (e.g.presence of L. monocytogenes in raw materials) a moderated foodsafety output can be expected (level 2). Level 1 means that the foodbusiness has various complaints which can be dedicated to multipleproblems with the functioning of the FSMS. When no complaintregistration and analysis is performed, level 0 should be filled in.
A fourth indicator is defined related to the complaint registrationof hygiene indicators, i.e. the ‘type of hygiene-related complaints’(Table 1). Hygiene is commonly determined by use of hygieneindicators. In the food safety performance diagnosis, we thereforefocused on major hygiene indicators such as E. coli (indicator offaecal contamination), Enterobacteriaceae (indicator of environmen-tal contamination) and Staphylococcus aureus (indicator of handhygiene of the personnel) (Ghafir et al., 2008; Jacxsens et al., 2009b;Sampers et al., 2010). No complaints of hygiene indicators providean indication that the FSMS is functioning good and results in safefood products (level 3). When the complaints can be dedicated toone specific aspect of the FSMS or one group of hygiene indicator amoderate food safety output can be expected (level 2). Crucial forsituation 3 is that there are no complaints about hygiene, whereas atlevel 2 the reason of the complaints can be dedicated to one specificaspect of the FSMS (e.g. inadequate control of temperature, poorhand hygiene, insufficient cleaning and disinfection). Levels 0 and 1are defined as for the previous indicator ‘type of microbiological foodsafety complaints’.
3.1.3. Internal food safety performance assessmentTable 2 is summarizing the indicators related to the internal
evaluation and the actual performance judgment. The first indictorthat has been selected is related to the characteristics of the productsampling procedure/method that is used by a food business to analysetheir microbiological performance, i.e. ‘advancedness of productsampling’. This indicator is aimed at getting insight the frequency ofsampling and the types of samples that are taken to judge internallythe actual microbiological performance of the FSMS. Regular samplingand different types of sampleswill provide amore comprehensive andaccurate indication of the actual microbiological performance (IFT,Institute of Food Technologists, 2004; Jacxsens et al., 2009a; vanSchothorst et al., 2009). Three types of samples can be distinguished.Final food product is the product which is sold to the next step in theagri-food chain, it can be a consumer product or intermediate productwhich is further processed (business-to-business). Raw materialsare products which are bought by suppliers as initial material forthe food business' own production. Environmental samples are sam-ples from food contact materials, hands of personnel e.g. by swab orcontact plates. Level 3 (good performance) corresponds with a struc-tured sampling (with fixed frequency and tailored sampling planpresent), which is conducted on final food product, raw material(s)and environmental samples. Level 2 is associated with regular sam-pling conducted on both final food product and raw material(s),whereas level 1 refers to ad-hoc sampling (on demand of customersor legislation) which is only executed for final food products. Level 0indicates that no samples are taken and no microbiological analysesare performed; so the food business has no insight in the actualperformance of its FSMS.
A second internal performance indicator is related to the judgmentcriteria which are used to interpret the microbiological results, i.e.the ‘comprehensiveness of microbiological criteria’ (Table 2). The ideabehind this indicator is the fact that using more criteria to criticallyinterpret obtained results of microbiological analyses gives a moreaccurate indication of themicrobiological performance of the FSMS (e.g.ICMSF International Commission on Microbiological Specifications forFoods, 2005; IFT, Institute of Food Technologists, 2004; Jouve, 1996; vanSchothorst et al., 2009). A level 3 is achieved, when a combination oflegal criteria, requirements and or specifications by external parties andadditional food business specific specifications established in internalguidelines are applied to interpret the obtained microbiological results.Crucial for level 3 is that the food business has, in addition to legal andexternal party requirements, specific specifications which are accom-panied with strict guidelines. The internal guidelines are defined by thefood business based on e.g. historical and internal knowledge, whichshould be scientifically supported. Level 2 indicates that a combinationof legal criteria and requirements and or specifications (set by externalparties) is used. Requirements and or specifications by external partiesare defined by e.g. customers, retailers or by sector organisations butthe food business can't discuss on these levels ofmicro-organisms. Level1 is related to the application of only legally demanded criteria as statedin the Regulation (EC) No. 2073/2005 and/or Regulation (EC) No. 853/2004 (Anonymous, 2004; 2005) and/or additional national legislation.Level 0 refers to no criteria known since no microbiological analysesare performed.
A last internal performance indicator is related to the character-istics of the non-conformities, i.e. the ‘type of hygiene and pathogennon-conformities’ (Table 2). The presence of a good system to registerand systematically evaluate non-conformities means that any non-conformity will lead to additional insight in the production and safetyof food products (ISO International Standard Organisation 22000,2005). Level 3 is obtained when no non-conformities regarding mi-crobiological food safety or hygiene indicators are identified by theregistration and evaluation system of the food business. Level 2corresponds with a situation where in a restricted number of non-conformities are registered and evaluated, which can be dedicated to
S185L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
one specific problem in the functioning of the FSMS. If several non-conformities are identified, which can be dedicated towards multipleproblems of the functioning of the FSMS a level 1 needs to beattributed. At the lowest level (0) we have the situation that non-conformities are not known, since no internal product analysis arebeing executed and registration of non-conformities is not conducted.
3.1.4. Food safety performance diagnosisBased on the 7 performance indicators and their grids, a food
safety performance diagnosis can bemade for a specific food business.The QAmanager or food business operator can judge or diagnose theirown situation regarding the food safety output. Table 3 shows theFSPI scores of the 9 European food businesses. Finally, an overall andaverage score is made over the 7 indicators and an assigned score isattributed, according to Section 2.3.
3.2. MAS results
Microbiological food safety profiles, derived from the MASprotocol and applied in the nine European food businesses, aredemonstrated in Fig. 1. The profiles indicate to which extent thedifferent parameters (level 1, 2 or 3) are contributing to the foodsafety output. For the poultry processing, a total score of 21 is themaximum that can be achieved. For food business ‘poultry 1’problems have been identified for several microbiological parametersi.e. total viable counts, Enterobacteriaceae, E. coli, Campylobacter spp.Its overall MAS score is 10/21. Detailed investigation of the MAS datademonstrated that they have a problem due to the handlings atevisceration and due to further spreading of the micro-organisms inthe production process although the company was monitoring itscritical control points and was audited and inspected by third partiesand the national food safety agency (Sampers et al., 2010). Also infood business ‘poultry 2’ some room for improvement is present buttheir food safety performance is already high (18/21). They canfurther work on a reduction of L. monocytogenes in their environment,and reduction of total viable counts and Enterobacteriaceae in theirfinal products, probably due with the use of herbs. For detailed dis-cussion of MAS results of ‘poultry 1’ and ‘poultry 2’ the reader isreferred to Sampers et al. (2010). ‘Poultry 3’ can specifically work onthe reduction of the presence of L. monocytogenes and Campylobacterspp. and they could focus on their hygiene parameters, both found inthe foodstuff and environmental samples (overall MAS score 15/21)(Luning et al., Submitted for publication).
‘Dairy 1’ and ‘dairy 2’ are both cheese producing businesses. Theiroverall safety level profile can be improved, by increasing from 14/18up to 18/18. However, it is obvious from Fig. 1 that both dairy busi-nesses need to work on different types of micro-organisms, because
Table 3Levels (1 poor, 2 moderate, 3 good) and assigned score of food safety performance of Europeato assigned scores obtained via microbial assessment scheme (MAS).
P
External food safety performance indicatorsHow would you typify your food safety management system evaluation? 2How would you indicate seriousness of remarks of the FSMS evaluation? 3How would you typify the microbiological food safety complaints of customers? 2How would you typify the hygiene related complaints by customers? 1Average score for external indicators 2
Internal food safety performance indicatorsHow would you typify your product sampling to confirm microbiological performance? 3Which judgment criteria are used to interpret microbiological results? 2How would you typify your hygiene and pathogen non-conformities? 1Average score for internal indicators 2
Overall score FSPI 1Assigned score FSPI 2Assigned score MAS 1
they have considerably different food safety level profiles. For ‘dairy 1’problems with S. aureus during their cheese making process havebeen identified, but also the control of the hygiene indicators can beimproved. ‘Dairy 2’ needs to pay attention to their total viable countsbut also avoiding the presence of S. aureus.
A similar overall score is obtained for ‘pork 1’ and ‘pork 2’ (15/18and 14/18 respectively), although they reveal differences in theirprofiles. ‘Pork 2’ seems to have problems with the control of L.monocytogenes, which was found in both the foodstuff and environ-ment samples in their production of dry cured ham, while foodbusiness ‘pork 1’ could make some improvements related to thehygiene indicators in their production of cooked ham. Food business‘beef’ received a very good MAS result (overall MAS score 17/18).However, food business ‘lamb’ needs to improve activities in the FSMSrelated to different micro-organisms, e.g. measured counts of E. coliand total viable counts were too high on the lamb parts, already fromthe carcass level throughout their whole production process (overallMAS score 13/18). Detailed MAS results of ‘beef’, ‘lamb’ are discussedby Luning et al. (Submitted for publication). ‘Pork 1’MAS results havebeen published by Jacxsens et al. (2009b).
From the microbiological food safety level profiles and the overallMAS score, an assigned score can be derived (explained in Section 2.3,Table 3). It can be seen that independent from the European countryor the size or sector (meat and dairy), still improvements can bemaderegarding the food safety performance. Only food business ‘beef’obtained a maximum assigned score of 3.
3.3. Food safety performance diagnosis compared with MAS results
In order to validate if the selected FSPI and the diagnosis provide arealistic indication of the microbiological performance of an imple-mented FSMS-DI, the assigned scores of FSPI were compared tothe assigned scores from the MAS data (Table 3). The assigned scoresof FSPI ranged from 2 to 2_3, indicating a moderate to moderate-goodfood safety performance. The assigned score of the MAS resultsranged from 1_2 to 3, indicating a poor-moderate to good food safetyperformance.
Comparing both types of assigned scores reveals that the diagnosisof the food safety performance, conducted by the QAmanager itself, isin line with the obtained MAS results, conducted by independentresearchers and based on microbiological analysis. Out of the ninefood businesses, three obtained different assigned scores betweenMAS and FSPI. Food businesses ‘poultry 1’ and ‘poultry 3’ receivedslightly higher assigned scores based on the FSPI diagnose comparedto the MAS assessment. This indicates that during the self assessmentthey judged themselves slightly better compared to the independentMAS measurement. For example, ‘poultry 1’ received an assigned
n food businesses obtained via food safety performance indicators (FSPI) and compared
oultry 1 Poultry 2 Poultry 3 Dairy 1 Dairy 2 Pork 1 Pork 2 Beef Lamb
3 3 3 3 2 3 3 23 2 3 3 3 3 2 21 3 3 3 3 3 2 22 3 2 2 2 3 2 22.25 2.75 2.75 2.75 2.50 3 2.25 2
3 3 3 3 3 2 3 33 2 3 2 2 3 2 22 2 2 2 2 2 2 22.67 2.33 2.67 2.33 2.33 2.33 2.33 2.33
4/21 17/21 19/21 19/21 18/21 17/21 19/21 16/21 15/212_3 2_3 2_3 2_3 2_3 2_3 2_3 2
_2 2_3 2 2_3 2_3 2_3 2_3 3 2
S186 L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
score of 1_2 based on MAS, whereas an assigned score of 2 wasattributed, based on the food safety performance diagnosis. On theother hand, one food business (‘Beef’), received a slightly lowerassigned scores by the FSPI assessment compared to MAS, i.e. 2_3instead of 3. The comparison, between the two types of food safetyperformance assessment for the nine European food businesses,reveals that the developed food safety performance indicators are inline with the MAS data, and provide a first indication of the micro-biological performance of an implemented FSMS without use ofexperimental data. In case, the assigned scores based on the FSPIshow lower levels, a food business can decide to apply the morecomprehensive MAS assessment. In Table 3, we can also see that theaverage score of internal versus the external indicators is more or lessequal. This is important to prove the objectivity of the diagnosis ifthe internal performance is equal to external performance.
4. Conclusions
The food safety performance diagnosis, based on the7 indicators andtheir grids, can be a useful and cheap tool to obtain a first indicationabout the microbiological performance of an implemented FSMS. Theassumption behind the food safety performance diagnosis is that foodbusinesses that evaluate their FSMS performance in a more structuredway and according to very strict and specific criteria will have a betterinsight in their actualmicrobiological food safety performance, becausefood safety problemswill bemore systematically detected. Compared tothe MAS protocol, two important principles are different in the foodsafety performance diagnosis. First, no microbiological analysis needsto be conducted to get information of the actual microbiologicalperformance. Second, the diagnosis is conducted by the food businessitself (their QA manager or food business operator) and not by anindependent person. However, objectivity is introduced in the devel-oped food safety performance diagnosis by the descriptions of thelevels in the grids and in defining internal and external indicators. Theoutcome of MAS and FSPIs for nine European food businesses wascomparable.
The selected indicators and their grids have been inserted in theFSMS-DI, developed by Luning et al. (2008, 2009) to allow a completeself assessment of an FSMS without applying actual microbiologicalexperiments (i.e. context, core control, core assurance and finally andfood safety performance). The self assessment provides insight in thestrong and weak points of the current FSMS and supports a foodbusiness in identifying what/how to improve (Sampers et al., 2010).The more comprehensive microbiological assessment by MAS canprovide more in-depth insight in the actual microbiological problems,as has been demonstrated by Jacxsens et al. (2009b), Sampers et al.(2010), and Luning et al. (Submitted for publication). The combinedFSMS-DI and MAS assessment enable a detailed insight in possibleinterventions to go towards more advanced control and assuranceactivities in the FSMS and or less risky context situations, to obtain abetter food safety performance (Fraser and Monteiro, 2009; Samperset al., 2010).
Besides the usefulness for an individual food business, the foodsafety performance diagnosis can also be applied on the level ofgovernments or sector organizations. At sector level, the tool can beapplied to benchmark the food safety performance of different foodbusinesses, independent of the type of QA standard(s), which hasbeen implemented. At governmental level, it could be applied tobenchmark the food safety performance of different food businesses,independent of the sector or type of food business (e.g. size). Quan-titative studies using the full FSMS-DI (including the FSPI) may giveinsight in typical FSMS profiles, which may enable the developmentof dedicated interventions. Also the impact of (governmental) QAmeasures or interventions on the long-term can be measured by theFSPI alone or in combination with the FSMS-DI.
This study demonstrates that the food safety performance diagnosis,based on seven indicators, can be applied instead of microbiologicalanalysis, according to the MAS protocol, to get a first insight of themicrobiological performance of a food safety management systempresent in a food business.
Acknowledgments
The research performed has been part of the project FOOD-CT-2005-007081 (PathogenCombat) supported by the European Commissionthrough the Sixth Framework Program for Research and Development.The food safety management system diagnostic instrument (FSMS-DI)and the food safety performance indicators are available on the websitewww.pathogencombat.com ≫ self assessment.
References
Albersmeier, F., Schulze, H., Jahn, G., Spiller, A., 2009. The reliability of third-partycertification in the food chain: from checklists to risk-oriented auditing. Food Control20, 927–935.
Anonymous, 2000.White paperon foodsafety. Commissionof theEuropeanCommunities,Brussels, 12 January 2000, COM (1999) 719 final. http://ec.europa.eu/dgs/health_consumer/library/pub/pub06_en.pdf.
Anonymous, 2002. Regulation (EC) No 178/2002 of the European Parliament and of theCouncil of 28 January 2002 laying down the general principles and requirements offood law, establishing the European Food Safety Authority and laying downprocedures in matters of food safety. Official Journal of the European Communities(01/02/2002).
Anonymous, 2004. Regulation No 853/2004 of the European Parliament and of theCouncil of 29 April 2004 laying down specific hygiene rules for on the hygiene offoodstuffs. Official Journal of the European Communities (25/06/2004).
Anonymous, 2005. Commission Regulation No 2073/2005 of 15 November 2005 onmicrobiological criteria for foodstuffs. Official Journal of the European Communities(07/12/2007).
Bechini, L., Castoldi, N., 2009. On-farm monitoring of economic and environmentalperformances of cropping systems: results of a 2-year study at the field scale innorthern Italy. Ecological Indicators 9, 1096–1113.
Block, C., Van Gerven, T., Vandecasteele, C., 2007. Industry and energy sectors inFlanders : environmental performance and response indicators. Clean Technologiesand Environmental Policy 9, 43–51.
BRC, British Retail Consortium, 2008. Global Standard for Food Safety. Version 5. TSO,London.
Carruthers, H., 2009. Using PEST analysis to improve business performance. In Practice31, 37–39.
Copnell, B., Hagger, V., Wilson, S., Evans, S., Spivulis, P., Cameron, P., 2009. Measuringthe quality of hospital care: an inventory of indicators. Internal Medicine Journal39, 352–360.
Cormier, R.J., Mallet, M., Chiasson, S., Magnússon, H., Valdimarsson, G., 2007. Effectivenessand performance of HACCP-based programs. Food Control 18, 665–671.
Dubrawski, A., Elenberg, K., Moore, A., Sabhnani, M., 2007. Proceeedings Twenty-FirstNational Conference on Artificial Intelligence (AAAI-06). Eighteenth InnovativeApplications of Artificial Intelligence Conference (IAAI-06), pp. 1782–1788.
Erol, I., Cahar, N., Erel, D., Sari, R., 2009. Sustainability in the Turkish retailing industry.Sustainable Development 17, 49–67.
Fazil, A., Rajic, A., Sanchez, J., McEwen, S., 2009. Choices, choices: the application ofmulti-criteria decision analysis to a food safety decision-making problem. Journalof Food Protection 71, 2323–2333.
Fulponi, L., 2006. Private voluntary standards in the food system: the perspective ofmajor food retailers in OECD countries. Food Policy 31, 1–13.
Fraser, R., Monteiro, D.S., 2009. A conceptual framework for evaluating the most cost-effective intervention along the supply chain to improve food safety. Food Policy34, 477–481.
Ghafir, Y., China, B., Dierick, K., Dezutter, L., Daube, G., 2008. Hygiene indicatormicroorganisms for selected pathogens on beef, pork, and poultry meats in Belgium.Journal of Food Protection 71, 35–45.
Gorris, L.G.M., 2005. Food safety objective: an integral part of food chain management.Food Control 16, 801–809.
Haughton, A., Bond, A., Lovett, A., Dockerty, J., Sunnenberg, G., Clarks, S., Bohan, D., Sage,R., Mallott, M., Mallott, V., Cunningham, M., Riche, A., Shield, S., Finch, J., Turner, M.,Karp, A., 2009. A novel, integrated approach to assessing social, economic andenvironmental implications of changing rural land-use: a case study of perennialbiomass crops. Journal of Applied Ecology 46, 315–322.
IFS, International Featured Standards, 2007. IFS food Standard for Auditing, Retailer andWholesaler Branded Food Products. Version 5.
IFT, Institute of Food Technologists, 2004. Managing food safety: use of performancestandards and other criteria in food inspection systems. An authoritative report ofthe Institute of food technologists, pp. 1–15.
ISO International StandardOrganisation 22000, 2005. Food SafetyManagement Systems—Requirements for Any Organization in the Food Chain. Revision 1. Switzerland.
ICMSF International Commission onmicrobiological specifications for foods, 2005. Micro-organisms in Foods 6. Kluwer Academic, New York. ISBN 0-306-48675 ,763 pp.
S187L. Jacxsens et al. / International Journal of Food Microbiology 141 (2010) S180–S187
Jacxsens, L., Devlieghere, F., Uyttendaele, M., 2009a. Quality Management Systems inthe Food Industry. ISBN: 978-90-5989-275-0.
Jacxsens, L., Kussaga, J., Luning, P.A., Van der Spiegel, M., Devlieghere, F., Uyttendaele,M., 2009b. A microbial assessment scheme to support microbial performancemeasurements of food safety management systems. International Journal of FoodMicrobiology 134, 113–125.
Jouve, J., 1996. La qualité microbiologique des aliments : maîtrise et critères. CNERNA-CNRS. Polytechnica, Paris. ISBN: 2-84054-040-1. 563 pp.
Luning, P.A., Bango, L., Kussaga, J., Rovira, J., Marcelis, W.J., 2008. Comprehensiveanalysis and differentiated assessment of food safety control systems: a diagnosticinstrument. Trends in Food Science & Technology 19, 522–534.
Luning, P.A., Marcelis, W.J., 2009. Food Quality Management. Technological and Manage-rial Principles and Practices. Wageningen Academic Publishers, Wageningen. ISBN987-90-8686-116-3, The Netherlands.
Luning, P.A., Marcelis, W.J., Rovira, J., Van der Spiegel, M., Uyttendaele, M., Jacxsens, L.,2009. Systematic assessment of core assurance activities in company specific foodsafety management systems. Trends in Food Science & Technology 20, 300–312.
Luning, P.A, Jacxsens, L., Rovira, J., Osés, S.M., Uyttendaele, M., and Marcelis, W.J.(submitted). Microbiological food safety output related to the performance of afood safety management system: cases frommeat processing industries. Submittedfor publication in Food Control.
McDonald, M., 2009. Approach to improving quality: the role of quality measurementand a case study of the agency for healthcare research and quality pediatric qualityindicators. Pediatric Clinics of North America 56, 815.
OECD, Organisation for Economic Co-operation and Development, 1991. EnvironmentalIndicators: A Preliminary Set. OECD, Paris.
PEST Analysis, 2009. Value based management.net site visited June 2009. 2009http://www.valuebasedmanagement.net/methods_PEST_analysis.html.
Sampers, I., Jacxsens, L., Luning, P.A., Marcelis, M.J., Dumoulin, A., Uyttendaele, M. (2010.Performance of food safety management systems in poultry meat preparationprocessing plants and relation to Campylobacter spp. contamination. Accepted inJournal of Food Protection.
Serapiglia, T., Kennedy, E., Thompson, S., de Burger, R., 2007. Association of food premisesinspection anddisclosureprogramwith retail acquired foodborne illness and operatornoncompliance in Toronto. Journal of Environmental Health 70, 54–59.
Tuominen, P., Maijlala, R., 2009. Evaluating food safety management in professionalkitchens. Archiv für Lebensmittelhygiene 60, 151–160.
Tsalo, E., Drosinos, E.H., Zoiopoulos, P., 2007. Impact of poultry slaughter housemodernisation and updating of food safety management systems on themicrobiological quality and safety of products. Food Control 19, 423–431.
Van Gerven, T., Block, C., Geens, J., Cornelis, G., Vandecasteele, C., 2007. Environmentalresponse indicators for the industrial and energy sector in Flanders. Journal ofCleaner Production 15, 886–894.
van Schothorst, M., Zwietering, M., Ross, T., Buchanan, R., Cole, M., 2009. Relatingmicrobiological criteria to food safety objectives and performance objectives. FoodControl 20, 967–979.
Recommended
