The case for PPS validation
Professor Jacqui Reilly Health Protection Scotland and Glasgow Caledonian University
UK
Introduction to validation
• Validation studies are essential to assess the sensitivity (Se), specificity (Sp), positive predictive value (PPV) and negative predictive value (NPV) of PPS.
• Reproducibility (or reliability) is also an important concept for ensuring data is robust.
• Validation studies are rarely published, there are a variety of approaches described:
Less than half of the prevalence surveys published to date included an evaluation of the data.
Of those that included either a validation or an IRR study, the results were varied substantially
Underscoring the need for a formal evaluation to add confidence to the interpretation of the data.
Ref: Llata E, Gaynes RP, Fridkin S. Measuring the scope and magnitude of hospital-associated infection
in the United States: the value of prevalence surveys. Clin Infect Dis 2009 05/15;48(10):1434-1440.
Why does it matter?
3Cs: Consistency, comparisons and confidence
Low sensitivity (false negatives, or underreporting) of HAIs is a frequently encountered problem in HAI surveillance systems.
Low specificity (false positives, or over reporting) is usually less of a problem
Both may be related to one or more of following factors:
Difficulty in confirming the case definition of an infection if signs and symptoms were not well documented in the patient’s records
If diagnostic tests included in the case definition of a particular HAI type were not done
Non compliance with the definition of the key term ‘healthcare-associated’: even if the case definition of an infection is matched due to cultural or financial/ political incentives and disincentives at a hospital or country level
Prevalence validation
The case for validation in multi-country studies
In order to investigate variation between countries the first question to ask is: is it the data? Validity Reliability
Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.
Validation results for PPS 2012
Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.
Interpretation
The large differences observed between HAI prevalence
across Member States are in reality smaller The overall weighted HAI prevalence of 5.7% is likely to be a
slight underestimate. .
International PPS validation
Indonesia: – Significant differences were noted between the teams of data collectors
in terms of:
completeness of data, identifying patients who had undergone surgery in the previous month, and most importantly in the number of HAI detected.
SSI, UTI and septicaemia (p=0.01). (κ) did not exceed 0.60 and 0.59 respectively for any infection type.
– Ascertainment was affected by underreporting in medical records, the retrospective nature of data collection for validation purposes, and suboptimal adherence to the PPS protocol and case definitions.
Ref: Duerink DO, Roeshadi D, Wahjono H, Lestari ES, Hadi U, Wille JC, et al. Surveillance of healthcare-associated infections in Indonesian hospitals. J Hosp Infect 2006 02;62(2):219-229
Challenges with undertaking validation remain…. No internationally agreed standard/ protocol?
– ECDC now have a validation protocol for use in EU for national validation
Practical issues?
– Time/ funding – Identifying an external gold standard /independent of the
primary data collection
National and international validation
National validation: – Required for interpretation of HAI prevalence + burden
estimates – At the same time as the primary PPS – Recommended: re-examine 750 patients in 25 hospitals – Minimum: 250 patients in 5 hospitals – Support contracts with ECDC (10 000 EUR / country),
budget to be spread over (at least) 2 years (2016-2017) International validation:
– Validation of national validation teams (VT) – Accompany national VTs in 1-2 hospitals/country – Who?: Part of HAI-Net support call for tender (published
soon): contractors + ECDC experts
How can ECDC help?
Contracts for validation- financial support, feedback of
national validation results Contracts for International validation
Summary
Validation is a key component of surveillance for comparisons, consistency and confidence Without it we do not know the true prevalence of HAI Without it we cannot investigate reasons for variation
in HAI prevalence between hospitals and/ or countries
Knowing the true burden makes the case for infection prevention and control measures and enables improvement in HAI