Upload
luke-harmon
View
216
Download
0
Embed Size (px)
Citation preview
Quality Management of Case Surveillance Systems
Session Overview We will:
Discuss what defines Quality when considering case surveillance
Articulate what value Quality brings to case surveillance
Consider how Quality can be assessed, assured, and improved upon
Quality Management of Case Surveillance Systems
Quality Management typically entails:
But wait… Why are we talking about Surveillance and M&E?
…. aren't they different things?
M&E vs. Surveillance
M&E Surveillance Aggregate counts Measure of criteria Measure vs. criteria Shows service/process
trends over time
Number of people tested Number of HIV+ tests Number of people on
ART
Unique, unduplicated counts Measure of disease burden Measure of influencing factors Shows disease-specific trends
over time
Number of unique people w/HIV Distribution of cases Factors influencing transmission
Number of unique people on ART Trends in ART access outcomes
But wait… Why are we talking about Surveillance and M&E?
…. aren't they different things?
Yes….. But…… How else will you know if your surveillance system gives you quality data?
M&E of Surveillance Measure of criteria Measure vs. criteria Shows service/process trends over time
Do we have a high-quality system? How is our surveillance system working? How can we improve the surveillance system?
Do we have high-quality data? Are the data reliable? Should we trust our data? How can we improve our data?
What is Quality, with respect to Case Surveillance?
Degree of Excellence of …What?
What is Quality, with respect to Case Surveillance?
Degree of Excellence of …What?
Yes!
Why Monitor and Evaluate a Case Surveillance System?
Success +
Change
Quality+
Confidence
Why is Quality Important for Case Surveillance?
Being able to speak to the quality of Collection- the data collected Collation- the process used to collect,
clean and manage the data Analysis- the trends, and findings
presented Interpretation- the action items
suggested Dissemination- the data given to key
partners
Will increase your confidence in the data and the process, and that will then better guide how others will use and react to the data.
Accuracy of data
Accuracy of
assumptions
Accuracy of action
taken
When should a quality management process be implemented?
Quality Management is a continuous process It should be planned early on It should begin the moment a program is initiated It should be routine and ongoing Its findings should be integrated into an adapting
and evolving program
How to Monitor and Evaluate a Case Surveillance System?
How to Monitor and Evaluate a Case Surveillance System
A. Define criteria of interest What is it that you want to know that will
instill confidence in your system and your data? Are all expected cases reported? Have all expected sites reported? Are all (key) fields entered on the case report
form? Are all fields entered correctly? Are the data accurate? Is reporting timely?
A. Define criteria of interestSample quality criteria for case surveillance
Influence Data Quality
Influenced by Process
A. Define criteria of interest Related to the Process
Process Monitoring will allow you to understand: What is working well What are the gaps
in the
Data collection process Data reporting process Data cleaning process Data utilization process
Success or limitation in this area will influence data quality
A. Define criteria of interest Related to the Data
This will allow you to understand: What is working well What are the gaps
in the
Data collection process Data reporting process Data cleaning process
Support and technical assistance in this area will allow you to have stronger confidence in using the data
B. Set measures by which to assess How can you measure and perceive change in your
defined criteria? What are your objectives/standards?
B. Set measures by which to assess ProcessProcess Monitoring should be implemented in a routine manner
Is typically observational Site level Intermediate level National level
Criteria/measures will be defined by the Standard Operating Procedures (SOP)
What is the gold standard for data collection and transmission?
What is actually happening?
B. Set measures by which to assess Process
How are patient level data gathered? Self report? Proxy? Any verification? Ideally, patient-level data should be:
Self reported (not assumed or by proxy) Validated (with identification, with questions for
clarification) Inaccurate data, pseudonyms render case matching difficult.
How are case data transcribed? How many people touch each case? How many times
are data copied? Any double entry or validation? Ideally, data should be:
Copied or transcribed minimally (source is best!) Validated if copied or transcribed
Errors of transcription create inaccurate data.
B. Set measures by which to assess Process
How are cases reported? Is responsibility clear? Are timelines clear? Is there
supervision? Any validation? Ideally, there should be:
Clearly defined roles and responsibilities Clear timelines Routine supervision that includes validation
If cases are not reported in a routine and timely way, data will be incomplete.
How are data managed? Are data cleaned? Are cases matched or de-duplicated? At
what level does this occur? How many people are responsible for this?
Ideally, data should be regularly: Cleaned and deduplicate by few, but consistent people Followed-up upon if there are questions
If cleaning is not routine, and feedback given, data will continue to be poor.
B. Set measures by which to assessData - Representativeness Representativeness describes your ability to
accurately describe the disease over time and by population and place Are all expected cases a part of the national data
set? If 100 HIV+ tests were performed at a site, 100 HIV
case report forms should be completed and submitted
Are all sites, types of service, regions represented? If there are 100 testing/treatment sites, are all 100
reporting cases on a regular basis? Examples:•90% of cases from the last quarter were reported •95% of eligible sites reported in the last quarter
B. Set measures by which to assessData - Representativeness
A) Calculating Representativeness of reported casesAre all cases that you expect to be in the system in the system?
Overall
By Region, Site, Network, etc.
# of reported cases (in time period)# of expected cases (in time period)
X 100
# of reported cases (in time period)# of expected cases (in time period)X 100
This could also be called system completeness
B. Set measures by which to assessData - Representativeness
B) Calculating Representativeness of sites reporting casesAre all sites that you expect to report to the system reporting?
Overall
By Region, Site, Network, etc.
# of sites reporting cases# of expected sites
X 100
# of sites reporting cases# of expected sites X 100
B. Set measures by which to assess Data - Completeness Completeness describes what proportion of an
expected count is ‘there’.
Are all expected data fields are complete? What proportion of fields on the case report form are
complete? What proportion of cases are reported with the (X) case-
defining variables complete?
Examples:•95% of reported cases have all case defining variables complete•95% of reported cases have Date of HIV Diagnosis complete
B. Set measures by which to assessData - Completeness
B. Set measures by which to assessData - Timeliness Timeliness is a measure of the speed between
steps in the process - time from diagnosis to reporting Are the cases submitted and processed within a
reasonable time? Are you presenting this year’s data, or five years
ago?
Example:•85% of cases are reported within three months of diagnosis
B. Set measures by which to assessData - Timeliness Can be measured two ways
A) Median time between diagnosis and receipt of case report at MOH (or other authority)
Calculated by subtracting the months between the date of report and the date of diagnosis.
Reporting delay values can be summed and the median calculated
B) Proportion of cases reported in a specified time period (some standards use 6 and 12 months )
Number of cases diagnosed within a year and reported within 6 months of diagnosis
Number of cases diagnosed and reported for that diagnosis year
Before measuring timeliness, determine representativeness of reportingIncomplete reporting will overestimate timeliness
B. Set measures by which to assessData - Accuracy Accuracy describes the degree to which your data
are ‘true’. Your measure of accuracy will help you establish a degree
of confidence (or not!) in your data
Data Accuracy is influenced by: Collection of data from patient to forms
Does the patient tell you the correct data? Does the counselor record the correct data with correct
spelling? Does the counselor ask all questions (rather than guessing)?
To assess the accuracy of data reported or collected from the patient, you will need to use process/observational methods
B. Set measures by which to assessData - Validity Validity describes the degree to which your data
are ‘true’. Your measure of validity will help you establish a degree of
confidence (or not!) in your data Are males pregnant? Are people 120 years old? Were people diagnosed with HIV in 1958? Do people have negative CD4 values?
Initial data checks can be done on data sets to look for anomalies
Data Validity is influenced by the: Transcription of data from one form to another
Spelling, number sequence, correct code/box Paper to paper and/or paper to computerExample:•95% of reported cases have a valid Date of Birth reported
B. Set measures by which to assessData - Validity
C. Gather Data Data collection should be regular, routine, and ‘low
barrier’ Create templates and checklists Train and support site-level staff to do routine
monitoring and reporting Use regional/national staff to do mentoring and
supportive supervision of the process, and periodic monitoring
Start soon and start small Pilot your M&E process and measures See if you are able to find the answers you are looking
for Modify as needed
C. Gather Data - Design a Monitoring and Evaluation Process Define the Details
How will the M&E Process be implemented?
This is your M&E Plan Brief description of the project and the evaluation
framework Detailed description of the indicators Data collection plan
Description of the data sources Description of data collection tools
Plan for how monitoring and evaluation will occur Dissemination and utilization plan for information
gathered Description of M&E plan update process
SOP
C. Gather Data Local Monitoring Methods
Observation/supervision
Data validation Selection of 5% of records Can use re-abstraction methods
Data double entry
Data triangulation Registers + Case Report + Patient Record + Lab
Record
C. Gather Data Regional/National Monitoring Methods
Site visits/observation/supervision
Assistance with data validation
Support for data triangulation National M&E Reports + Case Report Data + Other
Aggregate Counts
Feedback!!!
D. Summarize findings With respect to your defined criteria and
measures Use simple tools and outlines/templates to do this
Do this initially to document a baseline so you can measure/assess improvement
Do this frequently to ‘test your hypotheses’ (step E. Act on the Findings) and see if a change is worth it
Share your findings with all involved staff at all levels This can motivate them
This can validate their effort
D. Summarize findings Use Data Locally; Regionally
Share findings and empower a clinical site to:
Improve the quality of surveillance activities Quality and timeliness of data and reporting Quality of patient interactions to capture complete data
Analyze and use surveillance data to improve clinical outcomes Look at the clinical indicators collected; is this aligned
with standards of care?
Better understand their local epidemic Who needs to be reached? Do programs need to be altered to do so?
D. Summarize findings Use Data Regionally; Nationally
Share findings and empower national/regional programs to:
Propose and/or implement national program changes More/different investment in surveillance More/different investment in HIV program activities
Clinical outcomes Prevention for high-risk populations Referral programs
Propose and/or implement national policy changes Case reporting mandate Revised clinical guidelines Anti-discrimination policies
E. Act on the findingsExamples of Quality Improvement
Issue Improvement Result
Hand delivery of paper case report form from site to National level means six-month delay - Poor timeliness - Poor representativeness
Implementation of electronic interface (like:) - 3G cards - Excel-based - Scan and email
More timely and representative submission of case report data
Hand-written completion of case report form with low legibility and high spelling errors - Poor validity
Semi-automation of case report form (drop-down menus, standard spelling)
More accurate and valid submission of case report data; fewer data entry errors
Single case report from completed by 4+ staff: intake; counselor; nurse or MD; lab technician - Incomplete data - Poor validity
Single case report from used by one-to-two staff; multiple forms linked by unique ID
More complete data received (number of cases; volume of data)
E. Act on the findingsSimple Quality Improvement process
E. Act on the findings Step 1: Identify Issue/Area for Improvement
What is the issue that concerns you? Break it down into small components
Which components can be modified? Select one component and plan/implement a test of
change
Example1 2 3 4 5 6 7 8 9 10 11 12
Total sites MESI (M&E sites) % MESI (SE sites) % HIV site (HASS) % HIV+ (M&E MESI) HIV+ (HASS) % Unique (HASS) %ARTIBONITE 24 18 75% 12 50% 13 54% 4,323 3,298 76% 2,667 19%CENTRE 17 9 53% 4 24% 7 41% 1,900 1,365 72% 978 28%GRANDE ANSE 16 6 38% 6 38% 6 38% 943 957 101% 644 33%NIPPES 4 4 100% 3 75% 4 100% 996 1,050 105% 783 25%NORD 15 14 93% 14 93% 13 87% 3,635 3,675 101% 3,079 16%NORD EST 8 5 63% 5 63% 5 63% 1,009 945 94% 698 26%NORD OUEST 9 7 78% 5 56% 5 56% 1,352 1,383 102% 1,142 17%OUEST 85 82 96% 43 51% 42 49% 17,204 15,966 93% 11,104 30%SUD 25 20 80% 14 56% 14 56% 2,314 2,283 99% 1,909 16%SUD EST 9 8 89% 4 44% 4 44% 605 625 103% 589 6%(blank) 13 81 742 916% 671 10%
212 173 82% 110 52% 126 59% 34,362 32,289 94% 24,264 25%
Not all sites are reporting casesGrand Anse is the most challenged
E. Act on the findings Step 2: Measure performance
Define and Measure Who What Where When How
Set an objective for what you would like to see
Example1 2 3 4 5 6 7 8 9 10 11 12
Total sites MESI (M&E sites) % MESI (SE sites) % HIV site (HASS) % HIV+ (M&E MESI) HIV+ (HASS) % Unique (HASS) %ARTIBONITE 24 18 75% 12 50% 13 54% 4,323 3,298 76% 2,667 19%CENTRE 17 9 53% 4 24% 7 41% 1,900 1,365 72% 978 28%GRANDE ANSE 16 6 38% 6 38% 6 38% 943 957 101% 644 33%NIPPES 4 4 100% 3 75% 4 100% 996 1,050 105% 783 25%NORD 15 14 93% 14 93% 13 87% 3,635 3,675 101% 3,079 16%NORD EST 8 5 63% 5 63% 5 63% 1,009 945 94% 698 26%NORD OUEST 9 7 78% 5 56% 5 56% 1,352 1,383 102% 1,142 17%OUEST 85 82 96% 43 51% 42 49% 17,204 15,966 93% 11,104 30%SUD 25 20 80% 14 56% 14 56% 2,314 2,283 99% 1,909 16%SUD EST 9 8 89% 4 44% 4 44% 605 625 103% 589 6%(blank) 13 81 742 916% 671 10%
212 173 82% 110 52% 126 59% 34,362 32,289 94% 24,264 25%
Only 38% of sites in Grand Anse have reported an expected HIV case to HASS in the last year.
Aim for 50% in one month; 75% in six months
E. Act on the findings Step 3: Brainstorm Cause of Issue
Gather information Go meet with local staff Understand the issue
Collectively come up with strategies for improvement Enable the local staff to come up with solutions Before you act… Consider what is actually in your
control. It is likely more effective to focus on those!
Example Issues
High turnover of nurses
Interruptions in supply of test kits (tests not done)
Case reports out of stock
Power outages
Staff have additional duties
Data collection incomplete
Unclear who is responsible
Possible Solutions Cross-train staff
Increase staff mentorship
Define/assign roles and responsibilities
Create inventory checklist, implement use, and communication of needs
Procure battery back-up
Plan, Do, Study, Act Circle
PLAN
DO
STUDY
ACT
E. Act on the findings Step 4: Generate and Plan Improvement Ideas Select ONE area and ONE element
Brainstorm: What is one thing we can do on one day to see
improvement? Who do we need to train/support to make this one
change? How will we measure if this creates an improvement?
Hints: Start small Start local Start quickly
E. Act on the findings Step 5: Implement Change
For the ONE area and ONE element
Support Implementation: Take a baseline measure Train/orient the key staff Implement the change for one hour/one day Measure the result
Hints: Start quickly Don’t over think it
E. Act on the findings Step 6: Study Results of Change
For the ONE area and ONE element
Evaluate the result What happened from baseline to post-evaluation? Did you see a change in the direction that you
wanted? What are some possible influencing factors?
Hints: Don’t over think it Use the data There is no failure… you can learn from all outcomes
E. Act on the findings Step 7: Act Accordingly
One – Expected improvement outcome Expand pilot Communicate Incorporate into Standard Operating Procedure
Two – Not quite expected outcome Refine idea Run through PDSA again
Three – No improvement Okay Refine theory Or move onto another process
F. Repeat process frequently Monitoring, evaluation and quality
improvement should be: Routine and ongoing Low-barrier Simple Implemented at multiple levels Implemented regularly by those close to the
source Implemented periodically by those further from
the source Designed to be inspiring, and not punitive
Final/Special Considerations
Staffing for Implementation M&E and quality improvement, should be routine
and well integrated into the surveillance and/or HIV program Consider having a point-person at the national level (25%-
50% FTE) to: Review data Produce reports Provide training and TA as needed
Consider having sub-national point-people (10%-50% FTE) to: Implement site visits (review and validation) Provide training and TA as needed
Consider having site-level point people (10% FTE) to: Implement data review and validation and ensure reporting, Request training and TA as needed
Methods of Implementation As you identify staff that can be involved in the
M&E and quality improvement process, consider: Empowering them Getting them excited to be a part of a positive
initiative Train them… in content and as trainers Encourage them to be SUPPORTIVE and
EMPOWERING, not punitive Enable them to engage in:
Routine and supportive supervisory initiatives Hands-on technical assistance to support improvement Being a part of the solution, not just directing it
Summary Quality is something important to measure in relation to
the process, inputs, and uses of case surveillance system and data.
Quality should be assessed routinely via monitoring, and periodically via point-in-time evaluation.
There are simple criteria that you can use to define how to assess and measure quality – this can be quick and easy.
Quality should be considered at all levels of the system: site level, sub-national level, and national level.
Findings from a quality assessment can (and SHOULD!) be used to improve quality.
Being able to speak to the quality of your data will improve your confidence in your system, data, analysis, and suggestions.
Thank You
Working Together to Plan, Implement, and Use
HIV Surveillance Systems