S

Evaluation of Surveillance Systems

St Lukes-Roosevelt

Problems with our field

Programs often do more harm than good

Programs don’t collect data, so no benefit shown

The data we do collect is often not useful for improving program quality or guiding policy

How do we show benefit, impact, change?

Surveillance Ongoing

Surveys One point in time

Definition

Public health surveillance is the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event for use in public health action to reduce morbidity and mortality to improve health

*CDC, Atlanta GA

Key concept

Ongoing Action

Why evaluate a surveillance system?

Ensure that problems of public health importance are being monitored efficiently and effectively

Recommendations about the system should focus on improving quality, efficiency, and usefulness

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity Predictive value positive Representativeness Timeliness Stability

What should be evaluated?

System attributes: determine priorities Simplicity: combine a practical structure with ease of use Flexibility Data quality Acceptability Sensitivity Predictive value positive Representativeness Timeliness Stability

Figure 1: Monthly attendance by diagnosis following December drug distribution, Matoumbou Clinic, Kinkala

Dist., R.O.Congo

0

50

100

150

200

250

300

Jan. Feb. Mar. Apr.

Pa

tie

nt

Vis

its Other

Para.

Dia.

ARI

Mal./Fev.

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility: ability to adapt to changing information needs or

operating conditions with minimal time, effort, cost Data quality Acceptability Sensitivity Predictive value positive Representativeness Timeliness Stability

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality: completeness and validity Acceptability Sensitivity Predictive value positive Representativeness Timeliness Stability

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability: willingness of persons or organizations to

participate Sensitivity Predictive value positive Representativeness Timeliness Stability

Contraceptive prevalence rates in Afghanistan, WHO

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity: ability to detect cases OR ability to detect outbreaks Predictive value positive Representativeness Timeliness Stability

Incidence* of Shigella Dysentery

Central Bosnia, 1991-1993

Region Prewar May-July 1993

Sarajevo City 0.3 4.0 (+1250%)

Zenica City 0.3 4.4 (+1690%)

Tuzla Region 0.5 0.4 (-10%)

*Cases per 100,000 per month

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity Predictive value positive: proportion of persons identified as

cases who truly are cases Representativeness Timeliness Stability

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity Predictive value positive Representativeness: system accurately describes events over

time and space (time, person, place) Timeliness Stability

Causes of death, Kenema District, 2000 N=197

Febrile Illness39%

Chronic Conditions

4%

Unknown/Others18%

Trauma/Viol.3%

Bloody Diarrhoea

3%Diarrhoea

12%

Maternal deaths

2%

Miscellaneous Infections

5%

Neonatal deaths

5%

Resp. Infect.9%

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity Predictive value positive Representativeness Timeliness: speed between steps; appropriateness in delays Stability

Epidemic curve, outbreak of mumps, Montreal

Epidemic curve, cholera

What should be evaluated?

System attributes: determine priorities Simplicity Flexibility Data quality Acceptability Sensitivity Predictive value positive Representativeness Timeliness Stability: reliability and availability; resources

Steps in evaluating a surveillance system

Stakeholder engagement

Describe the system: importance, purpose, resources

Focus the evaluation design

**Gather evidence regarding performance

Justify and state conclusions, make recommendations

Malaria Surveillance

Purpose (CDC): (a) identify local transmission; (b) guide prevention recommendations for travelers

Additional benefits (JE) Identify emerging species; treatment failures; local outbreaks

Historically Tracking elimination

Case definition Malaria cases confirmed by blood film, rapid diagnostic tests,

PCR

Malaria Surveillance

The system **National Malaria Surveillance System National Notifiable Disease Surveillance System

(1878 cholera, smallpox, plague, yellow fever at overseas consules)

Direct CDC consultation