Upload
ahmed-refat
View
1.500
Download
4
Embed Size (px)
DESCRIPTION
health system assessment by using indicators
Citation preview
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
1
Evaluation & Monitoring of
Health Systems
Dr. Ahmed-Refat AG Refat
12/12/2012
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
2
Evaluation Vs Monitoring
Evaluation is:
“The systematic gathering, analysis and reporting
of data about a system*/program** to assist in
decision making.”
*A “system” a set of inter-connected parts that have to function together to be effective
A health system consists of all organizations, people and actions whose primary intent is to
promote, restore or maintain health (WHO 2007)
** Program is a group of related activities intended to achieve specific outcomes.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
3
When Is Program Evaluation Desirable?
Retrospective Evaluation - is often used when
programs have been functioning for some time.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
4
Prospective Evaluation.
is conducted when a new program is being introduced
or when a major changes introduced
A prospective evaluation identifies ways to increase
the impact of a program on clients; it examines and
describes a program’s attributes; and, it identifies how
to improve delivery mechanisms to be more efficient
and less costly.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
5
Formative Vs Summative Evaluation
The broadest and most common classification of
evaluation identifies two kinds of evaluation:
1. formative evaluation. =
evaluation of components of a program other than
their outcomes.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
6
A formative evaluation may evaluate the degree of need
for the program, or the activities used by the program to
achieve its desirable outcomes, but without evaluating
the degree of outcome.
Formative evaluation is analogous to Total Quality Management (TQM) and Continuous Quality Improvement (CQI) since all these
approaches are a commitment to constantly improve operations, processes and activities to meet client requirements efficiently,
consistently and cost-effectively. Formative evaluation then evaluates whether the program will use, or does use, the right mix and volumes of human resources, materials and activities to carry out the program.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
7
2. summative evaluation=
evaluation of the degree to which a program has
achieved its desired outcomes, and the degree to which
any other outcomes (positive or negative) have resulted
from the program.
Summative evaluations examine the changes that should or
did occur as a result of the program(outcomes).
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
8
Evaluation
Prospevtive – Retrospective
• prospective, meaning it determines what ought to
happen (and why); or
• retrospective, meaning it determines what actually
happened (and why).
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
9
Evaluation Internal And External
1. Internal evaluation (sometimes called self
evaluation), in which people within a program
sponsor, conduct and control the evaluation. Internal
evaluation can more fully engage the insights of
program personnel but runs the risk of overly
subjective evaluation results.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
11
Evaluation Internal And External
2. External evaluation, in which someone from
Outside the program. External evaluation has the
advantage of objectivity if done
Well.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
11
Evaluation
Descriptive Analytical
Descriptive elements of the evaluation are meant to
answer four of the questions that are the hallmark of
good journalism, just as they are the hallmark of good
descriptive evaluation: WHO WHAT WHEN WHERE
Description alone does not answer WHY & HOW.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
12
The why and how questions that are answered by
analytical evaluation.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
13
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
14
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
15
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
16
Monitoring
Monitoring is the constant or
recurring collection and examination
of selected information on program
activity over the life of the program.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
17
Monitoring Information can be used for two purposes:
1. to alert the program to changes in program operation
that might be signals of possible program failure; and
2. to provide a body of information that will be used
when each kind of evaluation is carried out.
Monitoring can emerge from prospective evaluations,
and can provide raw material for retrospective
evaluations.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
18
The Difference between Outputs and Outcomes
An output is a measurable result of activities within
a program, reflecting the immediate result of the
activities but not directly reflecting the effect on
clients of the program.
An outcome is a measurable positive or negative
change to clients of a program or to other stakeholders.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
19
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
21
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
21
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
22
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
23
Performance Indicators
in
Evaluation and Monitoring
of Health System
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
24
Indicators of access
Definition of Access
“access” refers to the presence or absence of physical
or economic barriers that people might face in using
health services.
Physical barriers are usually interpreted to mean
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
25
those related to the general supply and availability of
health services and distance from health facilities.
Economic barriers are usually interpreted to mean
those related to the cost of seeking and obtaining
health care, in relation to a patient’s or household’s
income. Many of the features of “access” are also
included in definitions of the structural aspects of quality
of care assessments.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
26
Physical Access
! Percent of (rural, poor) population residing within X kilometers of a health facility
! Percent of (rural, poor) population residing within X kilometers of a health facility
providing a package of basic health services
! Percent of (rural, poor) population residing within X kilometers of a health facility staffed by a doctor
! Percent of (rural, poor) population residing within X kilometers of a pharmacy
Percent of the population residing within X kilometers of a hospital
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
27
! Percent of population residing within X kilometers of a hospital providing 24-hour emergency (obstetric) care
! Percent of population served by 24-hour ambulance services
! Percent of health facilities equipped with telephones or radios
! Population per doctor
! Population per nurse
! Population per hospital bed
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
28
Economic Access
! Average total private cost of professionally treated outpatient illness episode as a percent of monthly per capita household income for consumers in the lowest per capita income quintile
! Average total private cost of medicines for a typical outpatient illness episode as a percent of monthly per capita household income for consumers in the lowest per capita quintile
! Average total private cost of an average hospital stay as a percent of annual per capita household income for
consumers in the lowest per capita income quintile
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
29
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
31
Indicators of equity
“The concept of equity as it relates to
health systems may refer variously to
differences in health status, utilisation,
or access among different income,
socio-economic, demographic, ethnic,
and/or gender groups”
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
31
Equity can be 1. horizontal and 2. vertical equity.
Horizontal equity refers to the equal treatment of
equals regardless of gender, marital status and so on.
Vertical equity is based on the principle that individuals
who are unequal in society should be treated differently
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
32
Definition of Equity
The concept of equity as it relates to health systems may
refer variously to differences in health status, utilization, or
access among different income, socio-economic,
demographic, ethnic, and/or gender groups.
Most health sector reform efforts directed toward system
performance in low- and middle-income countries
concentrate on definitions of equity related to access and
utilization.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
33
If equity is defined mainly in terms of ensuring access in
general—or universal access to a package of basic or cost-
effective health services in particular—
Access can be modified and calculated across the
population, or, using coverage rates that compare access
for different income, to socio-economic, demographic,
ethnic, gender, geographic, or other groups considered
especially undeserved.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
34
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
35
Indicators of Quality
Definition of Quality
“Quality of care” is clearly a multi-dimensional concept
and one on which there is as yet no consensus
definition (even in the United States).
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
36
Quality of care “that kind of care which is expected to maximize
an inclusive measure of patient welfare, after one
has taken account of the balance of expected
gains and losses that attend the process of care
in all its parts.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
37
Quality of care is a multi-dimensional concept on which there is
little current consensus.
Three aspects of quality of care are commonly distinguished in
the literature, between ‘structural’, ‘process’ and ‘outcome’
dimensions .
‘Structural’ quality refers to whether appropriate resources
are in place to provide health care of a minimum standard
(personnel trained for their tasks, well maintained equipment and
buildings, a regular drug supply).
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
38
‘Process’ quality generally refers to activities occurring
during the interaction between the health system and the client
(i.e. whether good quality
care is actually delivered). Also, both the presence of
accreditation and the quality of professional training and
continuing education may affect quality.
‘Outcome’ quality, in addition to health status, can include
patient satisfaction and perceived quality. Outcome flows from process and refers to “changes in a patient’s
current and future health status that can be attributed to the
antecedent health care.”
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
39
Structural Indicators
! Existence of national standards for professional qualifications of health manpower, including enforcement mechanisms ! Proportion of health workers possessing basic professional qualifications, including skills for specific primary health care services ! Existence of national facility standards, including enforcement mechanisms ! Proportion of health facilities meeting basic structural standards, based on the services to be provided ! Presence of clear national standards for high priority health services
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
41
! Proportion of facilities in which current diagnostic and treatment guidelines are available in writing ! Presence of a national quality assurance program, including trained staff and established procedures for quality design, monitoring and improvement. Sub-indicators of the presence of effective quality assurance include: > Routine review and updating of technical standards > Effective methods for communicating standards to the field level > Routine application of methods for comparing performance with standards > Routine application of data-based problem-solving methodologies
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
41
> Routine application of methods for incorporating community input into system design and management ! Proportion of health facilities that did not experience drug stockouts during the preceding three months ! Proportion of cases in which all recommended drugs are available
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
42
Process Indicators
! Proportion of clinics in which services are fully integrated, per national standards ! Proportion of health workers receiving appropriately timed and effectively conducted supervision, per national policy ! Proportion of patient contacts in which treatment received is consistent with national diagnostic and treatment protocols, including guidelines for client-provider interaction ! Proportion of referrals made and consummated in accordance with national guidelines and standards
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
43
! Proportion of clients who know and understand essential actions needed to complete treatments and avoid future preventable conditions ! Proportion of clients who follow through to completion of recommended treatments (drop out rate) ! Client satisfaction Indicators for Health System Performance 23
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
44
Indicators of efficiency
The important distinction between effectiveness and efficiency is
that the latter takes into account costs.
Efficiency is defined by health economists in several different
ways, and can be applied to health services as well as health
outcomes. Efficiency essentially concerns how and which
health services are produced, and has three dimensions:
technical, economic8 and allocative .
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
45
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
46
Indicators of Efficiency
! Outpatient visits per hour of physician labor (or per physician) ! Outpatient visits per hour of nurse labor (or per nurse) ! Ratio of outpatient visits to personnel costs (i.e., unit personnel costs) ! Cost per outpatient visit (or Operating cost per outpatient visit) ! Cost per hospital bed-day (or per hospital admission or per hospital discharge) ! Percent of outpatient visits obtained from the private sector ! Private hospital beds as percent of total
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
47
! Personnel expenditure as a percent of total recurrent health expenditure ! Expenditure on drugs and supplies as a percent of total recurrent health expenditure ! Number of nurses per doctor ! Number of nurses per hospital bed ! Number of doctors per hospital bed ! Ratio of average salary of government health worker (e.g., doctor, nurse, technician) with a given level of experience to the income of a comparable private sector health worker ! Salaries of government health workers are paid on time (yes/no)
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
48
! Adequate performance incentives exist for government health personnel (yes/no) ! Generic drug expenditure as a percent of total drug expenditure ! Government health system uses basic drug list for procurement (yes/no) ! Percent of government recurrent health budget spent on public health services ! Primary health care expenditure as a percent of recurrent costs ! Percent of total government drug expenditures allocated to primary care facilities ! Fees are charged in all facilities (yes/no)
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
49
! Fee levels promote efficiency (yes/no) ! Referral system functions effectively (yes/no) ! Average length of hospital inpatient stay ! Hospital bed occupancy rate ! Percent of insured enrolled in plans which use copayments and deductibles, managed care plans,
or plans subject to global budgeting
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
51
Sustainability
the capacity of the system to continue its normal activities successfully in the
future
Most definitions of sustainability also include the additional requirement that the system be able to
expand its activities as needed to keep up with increases in demand due to economic and
population growth.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
51
Sustainability includes both financial and
institutional dimensions.
Financial sustainability refers to the capacity of the
health system to replace withdrawn donor funds with funds from other, usually domestic, sources. Institutional sustainability refers to the capacity of the
system, if suitably financed, to assemble and manage the necessary non-financial resources to successfully carry on its normal activities.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
52
Indicators of Sustainability
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
53
Financial Sustainability
! Percent of total health system financed by tax revenue
! Percent of government health system financed by tax
revenue
! Government health expenditure as percent of total
government budget
! Government health expenditure as percent of GDP
! Percent of total health spending financed by donors
! Percent of government health spending financed by donors
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
54
! Percent of government recurrent health spending financed
by donors
! Percent of government health expenditure directed to
primary care
! Percent of government health expenditure directed to
preventive care
! Percent of government health expenditure directed to MCH
services
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
55
Institutional Sustainability
! Foreign doctors as a percent of all doctors
! Number of months of foreign technical assistance funded
by donors
! Donor expenditures on technical assistance as a percent of
all donor health expenditures
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
56
Key characteristics
of
good service delivery
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
57
Key characteristics of good service delivery
.Comprehensiveness: A comprehensive
range of health services is provided,
appropriate to the needs of the target
population, including preventative, curative,
palliative and rehabilitative services and
health promotion activities.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
58
Key characteristics of good service delivery
.Accessibility: Services are directly and
permanently accessible with no undue barriers of cost, language, culture, or geography. Health services are close to the people, with a routine point of entry to the service network at primary care level (not at the specialist or hospital level). Services may be provided in the home, the community, the workplace, or health
facilities as appropriate .
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
59
Key characteristics of good service delivery
.Coverage:
Service delivery is designed so that all
people in a defined target population are
covered, i.e. the sick and the healthy, all
income groups and all social groups .
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
61
Key characteristics of good service delivery
.Continuity
Service delivery is organized to provide an individual with continuity of care across the network of services, health conditions, levels of care, and over the life-cycle.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
61
Key characteristics of good service delivery
.Quality:
Health services are of high quality, i.e. they are effective, safe, centered on the
patient’s needs and given in a timely fashion.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
62
Key characteristics of good service delivery
.Person-centredness:
Services are organized around the person, not the disease or the financing. Users perceive health services
to be responsive and acceptable to them. There is participation from the target population in service
delivery design and assessment. People are partners in
their own health care.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
63
Key characteristics of good service delivery
Coordination:
Local area health service networks are actively coordinated, across types of provider, types of care, levels of service delivery, and for both routine and
emergency preparedness. The patient’s primary care provider facilitates the route through the needed
services, and works in collaboration with other levels and types of provider. Coordination also takes place with other sectors (e.g. social services) and partners
(e.g. community organizations).
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
64
Criteria for
Evaluating Individual Indicators
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
65
The indicators should be selected according to the following criteria:
! Validity. Does it measure what it is supposed to measure
Precision. Is the indicator clearly and unambiguously defined?
! Reliability. Will two measurements of the indicator for the same health system produce
the same result? ! Timeliness. Is the indicator available on an annual
basis and without undue delay?
! Comparability. Can the indicator be used to compare health systems meaningfully across countries?
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
66
! Additivity. Can the indicator be readily and meaningfully applied to sub-regions and to population
sub-groups (e.g., gender, income)?
! Interpretability. Does a higher (or lower) value of the indicator consistently imply that a health system performs
better?
! Cost. Is the cost manageable? There is often an unavoidable tradeoff between cost, on the one hand, and
validity, reliability, and timeliness, on the other hand.
Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed
67
Cited References
Knowles, James C.,Charlotte Leighton, and Wayne Stinson, 1997. Measuring Results of Health
Sector Reform for System Performance: A Handbook of Indicators. Special Initiatives Report
No. 1. Bethesda, MD: Partnerships for Health Reform, Abt Associates Inc
Monitoring and evaluation of health systems strengthening An operational framework WHO, Geneva. October
2010
Margaret El. Kruk and Lynn P. Freedman.Assessing health system performance in developing countries: A
review of the literature. Health Policy 85 (2008) 263–276
http://www.ppmrn.net/storage/ppmrn/Margaret%20Elizabeth%20Kruk%20Lynn%20P.%20Freedman.pdf
.edu/index.cfm/go/viewCourse/course/HSRE/coursePage/lectureNoteshttp://ocw.jhsph/
http://www.healthsystems2020.org/content/resource/detail/528/
-for-measurement-are/partners/observatory/studies/performance-we-http://www.euro.who.int/en/who
prospects-and-hallengesc-experiences,-improvement-system-health
are/partners/observatory/studies-we-http://www.euro.who.int/en/who