Managing for the World

M&E: Tool for Internal

Management

MIT Sloan School of Management

Global Health Delivery Lab Team:

Yuko Amizaki, Terry Hu, Shirley Li, Jeongyeon Shim

April, 2011

[MODULE OVERVIEW] M&E: TOOL FOR INTERNAL MANAGEMENT

Many healthcare delivery organizations in resource-constrained setting focus on M&E for donors, collecting and reporting what the donors specify. However, many organizations do not pay much attention on what they need to measure and how they should interpret the data collected for internal management purposes; nor do they scrutinize the data collected over time to track progress.

In one fieldwork example, we have developed a dashboard-style management tool that collect key performance indicators longitudinally to facilitate the use of M&E output in day-to-day management. This module provides a high-level overview of M&E, how they are used by nonprofit healthcare delivery organizations and the challenges they face (especially in the context of the fieldwork organization), how indicators were selected for monthly longitudinal dashboard, and an example of the managerial questions to be asked to use the data for operational improvement.

Note that a summary of open-source EMR systems available is included at the end of the document for organizations that aspire to upgrade their patient data collection system as well.

2

Agenda

• What is Monitoring and Evaluation (M&E)?

• How are M&E outcomes used in nonprofit organizations?

• What are the common challenges in using M&E results in day-to-day operation?

• How can an internal dashboard be created to help utilize data in operation monitoring and improvement?

3

What is monitoring and evaluation (M&E)?

4

Monitoring Evaluation

• Does not provide an exhaustive view of the project or program, as often focuses on what is easily measureable

• Requires significant resource and time investment, limiting the frequency of implementation

Limitation

Definition• “A continuing function that uses systematic

collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds”

• “The process of determining the worth or significance of a development activity, policy or program ….. to determine the relevance of objectives, the efficacy of design and implementation, the efficiency or resource use, and the sustainability of results”

Benefits• Regularly tracks indicators for inputs,

activities, outputs, outcomes and impacts • Facilitates daily management of operation

• Enables an exhaustive and balanced understanding of the project

• Facilitates derivation of lessons from the program and their inclusion in decision-making for different parties including partner and donor

Source: World Bank, http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAPDEV/0,,contentMDK:22293310~enableDHL:True~menuPK:4585753~pagePK:64829573~piPK:64829550~theSitePK:4585673,00.html, accessed March 8, 2011.

Monitoring and evaluation are related but have slightly different focus

5

Timeframe

Monitoring Evaluation

• Continuous, as part of daily operation • Periodic, such as midpoint or end of program review

• Early warning of problems• Generation of corrective action or

alternative tactics

• Feedback to strategy • Generation of alternative policiesValue

Activities• Tracking and documentation of progress• Oversight and ongoing analyses

• In-depth analyses of implication• Comparison of performance against pre-

determined goals

Focus• Implemented activities• Results are achieved• Likely to focus on output and outcome

• Reasons why and mechanism through which the result is achieved

• Organization’s model of change• Focus on impact

• Assessment by internal stakeholders such as program lead, managers, and related stakeholders such as community stakeholders and donors

• In addition to assessment by internal and related stakeholders, can also include assessment by external evaluators

Method

Source: UNICEF, WFP, UNFPA M&A Toolkit

Different types of data collection methods can be used in M&E, depending on the objective of M&E, data availability, and resource availability

6

Use

Quantitative methods Qualitative methods

• Measurement of the frequency and extent of change, by sub-categories of target group

• Qualitative assessment of the reason and mechanism of change

• Reporting of indicators• Standardized interviews or surveys with

closed-ended questions

• Analyses of reported indicators• Free/guided interviews or surveys with

open-ended questions

Examples

Strength• Objective and undisputable data with a

precise “hard number” • Possibility to test statistical correlation

between a problem and potential drivers• Representative of the target population• Possibility of comparisons across different

groups

• Interpretation and understanding of the program background and quantitative data

• Understanding of the psychosocial and behavioral factors

• Relatively low cost given small sample size• Strength in collecting stakeholder feedback

and planning for social change

• Proxy of the impact to be measured, rather than actual measurement

• Limited explanation of reasons and drivers of status quo or change

• Small subset of the target population, limiting representativeness

• Potentially disputable objectivity, given the possibility of bias from those involved in the data collection

Weakness

Source: UNICEF, 1991; Gosling, 1995. USAID TIPS Number 2, 1996; UNFPA M&A Toolkit

Agenda

• What is Monitoring and Evaluation (M&E)?

• How are M&E outcomes used in nonprofit organizations?

• What are the common challenges in using M&E results in day-to-day operation?

• How can an internal dashboard be created to help utilize data in operation monitoring and improvement?

7

M&E programs and their results can be used for multiple objectives

Strategic decision-making• Which strategies warrant more

investment?• How should we revise a specific strategy?• How should future programs be

formulated?

Stakeholder management• Why should a donor, partner, or

other stakeholder be interested in the given program?

• How do stakeholders feel about the program?

• How is the program held accountable to the stakeholders?

Operation management• What is the quality of the project

and how can it be enhanced?• How is the project managed and

how can it be improved??

Organizational impact and learning• What is the impact of the intervention?• What are the lessons learned from the

project?• What components of the strategy were

successful and why?

8

Source: UNFPA M&A Toolkit

Externally, presentations can be used to enhance the organization’s reputation as well as financial and intellectual resources

9

Report to donors

Brochure of organization

Presentation at international conferences

Media

Ask for feedback from experts

Usage

Increase presence among international healthcare

community

Raise follow-up funding

Source: Team analysis

Internally, results of M&E can be used to facilitate organizational development at multiple levels

10

Which areas should we allocate more resources

in the future?

What is our progress against our strategic plans and targets?

How successful are our programs and why are

they successful?

What is the impact of our programs in the community?

What are the areas of our operation that

need improvement?

How efficient are we in using available

resources?

What should be our program objectives

and targets?

Source: Team analysis

Agenda

• What is Monitoring and Evaluation (M&E)?

• How are M&E outcomes used in nonprofit organizations?

• What are the common challenges in using M&E results in day-to-day operation?

• How can an internal dashboard be created to help utilize data in operation monitoring and improvement?

11

Impact on M&E

•A large share of M&E resource devoted to donor-focused areas, e.g., HIVo Close to 10 different reports

generated for HIV operation only

o Over 350 health outcome indicators required, many of which further track by gender and age group breakdowns

o Further breakdown categories inconsistent with one another

•Areas with limited donor interest with limited tracking of results

Organization background

M&E resources are deployed unevenly and often overstretched, especially given differing donor requirements

12

EXAMPLE

Source: Organization interview, Team analysis

Areas of operation

Description

• Operational focus is on all areas of health, including, but not limited to, HIV/AIDS

• Areas of operation also include socioeconomic development to improve health outcome of HIV orphans and vulnerable population

Resource allocation

• Program-specific funding drives resource allocation within the organization

• HIV is a major focus of multiple partners, who also require result and process reporting in specific formats

Not all M&E results are useful in daily operation monitoring, calling for careful selection of indicators (1/2)

13

• Use baseline data to identify program targets and understand their needs

• Improve program design

• Improve program implementation through early identification of gaps and issues

• Articulate and modify program objectives, and assess progress against activities and outcome targets in the plan

• Improve future program content by compiling lessons learned

• Incorporate lessons on program process and management learned from previous phases

Program design

Program implemen-

tationProgram follow-up

Focus of daily operation

Source: Family Health International , Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries; Team analysis

Not all M&E results are useful in daily operation monitoring, calling for careful selection of indicators (2/2)

VCT EXAMPLE

Program output

Service aspects

Outcome

Impact

• % of people aware of the service• # of people counseled and tested• % of people counseled and tested who returned to receive results• % of HIV positives referred to appropriate care• % of tested and counseled who informed their partners

• Adequacy of counseling protocol• Adequacy of testing protocol• Adequacy of counseling and testing actually provided• % of clients counseled who are also tested for HIV• Accessibility of service• Barriers to counseling and testing

• Change in risky behavior of VCT clients and their partners• Change in STI trends in the program area• Reduced stigmatization of people affected by HIV• Increased community support for people living with HIV/AIDS

• Change in trends in HIV/AIDS incidence and prevalence• Reduced mother-to-child transmission in the women targeted• Sustained changes in societal norms in the community

Indicator example

Source: Family Health International , Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries; Team analysis.

14

Pro

cess

ind

icat

ors

Effe

ctiv

en

ess

ind

icat

ors

• Process indicators are more appropriate to measure progress in everyday operation

• Program output indicators are relatively easy to track over time

Agenda

• What is Monitoring and Evaluation (M&E)?

• How are M&E outcomes used in nonprofit organizations?

• What are the common challenges in using M&E results in day-to-day operation?

• How can an internal dashboard be created to help utilize data in operation monitoring and improvement?

15

An organizational dashboard or scorecard can facilitate monitoring of operation and outcome

16

CONCEPTUAL EXAMPLES

Source: “Breaking free of the one-page dashboard rule,” http://www.juiceanalytics.com/writing/breaking-free-one-page-dashboard-rule/, accessed March 2011; “Dashboard vs. Portals: What’s the difference?,” http://simplecomplexity.net/dashboards-vs-portals-whats-the-difference/, accessed March 2011.

Development and use of dashboard is a three-step process

17

•Select indicators which are:o Sensitive enough to show

short-term results and serve as early warning system

o Representative of the focus of activities

o Easily measurable for tracking purpose

•Avoid selecting too many indicators to ensure focus

Select dashboard indicatorsTracking indicators over time

•Track selected indicators longitudinally—e.g., monthly—to easily understand trends in operation

•Show current data alongside historical data to ensure month-to-month comparison as well as to capture any seasonality

Asking appropriate managerial questions

•Understand managerial implications, such as:o Reason for specific

decrease or increaseo Impact of any specific

initiative in the given periodo Expected impact of the

change in the longer term•Develop action plan, such as:oHow to mitigate an adverse

impactoHow to sustain advantage

gained

Source: Team analysis

18

Indicators in the dashboard should capture the health of a given project in the short run

Key process output indicators

• Number of people sensitized on HIV/AIDS per month

• Number of patients counseled per month

• Number of individuals that underwent HIV testing per month

• Number of positive patients per month

• Number of patients receiving palliative care (total)

• Number of patients that went through CD4 count per month

• Number of patients receiving ART (total)

• Number of ART receivers who were monitored per month

Sensitization

Counseling

Testing

Positive results

Palliative care

CD4 count

ART

Monitoring

Exit• Number of ART patients who transferred out per month

A dashboard should include a select number of key output indicators that can show the health of a given project in the short term, e.g., monthly

18

Suggested for dashboard/ longitudinal tracking

HIV/AIDS EXAMPLE

Source: Team analysis

Longitudinal tracking of data can facilitate understanding of performance and uncovering any issues…

19

0

50

100

150

200

250

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

0

5

10

15

20

25

30

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

HIV tested

HIV tested (+)

Received palliative care

0

50

100

150

200

250

300

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

Currently on ART

0

50

100

150

200

250

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

HIV/AIDS EXAMPLE

Source: Team analysis

…And asking the right questions for operational management

20

0

50

100

150

200

250

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

0

5

10

15

20

25

30

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

HIV tested

HIV tested (+)

Received palliative care

0

50

100

150

200

250

300

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

Currently on ART

0

50

100

150

200

250

Jan

Feb

Mar

Ap

r

May Jun

Jul

Au

g

Sep

t

Oct

No

v

Dec

M

F

What explains the surge in testing? Can this be replicated?

HIV/AIDS EXAMPLE

Are we targeting the right population?

How effective was a specific initiative?

What explains the decline despite the increase in HIV+ tests?

What % of newly diagnosed receive palliative care?

What is the reason behind the decrease in ART receivers?

How can we ensure that the population at risk are tested?

What explains the gender difference in the decline? What steps need to be taken to ensure equal access?

Source: Team analysis

Reference

Resources referenced in this document• World Bank. What is Monitoring and Evaluation?

http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAPDEV/0,,contentMDK:22293310~enableDHL:True~menuPK:4585753~pagePK:64829573~piPK:64829550~theSitePK:4585673,00.html. Accessed March 8, 2011.

• World Bank. Monitoring and Evaluation: Some Tools, Methods, and Approaches. 2004.• UNFPA. Programme Manager’s Planning Monitoring & Evaluation Toolkit. 2004.• Rehle, Thomas; Saidel, Tobi; Mills, Stephen; and Magnani, Robert. Evaluating Programs for HIV/AIDS

Prevention and Care in Developing Countries: A Handbook for Program Managers and Decision Makers. Family Health International (year unknown).

Additional resources on M&E• Bertrand, Jane T., and Manuel Solís. Evaluating HIV/AIDS Prevention Projects: A Manual for

Nongovernmental Organizations. MEASURE Evaluation Manual Series, No. 10. Carolina Population Center, University of North Carolina at Chapel Hill. July 2000. (Translation to English 2004).

• International Center for Research on Women. A Measure of Success: Building Monitoring and Evaluation Capacity in Small, Community-Based Programs. 2007.

• New York State Department of Health AIDS Institute. Measuring Clinical Performance: A Guide for HIV Health Care Providers. 2006.

• Avahan India AIDS Initiative. Use It or Lose It: How Avahan Used Data to Shape its HIV Prevention Efforts in India. Bill & Melinda Gates Foundation. 2008.

• New York State Department of Health AIDS Institute. HIVQUAL Workbook: Guide for Quality Improvement in HIV Care. 2006.

21

APPENDIX

22

Potential dashboard/longitudinal indicators: PMTCT and tuberculosis

TuberculosisPMTCT

Key process output indicators

• Number of HIV+ women that have accepted family planning

• Number of HIV+ women using family planning

• Number of HIV+ women who are pregnant

• Share of pregnant HIV+ women who are receiving PMTCT

• Number of HIV+ women giving birth at the health center

• Number of HIV+ women giving birth with traditional birth att’nt

• Number of babies born with HIV and given ART

Family planning

Pregnancy

Delivery

Birth

23

Key process output indicators

• Number of families sensitized in TB

• Number of patients referred for X-ray• % of patients who return with X-ray

Advocacy

TB testing

Positive

X-ray

Treatment

Completion• Number of TB patients completing

treatment• Number of patients cured• Number of patients with resistance

• Number of TB tests carried out• Number of TB tests on HIV patients

• Share of positive TB test results

• Number of TB patients in treatment• Number of HIV+ patients in

treatment• Number of new TB patients entering

treatment

Suggested for dashboard/ longitudinal tracking

24

Potential dashboard/longitudinal indicators: Antenatal care, immunization, and water & sanitation

Antenatal care, immunization, and other Water and sanitation

Antenatal care

Immunization

Malaria

Sanitation education

Latrine

Dish rack

Access to clean water

Key process output indicators Key process output indicators

24

• Number of children receiving immunizationoAt health centeroAt school and outreach

• Number of female s receiving TToNumber of pregnant women

with TToNumber of girls receiving TT

• Number of ITN distributed• Number of mother who received

ITN at birth

• Number of education talks offered• Number of households that

attended the talks

• Number of households with access to latrines

• Number of households actually using latrines

• Number of households with dish racks or equivalent

• Number of households actually using dish racks or equivalent

Suggested for dashboard/ longitudinal tracking

• Number of households with access to clean water

• Number of households using Kyetume’s water facility

• Number of days of interruption at Kyetume’s water facility

• Number of birth given at health center

• Number of birth assisted by traditional health attendants

• Number of children whose weight is monitored at birth

Potential dashboard/longitudinal indicators: Domestic violence and OVC program

* For heifer and goat separately

Domestic violence Orphan & vulnerable children (OVC)

25

Key process output indicators

• Number of change-makers, monthly and cumulativeoRecruited by officeroRecruited by existing

change-makers• Number of outreach and

training participants• Number of circles formed

and member count• Number of domestic

violence incidents reportedo% receiving counseling

from Kyetumeo% receiving counseling

from the police

OVCs

Agriculture

Rep. health education

Orphan support

Counseling /Guidance

Training

Suggested for dashboard/ longitudinal tracking

Key process output indicators

• Total number of youth counseled• Number of schools visited

• Number of HIV+ OVCs• Number of HIV+ OVCs who receive treatment• Number of OVCs that receive direct assistance• Number of OVCs that attend school

• Number of artisan mentors trained• Number of OVCs who completed vocational training• Number of farmers (caregivers) trained• Number of local leaders trained

• Number of beneficiaries starting keyhole gardens

• Number of OVCs counseled• Number of child care workers trained

• Number of households provided with animals*• Number of first female offspring returned*• Average milk yield per animal*• Number of OVC beneficiaries

END OF DOCUMENT

26

Open source tools for data collection on patient level

27

OpenMRS

RyanWhiteCareware

WorldVistA

Description

• Electronic medical records system designed for resource-constrained environment, including clinics

• Designed for flexibility of data and disease type• Developed by Partners in Health and Regenstrief Institute

• Software to manage and monitor HIV care• Automatically produce Ryan White program-specific reports• Developed by US HHS’s Health Resources and Service Admn.

• Developed by US VA to be used in veterans’ health care• Includes functionalities targeting clinics and clinician offices such as

billing, quality measures, and prescription

Sites used

• Over 120 sites including 5 users in Uganda, e.g., ISS Clinic & MakarereUniv.

• 800 Ryan White-funded agencies in the US

• All VA medical centers and outpatient clinics

• Healthcare information system to manage patient visits, medications, and other clinical and administrative tasks

• UnknownPatient OS

Tolven• A platform providing electronic personal health record and clinician

health record solution• Downloadable from the Web

• US, Europe and Asia• Partners include academic medical

centers and pharmaceutical companies

• Mobile device based data collection software (e.g., surveys)• Peter Wakholi, Makerere University , as managing committee

member

• Three sites in Africa and one site in Asia, including Makerere University’s project to track attendance across 100 schools

OpenXData

Source: Internet search, as of March 2011, based on ghd online postings.