WHAT IS IMPACT EVALUATION? BRIEF IE DESCRIPTIONS

Kenya Patient Safety Impact Evaluation, KePSIE

The lack of effective systems to regulate, measure, and improve patient safety and the quality of care places severe limits on evidence-based decision making. For instance, frequent calls for more inspections and greater regulation are not backed by evidence on the impact of such measures on patient safety and quality of care, particularly in the private sector. In Kenya (one of the only low-income countries with a national survey on patient safety), only 2% of health facilities in 2012 were compliant with minimum protocols and systems to assure patient safety (IFC and WHO, 2012).

The Kenya Patient Safety Impact Evaluation is a unique partnership with the Kenyan Government that builds on long- term IFC support for regulatory reform in the health sector. Using health markets as the unit of intervention (defined as clusters of health facilities where no facility is more than 4 KM from the center of the market), we will experimentally allocate all 1,100 private and public health facilities in three Kenyan counties–Kakamega, Kilifi and Meru–into one of three arms: (1) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities; (2) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities and public disclosure of inspection results, and (3) “business-as-usual” low-probability inspections (the control group). The results from this study will contribute to understanding the extent to which governance

i2iDIME

TRANSFORM DEVELOPMENT

CHANGING POLICY THROUGH RESEARCH: DIME IE PORTFOLIO IN KENYA

Impact evaluation (IE) uses rigorous experimental and quasi- experimental methods to identify the direct causal impact of a well-defined intervention against a scientifically valid counterfactual. This counterfactual represents what would have happened to program recipients had the program not taken place, or had a different program design been chosen. Thus, IE builds on traditional monitoring and evaluation (M&E) by directly attributing specific changes in outcomes to specific interventions. When undertaken prospectively (i.e. designed and planned prior to the implementation of interventions) and integrated into the operational planning and implementation of a program or policy, IEs are a powerful research and development tool to identify the most effective and cost-effective design options. For this reason, IEs are a management tool for results-based policy planning. At the same time, they are a key instrument for improving aid effectiveness and increasing fiscal accountability.

There are currently seven ongoing DIME impact evaluations in Kenya:

1 Inspections Regimes in Healthcare and their Impact on Patient Safety and Quality of Care in Kenya (P149879)

2 Turning Pipe Dreams into Reality: Understanding the Demand Curve for Sanitation in Nairobi Slums (P144115, IDA16 commitment).

3 Judicial Performance Improvement Project (P105269)

4 HIV Life Skills for In-School Youth (P149162)

5 HIV Sensitization Campaigns for Out-of-school youth (P149168) under the Total War Against HIV/AIDS-TOWA Project (P081712)

6 Nzoia Irrigation Program (P145556, IDA 16 commitment).

7 The Demand for Costs and Socio-Economic Impacts of Rural Electrification in Kenya (P155238).

8 Road Safety in Kenya: Matatu and Insurance Markets (P158664)

i2iDIME

TRANSFORM DEVELOPMENT

Departmentfor InternationalDevelopment from the British people

Impact Kenya

Kenya Impact Evaluations

2 | i2i

and accountability mechanisms can help improve service delivery in low-income countries, particularly in this case related to improving patient safety and quality of care in the public and private sectors, and if so, how these systems may operate when they are implemented “at scale.” Another important contribution of this project is the development of a set of tools and instruments that can be broadly deployed both in surveys of quality and in the design of inspection systems across diverse low-income settings.

Some results of KePSIE to date include:

• Reform to the Regulatory Framework of the Inspection System for Minimum Patient Safety. The new regulatory framework includes launched by the government in 2015 (a) a refined Joint Inspection Health Checklist with itemized patient safety indicators to make it leaner and less discretionary, easier to deploy and further focused on the fundamentals of patient safety; (b) a scoring system that allows facilities to be categorized according to the level of risk presented to patients; (c) warnings and sanctions to be enforced according to a facility’s level of risk. Additionally, it will be extended to public facilities (only private facilities were covered by the previous legislation).

• Design and Validation of Patient Safety and Quality of Care Measures: During 2014 KePSIE developed a set of tools and instruments in close collaboration with the Ministry of Health and the Boards and Councils that can be broadly deployed in diverse low-income settings, both in surveys of quality and in the design of inspection systems. A combination of methods including surveys for health facilities, patients, and health workers and observation (both known and unknown to the health care provider) were used to develop these tools.

• Collection of Baseline Data in All Health Facilities in the Three Study Counties: Between January and September of 2015, KePSIE collected data in all public and private health facilities of Kakamega, Kilifi and Meru to assess patient safety, quality of care, demand and prices,

before the beginning of the intervention. Data collection included around 1,100 health-facility level surveys, 8,000 patient exit surveys, 1,600 health-care worker surveys, and 19,000 observations of healthcare-worker-patient interaction. The scale and scope of this survey makes it the world’s biggest patient safety survey conducted so far.

The team is currently analyzing the results from the baseline data collection, and preparing for the implementation of the inspections regimes, which is planned to start in 2016.

Turning Pipe Dreams into Reality: Understanding the Demand Curve for Sanitation in Nairobi Slums

The overarching question being asked with this evaluation is how to maximize the development impact of large bulk infrastructure investments in water and sanitation. This evaluation has two main objectives. The primary objective is to understand the demand curve for sanitation services. By offering different levels of subsidies, we will be able to learn what the willingness of poor households is to pay for sanitation connections. By doing so we will identify whether financial constraints are the main underlying reason for low take up rates. Additionally, we will study how take-up rates vary by household structure (compound landlord is resident versus non-resident, etc.) and thus understand if collective action issues and coordination failures are also constraining factors for sanitation services demand. The study will explore the fiscal implications for the utility for expanding access to low-income households that may struggle to repay monthly water expenses.

The second objective of this evaluation is to examine the welfare impacts of large infrastructure sanitation services. Different levels of subsidies and the induced variability in sewerage connections that they generate among the population studied, will ultimately be used to assess impacts on welfare. The question of whether urban upgrading improves the lives of those targeted, or shifts low-income households out through increased rental and housing costs will be explored..

Judicial Performance Improvement Project

The primary goal of this impact evaluation is to test alternative implementation strategies for maximizing the effectiveness of the “Performance Management and Measurement Understandings” (PMMUs). These performance contracts were introduced in the Kenyan Judiciary in 2015 as part of larger reform process to improve the Judiciary, which experiences low public trust and confidence. Several assessments of the Judiciary have been conducted and they all point to persistent capacity and integrity problems. These reports described the Judiciary’s failure to deliver timely justice, significant case backlogs, ineffective case management systems, poor conditions of service for its officers and deficient physical infrastructure. The PMMUs are

i2i | 3

Kenya Impact Evaluations

seen as an intervention that can potentially improve these issues. They are signed between the Chief Justice and heads of court divisions, as well as key administrative personnel and then cascaded down to judicial officers and staff. The PMMUs set targets focused on improving the efficiency of case processing and the quality of judicial services.

The IE will test measures that could enhance the impact of the PMMUs on performance and service delivery in the courts, in particular it will test three of such measures: (i) information that illustrates to PMMU signatories how they are performing against their targets and compared to peers; and (ii) complementing the information with calls from management for supervisory accountability, and (iii) sharing this information with court user committees as a means of local accountability. Given the Judiciary’s current emphasis on reducing its backlog of cases, a key secondary concern for the evaluation is to ensure that performance contracts designed to improve the speed of justice do not undermine the quality of case processing.

To date, the IE has scored some success in affecting policy. Indeed, the IE has supported the development of the Daily Court Returns Template, which was recently gazetted as policy. This tool electronically gathers and organizes information on daily court activities, which in turn feeds into a monthly report on how each court is doing. This data will be key for the experiments of the IE and will remain as a policy tool beyond the IE.

HIV Life Skills for In-School Youth , and HIV Sensitization Campaigns for Out-of-school youth, under the Total War Against HIV/AIDS-TOWA project

Young women are disproportionately affected by the HIV/AIDS epidemic. Women age 20–24 are four times more likely to be infected than men of the same age. Gender-based violence, physical or sexual, is common in Kenya particularly

in western Kenya (formally Nyanza) where six in ten women have reportedly suffered from it. Violence against women has serious consequences for their mental and physical well-being, including their reproductive and sexual health.

These two impact evaluations study the effectiveness of two common behavior change communication programs supported by the government of Kenya: A life skills curriculum that targets in-school youth and video screenings that targets out- of-school youth in villages. Both take a multifaceted approach by implementing behavior change interventions in different social contexts and share the objective of reducing risky sexual behavior and gender-based violence among 15–24 year olds in Nyanza province. The baseline survey was collected in 2013. Both the in school and out of school interventions were carried out in 2013 and 2014. The follow up survey is planned for 2016.

Using data and background research for these evaluations, the research team wrote a paper about the determinants of HIV Testing Behavior in Kenya and its implications for ARV scale-up.

Nzoia Irrigation Program

Large per-farmer investments are made in irrigation with the hope that this will move these farming households from subsistence to commercial production. The reality is that infrastructure alone may not be enough to realize this transformation. The evaluation will explore the impact of the Nzoia irrigation scheme on the ability for smallholder farmers to switch to commercial crops and its role in improving food security. Beyond the infrastructure investment the impact evaluation also plans to unpack the role of credit and financial education for farmers. The concept note has been completed and preparation for the baseline survey is ongoing.

The Demand for, Costs, and Socio-Economic Impacts of Rural Electrification in Kenya

This IE will generate new data on the demand for, costs, and social and economic impacts of national grid electrification in Kenya. It will address the following questions:

1. What is the demand for household electricity connections at various levels of prices?

2. What are the economies of scale in costs by connecting neighboring clusters of households at the same time?

3. What are the effects of electricity access on appliance ownership and monthly electricity consumption for newly connected households over time?

4. Impact of modern energy on social and economic outcomes, such as children’s education, health?

5. Impact on small business competition? Evaluation Design: Random assignment at village level

Associated WB Project: Kenya Electricity Modernization Project (P120014)

SmarTTranS: Road Safety in Kenya

Road traffic accidents (RTAs) take a huge economic toll worldwide. Kenya ranks among the top 10 countries in death rate due to RTAs in the world, with 19 deaths per 100,000 inhabitants and more than 420,000 non-fatal injuries per year. Evidence, albeit scarce, shows that matatus (mini-buses, the second most used means for commuting after walking) are involved in a disproportionately large proportion of RTAs.

There is a clear need for policy efforts that encourage safer driving and increased accountability in contexts like Kenya. Existing policies, like the 2004 Michuki Rules in Kenya (e.g., speed governors, seatbelt requirements, set routes, better contract conditions for drivers) have been ineffective in changing the current system, as they aimed to address the consequences, but not the causes, of misaligned incentives and poor accountability.

This project has three primary components aimed at sustaining a system of accountability. First, a measurement framework will be developed for road safety that can be used to organize data and monitor on-road behavior. Second, an ICT-based monitoring system will be built to measure road safety, leveraging the high penetration of mobile phones and low-cost technologies. The technologies will be integrated into a big data system that will provide information to improve decisions by all actors in the urban transit system. Finally, a structured set of field experiments will randomly assign different information and incentives to owners, drivers, and riders to assess the impact of accountability sub-systems (top-down, self-monitoring and bottom-up) on driving behavior.

Kenya Impact Evaluations

IMPACT EVALUATION TEAMKenyan Government, Private Sector and NGOsMinistry of HealthCounty Governments of Kakamega,

Kilifi, and MeruDirectline AssuranceBusara Center for Behavioral EconomicsRural Electrification AgencyThe Kenyan JudiciaryNational Transport and Safety AuthorityNational AIDS Control CouncilMinistry of Water and Irrigation

National Irrigation BoardInnovations for Poverty Action, Kenya

World BankDevelopment Economics Research

GroupTrade and Competitiveness Global

PracticeHealth, Nutrition and Population Global

PracticeWater Global PracticeEnergy and Extractives Global PracticeAfrica RegionTransport and ICT Global Practice

Academic PartnersSwarthmore CollegeUniversity of California, BerkeleyCambridge Mobile TelematicsUniveristy of California, San DiegoMIT Department of Urban Studies and

PlanningCenter for Global DevelopmentUniversity of NairobiKenya Medical Research CenterUniversity of North Carolina at

Chapel HillKenyatta University

i2iDIME

TRANSFORM DEVELOPMENT

Departmentfor InternationalDevelopment from the British people

Impact Kenya