Rachel Steinacher, Research ManagerInnovations for Poverty Action – June 2014

Innovations for Poverty Action –

Generating and Using Evidence

Outline

Introduction to Innovations for Poverty Action What we do and why

Different Types of Evidence Process evaluation Qualitative data Experimental evidence

Why research impact Determining and defining impact

What can we use the results of RCTs for? Example: Primary School Deworming

Conclusion Opportunities at Innovations for Poverty Action

Innovations for Poverty Action Founded in 2002, we partner with academics to

design and evaluate potential solutions to global poverty problems using randomized evaluations.

We also work to mobilize and support these decisionmakers to use this evidence / these solutions to build better programs and policies at scale. 

In collaboration with over 250 leading academics and implementing organizations, we have evidence from over 175 completed studies with over 225 in progress around the world.

Innovations for Poverty Action Our Vision

More Evidence, Less Poverty

Our Mission To discover and

promote effective solutions to poverty problems around the world

Different Types of Evidence

We’ve been referring to ‘evidence’ but what we mean might not be clear as there are many different types of evidence

Types of evidence include (but are not limited to!)…. Process Evaluation Data (Monitoring and

Evaluation) Qualitative Data Experimental Evidence

Process Evaluation

Process evaluation analyzes the extent to which program operations, implementation, and service delivery are being implemented properly. 

When process evaluation is ongoing it is called program monitoring (as in Monitoring and Evaluation: M&E).

Process evaluations help us determine, for example: Whether services and goals are properly aligned Whether services are delivered as intended to the appropriate recipients How well service delivery is organized The effectiveness of program management How efficiently program resources are used

Process evaluations are often used by managers as benchmarks to measure success, for example: the distribution of chlorine tablets is reaching 80% of the intended beneficiaries each week. 

These benchmarks may be set by program managers and sometimes by donors.

Qualitative Data

Qualitative data is data that is captured in a non-numerical way.

Collection can happen in many different ways, including: In-depth interviews

One-on-one interviews with a participant with the purpose of probing the thoughts, ideas, knowledge, etc. of the person being interviewed. These will often be recorded and transcribed for analysis later.

Focus groups Same as an in-depth interview, but includes multiple participants

Direct observation (ethnography, etc.) This differs from interviewing as the person collecting the data does

not actively try to ask questions to the participant about the topic being studied. They simply watch and observe, with the goal of recording and interpreting what they say, their behaviour and attitudes. Sometimes this involves recording (video and/or audio) or writing notes about what they observe for analysis later.

Etc…

Qualitative Data

Qualitative data is useful to answer questions like: Does this intervention ‘make sense’ in this

context? Is this intervention socially, politically, religiously

acceptable? How do people perceive / understand this

intervention and the problem(s) it is meant to solve?

Hypothesis generation: what should we be studying / collecting data on to better understand the intervention or topic we’re interested in studying? Are we asking the right questions?

Experimental Evidence – Why Research Impact? Experimental evidence allows us to determine

impact. By impact, we mean changes / outcomes that have happened because the intervention happened.

This let’s us know what interventions….. Are effective [How effective is it? In what context?

For what demographic(s)? At what cost?] Have no impact (Why wasn’t it effective? Was the

program implemented properly?) Are harmful (Even something that seems like

common sense can have devastating consequences; see the ‘Campbell-Somerville Study’)

Why Research Impact – Influencing Programs and Policy We can use this information to try to

influence the decisions being made by governments, NGOs, and other policymakers / program implementers to make evidence-based decisions

Determining Impact

Different research methods yield different types of evidence…. Some of this evidence

demonstrates the impact of a program

Some evidence cannot be used to demonstrate impact. in fact, it can lead us to believing things that aren’t true, even though the data seems to support a certain conclusion….

Correlation does not imply causation

Evidence that Pirates Prevent Global Warming

Determining Impact

Impact is captured by analyzing a comparison between:

1. the outcome some time after the program has been introduced

2. the outcome at that same point in time had the program not been introduced (the ”counterfactual”)

Impact: What is it?

Time

Prim

ary

Out

com

e

Impact

CounterfactualIntervention

Impact: What is it?

Time

Prim

ary

Out

com

e

Impact

CounterfactualIntervention

Constructing the Counterfactual Counterfactual is often constructed by

selecting a group not affected by the program

Randomized: Use random assignment of the program

to create a control group which mimics the counterfactual.

Non-randomized: Argue that a certain excluded group

mimics the counterfactual.

Counterfactual

The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e. had they not participated in the program)

Problem: Counterfactual cannot be observed

Solution: We need to “mimic” or construct the counterfactual

Non-Random Assignment

HQMonthly income, per capita

1000

500

0 Treatment Control

1457

947

Demonstrating Impact through RCTs

RCTs are by far the most powerful research methodology by allowing us to determine the impact of a program by minimizing the likelihood that the treatment and control groups are meaningfully different from each other

Random Assignment

Monthly income, per capita

1000

500

0Treatment Control

1257 1242

HQ

An example from IPA…

What can we use RCT results for?

Primary School Deworming One of Innovations for Poverty Action’s most famous

studies, that is currently be scaled out across the Kenya

Primary School Deworming Principal Investigators: Michael

Kremer and Edward Miguel Partners:

Investing in Children and their Societies (ICS)

Location: Western Kenya Sample: 30,000+ primary school

children, 6-18 years old Timeline: 1997-2001

Primary School Deworming Policy Issue

Intestinal worms infect more than one in four people worldwide and are particularly prevalent among school-aged children in developing countries.

These intestinal worms are believed to have a negative impact on education, hindering child development as well as school attendance and reducing income later in life.

Primary School Deworming Context of the Evaluation

Busia district is a poor and densely-settled farming region in western Kenya adjacent to Lake Victoria. It has some of the country’s highest worm infection rates.

One quarter of Kenyan student absenteeism is attributed to abdominal pains which likely due to intestinal worm infections. In addition, older children may miss school to take care of siblings who are sick with worm infections

Primary School Deworming Details of the Intervention

This study evaluated the Primary School Deworming Project (PSDP), which was carried out by International Child Support in cooperation with the Busia District Ministry of Health.

The program randomly divided 75 schools into three equal groups which were phased into treatment over three years.

Within each group, a baseline parasitological survey was administered to a random sample of pupils. Schools with worm prevalence over 50% were mass treated with deworming drugs every six months.

Primary School Deworming Details of the Intervention

In addition to medicine, treatment schools received regular public health lectures, wall charts on worm prevention, and training for one designated teacher. The lectures and teacher training provided

information on worm prevention behaviors—including washing hands before meals, wearing shoes and not swimming in fresh water.

Primary School Deworming Results and Policy Lessons:

Impact on Infection Intensity Deworming reduced serious worm

infections by half amongst children in the treatment groups.

Pupils that received treatment reported being sick significantly less often, had lower rates of severe anemia, and showed substantial height gains, averaging 0.5 centimeters.

Primary School Deworming Results and Policy Lessons:

Impact on School Attendance Deworming increased school participation by

at least 7 percentage points, which equates to a one-quarter reduction in school absenteeism.

When younger children were dewormed, they attended school 15 more days per year, while older children attended approximately 10 more school days per year.

Primary School Deworming Results and Policy Lessons:

Treatment Spillover The entire community and those living up to 6

kilometers away from treatment schools benefited from “spillovers” of the deworming treatment. Spillover effects occur because medical treatment

reduces the transmission of infections to other community members.

Reductions in infection in non-treated children resulted in an additional 3 to 4 days of schooling per year.

Primary School Deworming Results and Policy Lessons

Including the spillover benefits of treatment, the cost per additional year of school participation is US$3.27, considerably less than the cost of many alternative methods of increasing primary school participation.

Creation of the ‘National School-based Deworming Program’ and ‘Deworm the World’

National School-Based Deworming Programme

National School Health Programme

Ministry of EducationScience and Technology Ministry of Health

National School-Based Deworming Programme

GOAL:

Treat 5 million children each year for 5 years

TARGET:Treat ALL at risk children

aged 2-14

Primary School Deworming Scaling-up the Program

Additionally, in 2007 the ‘Deworm the World Initiative (DtW)’ was founded. It was managed and operated through Innovations for Poverty Action up until the end of 2013, when it our sister organization, ‘Evidence Action’ spun off with the goal of scaling evidence-based programs – including DtW

To date, DtW’s programs have reached 37 million children in 27 countries

Impact evaluation can and has had a profound impact on the world

In Sum

There are many types of evidence that one can collect and use to better understand an intervention or topic

Experimental evidence generated by RCTs is a powerful tool that we can use to promote evidence-based development work by influencing policymaking and program design.

Innovations for Poverty Action Moving Forward

Innovations for Poverty Action is a rapidly growing organisation with plans to expand in Tanzania (we currently have 1 project here)

We are always looking for: Talented, innovative researchers

to partner with on new research studies

For young researchers, this typically means partnering with a senior researcher to allow for mentoring and expert oversight

Innovations for Poverty Action Moving Forward

We are also looking for bright, eager people interested in working in working on our research projects as: Project Associates (entry-level managers with a

Bachelors degree + 0-2 years experience) Project Managers (early to mid-career professionals

with a Bachelors degree + 2-5 years experience) Research Managers (early to mid-career managers

with a Masters degree + 3-5 years of experience) For more information, visit:

http://www.poverty-action.org/getinvolved/jobs