agricultural finance - RAFLearning · and agricultural finance 2. Assess the current evidence base for the impact of financial products on smallholder and rural households, particularly

Understanding the impact of rural and agricultural finance on clients

December 2015

Learning Lab Technical Report No. 2

Acknowledgments

The Learning Lab and its operators, GDI and Dalberg, would like to acknowl-edge and thank The MasterCard Foundation for making the Lab and this study possible, and for the input of its Rural and Agricultural Finance team, including Abdoul Karim Coulibaly, Rewa Misra, Paula Tjossem and Lindsay Wallace.

This study was authored by Radoslava Dogandjieva, Laura Goldman, Klaudine Wakasa, and Jason Wendle. It builds on the existing knowledge and research of many experts in rural and agricultural finance, as well as the recently commis-sioned literature review conducted by EPAR’s Leigh Anderson, Chris Clark, Katie Panhorst Harris, Pierre Biscaye, and Mary Kay Gugerty.

The findings and analysis in the pages that follow would not have been possible without the individuals from more than 15 additional organizations who shared data, insights, and perspectives. The authors would like to thank specifically Genzo Yamamoto, Mary Pat McVay, and Isaac Kojo Gyesi at Opportunity Inter-national, as well as Kim Siegal, Kalie Gold and Hailey Tucker at One Acre Fund, who were generous not just with their insights, but also with their support for our team’s fieldwork. Special thanks are also due to external reviewers includ-ing IDinsight's Paul Wang and Jeffery McManus, and CGAP’s Michael Tarazi for their thoughtful feedback on earlier versions of this report.

We would also like to express our appreciation for the contributions of the following individuals: Hedwig Siewertsen at AGRA, James Hokans and Leesa Shrader at Mercy Corps, Matt Forti and Andrew Young at One Acre Fund, Di-anne Calvi and AJ Dot at Village Enterprise, Anushka Ratnayake at MyAgro, Yuli-ya Tarasova and Kulsoom Ally, both formerly at Juhudi Kilimo, Veronica Olazabal and Kristin Lindell at Nuru International, Bethanie Pascutto and Ines Arevalo at the Aga Khan Agency for Microfinance, Jyotsna Puri and Bidisha Barooah at 3ie, Beniamino Savonitto and Pooja Wagh at Innovations for Poverty Action, Carlos Cuevas at the University of Washington, Rohit Gawande at Acumen, Janine Fir-po at the Bill and Melinda Gates Foundation, and Jamie Anderson at CGAP.

3

CONTENTS

ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

ABOUT THIS STUDY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

SECTION I: THEORY OF IMPACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8A general theory of impact for rural and agricultural finance . . . . . . . . . . . . . . . . . . . 8

The customer’s perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Findings from the academic literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

SECTION II: EVIDENCE TO DATE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15Results from practitioner evaluation efforts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Measurement methods observed in the academic literature . . . . . . . . . . . . . . . . . . . 25

SECTION III: MEASUREMENT APPROACHES . . . . . . . . . . . . . . . . . . . .25Practitioner approaches to measurement and evaluation . . . . . . . . . . . . . . . . . . . . . 27

Overcoming challenges to measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

A PATH FORWARD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

ANNEX I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42Sample theories of change related to rural and agricultural finance . . . . . . . . . . . . 42

Initiative for Smallholder Finance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

CARE Pathways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Opportunity International . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

One Acre Fund . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Asian Development Bank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

ANNEX II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45Definitions of products and impact variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

UNDERSTANDING THE IMPACT OF RURAL AND AGRICULTURAL FINANCE ON CLIENTS 4

Product definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Sample metrics of impact used by researchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

ANNEX III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47Pipeline of notable rural and agricultural finance impact studies . . . . . . . . . . . . . . . 47

ANNEX IV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49Participatory research work plan and questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Approach to fieldwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

In-depth interview guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52


ABOUT THIS STUDY

The Rural and Agricultural Finance Learning Lab is a MasterCard Foundation initia tive to foster learning and collaboration that leads to better financial solutions provided to more smallholder farmers and other rural clients. The Lab was launched in 2015 and is run jointly by Dalberg and the Global Development Incubator. Among other activities in support of the Foundation’s Rural and Agricultural Finance (RAF) portfolio and the broader RAF community, the Learning Lab researches key questions that comprise its learning agenda. These questions are centered on impact and measurement, client de-mand, financial solution provision and ecosystem development.

At the impact level, the Learning Lab is interested in how fi nancial solutions contribute to poverty reduction and improved livelihoods for smallholder and rural households. For this, the Lab’s first major study, a Dalberg team looked at this question from theo-retical, empirical, and methodological angles; i.e., how do we think about impact, what do we currently know about impact, and how do we measure impact. The study builds upon the literature review of RAF impact in Africa commissioned by the Lab and con-ducted by the Evans School Policy Analysis and Research Group (EPAR). The EPAR liter-ature review is an important companion to this study, providing additional details on the evidence presented in Section 2, and can be downloaded here: http://www.raflearning.org/post/literature-review-raf-impact-africa

The Lab is committed to actionable and collaborative learning, and we invite the en-gagement of our readers, including feedback on this report, contributions of additional data/information, or questions/suggestions about future areas of study. At our website, www.raflearning.org, users can contact the Lab directly or comment on this or any other publication. We are on Twitter @raflearning, or the Rural and Agricultural Finance pro-fessional group on LinkedIn.

WWW .RAFLEARNING .ORG


sive offering for their clients. This report focuses specifically on RAF provided di-rectly to rural households, or groups of households, rather than to producer or-ganizations or other stakeholders in the agricultural value chain (e.g., agro-pro-cessors, input providers). There is great diversity within RAF offerings; some products and services are customized specifically to agricultural needs, such as input loans or weather-index insurance, while others, such as savings accounts, are suitable for more general customer needs. Annex 2 provides a comprehen-sive list of RAF products.

As significant amounts of funding are being channeled to rural and agricul-tural finance, it is important to under-stand the current base of evidence concerning the impact of these inter-ventions. The Initiative for Smallholder Finance estimates that during the period 2011 – 2013, major public funders com-mitted about USD 5 billion to projects and investments with smallholder farm-er (SHF) financial and/or digital services components (note that funds specifically directed towards SHF finance activities represent a subset of this USD 5 billion2). During this same period, more than 60 foundations committed a further USD 170 million to investments with a full or partial focus on smallholder finance.

2 Due to data availability limitations, this sub-portion of the USD 5 billion cannot be isolated.

The purpose of this report is to:

1. Articulate a theory of impact for rural and agricultural finance

2. Assess the current evidence base for the impact of financial products on smallholder and rural households, particularly in Africa

3. Highlight current approaches and challenges faced in measuring the im-pact of rural and agricultural finance

4. Recommend a path forward for ad-dressing current evidence gaps

This study emphasizes financial solutions and their impact in Sub-Saharan Africa, which is the location of the MasterCard Foundation’s financial inclusion portfolio, and where poor smallholders, as a share of population, are most prevalent. The report does not focus on how to improve the impact of RAF interventions them-selves, or seek to identify research ques-tions unrelated to impact, though these are subjects for future study.

Rural and agricultural finance refers to financial solutions for poor agricultur-al and rural households. These finan-cial solutions can include credit, savings, insurance, or payment products,1 with providers often bundling two or more of these products into a more comprehen-

1 We recognize that clients of financial solutions may not think in terms of these product categories, but it is useful when assessing impact to use a product characterization recognizable to providers, even while noting the need for new perspectives about the offering.

INTRODUCTION


3. Client-level risks: Rural households face obstacles and repayment risks different from those of urban house-holds, particularly in rural house-holds’ reliance on the weather and their limited access to markets.

4. Provider profiles: The unique chal-lenges associated with reaching rural clients discourage the entry of many traditional financial service providers with a focus on profits, unless they have the ability to invest with a long-term view.4 Instead, RAF providers have often included social entre-preneurs, agribusinesses, and mis-sion-oriented microfinance providers.

These differences are substantial and are likely to have a significant effect on the scope of RAF impact and how that impact is realized. For this reason, this reported and the preceding literature review focus on RAF-specific studies.

The findings in this report were devel-oped after interviews with nearly 30 ex-perts among funders, researchers, and providers; human-centered design, or “participatory evaluation”, exercises gath-ering input from farmers in Kenya and Ghana; and a synthesis of impact reports and reviews, including the Lab-commis-sioned EPAR literature review mentioned previously that should be read as a com-panion to this report.

4 Even so, the banks that have sought to expand their reach in rural areas tend to focus on value-chain financing.

Although there is a growing evidence base on the impact of broader finan-cial inclusion interventions, these are not generalizable to rural and agricul-tural finance. The underlying premise of RAF is similar to that of broader financial inclusion interventions that are not ex-plicitly targeted at rural/agricultural cli-ents: providing access to needed finan-cial services can help poor households engage in better-paying and more stable income-generating activities, offset in-come shocks, and build wealth, thus con-tributing to improved well-being.3 How-ever, there are a number of factors that make the realization of impact for small-holders and other rural clients different from that of urban clients more typical-ly targeted by microfinance institutions (MFIs):

1. The nature of the opportunity: Most rural clients are engaged in agricul-tural activities and have access to a productive asset (i.e., land or live-stock) that produces net value. This is in contrast to many urban clients who are often involved in microenterpris-es that rely merely on resale margins or who may have more irregular income sources.

2. The offering: Rural and agricultural finance is increasingly associated with a distinct set of financial solu-tions tailored to the rural context (i.e., accounting for seasonality or using remote delivery mechanisms). Furthermore, in the case of agricul-tural solutions, the finance is typically complemented by a greater level of non-financial support, such as ex-tension services or market access / value-chain development support.

3 Cull, Robert, Tilman Ehrbeck and Nina Holle. 2014. “Financial Inclusion and Development: Recent Impact Evidence.” CGAP Publications. Available at: http://www.cgap.org/publications/finan-cial-inclusion-and-development-recent-impact-evidence.


loan) and focus on agricultural outcomes when measuring impact. The financial inclusion perspective has emerged from an accumulation of lessons around mi-crocredit and, later, microfinance. This view emphasizes providing a suite of fi-nancial solutions to poor households (beyond credit), and sees rural and agri-cultural customers as a financially under-served segment that would benefit from these offerings. A provider taking this view may offer RAF solutions comprising multiple products targeting a broader set of rural household needs (e.g., a gen-eral loan with a savings component, ac-cessed through a mobile payments plat-form). Impact measurement in this view may be less focused on intermediate outcomes like crop yields or client adop-tion of a farming practice, and more on overall household well-being, which may include health and education. A single provider may, of course, be influenced by both perspectives.

Figure 1 lays out a theory of impact (ToI) that captures these different pathways to impact .1 The ToI starts with inputs on the left—in this case, finan-cial solutions (and associated products

1 We include in the Annex the theories of change of several key stakeholders in RAF for reference purposes.

A general theory of impact for rural and agricultural finance

Sponsors and providers of rural and agricultural financial solutions ap-proach the question of impact on clients through at least two differ-ent lenses: 1) agricultural develop-ment and 2) financial inclusion. The two perspectives emphasize different, though not contradictory, pathways to impact for the customer. The agricul-tural development view often starts from the recognition that Africa (and its smallholders) can feed itself and the rest of the world. From this perspec-tive, RAF is a critical tool—addressing a binding constraint—for improving ag-ricultural productivity, complementary to other tools such as improved inputs and extension services. A provider with this perspective tends to prioritize farm-related needs when designing fi-nancial solutions (sometimes trying to bake specific agricultural practices into those solutions, e.g., by including spe-cific kinds of seeds as part of an in-kind

SECTION I THEORY OF IMPACT


and supporting services)—and progress-es through outputs and outcomes to im-pacts. Vertically, the ToI is partially divided into two parts—the top focuses on agricul-ture-specific financial solutions while the bottom captures non-agriculture specif-ic solutions, often linked to the “financial inclusion” perspective described above. This is meant to highlight the different pathways, though in actuality there is both

overlap and convergence, especially at the impact level (as depicted). Note that the ToI has been adapted to incorporate the perspective of various stakeholders in the RAF space, and thus should be seen as an aggregated view.

• Technical assistance / agronomy training

• Upstream and downstream market linkages

•  Ag credit (working and investment capital) •  Ag-‐specific savings*

•  Insurance (e.g., crop, livestock, price)

•  Payment services in ag value chains

Agriculture-‐specific financial (and complementary) solu7ons

•  EffecEve use of more and beHer inputs (incl. labor) •  Purchase/rent of producEve assets (incl. land, irrigaEon equip.) •  InstallaEon of storage/processing faciliEes

General financial solu7ons

•  Formal protecEon of farm revenues

Broader (non-‐agriculture specific) financial solu5ons:

•  Non-‐ag credit •  Savings •  Insurance (life, health, etc.) •  Payment soluEons

Less likely to be complemented with a non-‐financial solu7on

•  Increased financial liquidity •  Investment in off-‐farm income-‐generaEng acEviEes •  Purchase of savings-‐generaEng or producEvity-‐enhancing assets (e.g., solar lamp) •  AccumulaEon of savings •  Formal protecEon against shocks •  Access to remiHances

•  Ability to make bulk payments (e.g., school fees) •  DiversificaEon of income sources •  Reduced household expenses / enhanced producEvity •  Financial buffer against shocks •  Reduced distressed sales of assets •  Increased sense of security / improved psychological state

•  Efficient, safe access to input and output markets

•  Improved nutriEon and food security

•  Higher and more stable net agricultural income and consumpEon

•  Recurring investment in farm producEvity

• Wealth accumulaEon

•  Improved household well-‐being (educaEon, health)

•  Strengthened household resilience

•  Ability to contribute to community

Inputs Outputs Outcomes Impact

•  Higher yields / surplus producEon •  Less volaElity in output •  Reduced postharvest losses •  Control over Eming of surplus sales

•  Reduced distressed sales of farm assets •  Increased sense of security

•  Lower transacEon costs, losses •  BeHer pricing beyond farm gate •  Access to higher quality inputs

Figure 1: Rural and Agricultural Finance Theory of Impact


Even though the ToI lists product cat-egories for the sake of illustration, in reality many financial products are interchangeable. For example: in the event of an unexpected health incident, a household can 1) draw down a previous-ly purchased insurance policy, 2) draw down savings or sell off liquid assets, or 3) take out a loan to cover the associat-ed costs. Similarly, financial products are often used by households in many differ-ent ways. For example, given the fungi-bility of money, a cash loan—regardless of its intended purpose—can be used for any of a variety of household needs (including some that may be better met by a different financial product). The in-terchangeable nature of many financial products is important to keep in mind, as it has important implications for impact measurement.

As expected, there is significant over-lap in the desired final impacts, re-gardless of whether the financial solu-tion employed is agriculture-specific or more general. For instance, provid-ers offering both agriculture-specific and more general RAF products often aim

for impact that includes higher and more stable income, wealth accumulation, and strengthened resilience for clients. Pro-viders of RAF may, however, differ on the relative importance of different impact measures, with differences often stem-ming from the design of their program and/or the profile of their customers. For instance, interviews indicate that organi-zations working with the ultra-poor gen-erally prioritize resilience, while those serving farmers who are slightly better off (e.g., have more land, produce cash crops, etc.) are more focused on in-creasing productivity. This is consistent with some of themes emerging from our team’s participatory discussions about impact with farmers, which are present-ed in the next section.

Generally, stakeholders in the rural and agricultural finance space have a holistic perspective on impact. This can be observed in the variety of out-comes considered in the ToI, which go beyond just increases in yields and/or in-come to capture improvements in house-hold health, education, and even psycho-

INPUTS

OUTPUTS + OUTCOMES IMPACT

Enabling environment: Ecosystem and contextual factors such as policy and culture

Customer: Client characteristics, e.g., age, gender, initial wealth

Figure 2: Factors influencing impact outside of the standard results chain


The customer’s perspective

To understand more what a theory of impact might look like coming from actual beneficiaries – smallholder cli-ents – the Dalberg team employed a human-centered design approach in conducting in-depth interviews with 28 farmers in Ghana and Kenya. Two of the Learning Lab’s partner organi-zations—Opportunity International and One Acre Fund—supported the team in choosing a diverse group of clients to interview; the team also conducted in-terviews with a group of farmers not af-filiated with the partner organizations. The sample was, of course, not meant to be representative, as the objective of this qualitative research was to try to surface new farmer-generated insights that might inform future approaches to measuring impact. Each conversation took place on the interviewee’s farm and lasted about two hours, covering topics ranging from farmers’ daily activities to their use of financial instruments to their aspirations. Pictures and quotes from these interviews are captured in the vi-sual excerpt of this chapter, “How Would Farmers Measure the Impact of Financial Solutions,” posted at raflearning.org.

The farmers we interviewed – in-cluding financial service clients and non-clients – cited measures of suc-cess similar to the impact goals noted by providers, funders, and research-ers; however, the farmers expressed a distinct order of priorities. The farmers interviewed had a very clear prioritiza-

logical well-being.2

In practice, a complex range of other factors influence the realization of any impact pathway. In particular, the inputs (or financial products) alone do not de-termine impact. As illustrated in Figure 2 below, developing a complete perspec-tive on the impact of RAF also requires assessing the many factors that can influ-ence the realization of impact, in partic-ular:

1. How does the product experience associated with a financial solution affect its impact—e.g., design fea-tures, delivery channel, marketing, training, etc.?

2. What characteristics of the enabling environment affect outcomes—e.g., government policies, cultural norms, etc.?

3. In what ways do customer character-istics affect the impact of a financial solution—do factors such as initial wealth, education, risk aversion, etc. lead customers to behave different-ly?

We discuss in the “Evidence to date” sec-tion of this report the degree to which the literature considers these factors.

2 While measuring impact beyond income can generate valuable knowledge, some stakeholders expressed concern that it may be too invasive.


Figure 3: Evolution of smallholder measures of success

Basic household needs

Farm productivity and income

Community investment

Beyond farming

Ensure household is producing enough to eat and send kids to

school

Evol

utio

n of

hou

seho

ld in

com

e

Subsistence farmers Commercial farmers

Invest surplus in the farm or home

(eg, increase land under cultivation, purchase

livestock)

Take on a more active role in the community

Generate sufficient profit to invest in diversifying economic activities and

retiring from farming

Figure 4: Farmer measures of success – Kenya interviewee

Foodsecurity

Basiceducation

Morelivestock

Biggerfarmland

Biggerhomestead

Retirementplans

Supportextended

family

Stablebasic income

Communityleader

Commercial real estate


cation for children, as demonstrated in the graphic below. Once they succeed in providing these basics for their families, the farmers see impact in their lives as ex-panding their farms, supporting extend-ed family and their broader community, and later, making investments for retire-ment.

For emerging commercial farmers, the priority outcome was increased income in order to be able to invest. Farmers interviewed in Ghana had a stronger commercial focus—typically dedicating at least a portion of their farm to cash crop production—and prioritized increasing their production and income as demonstrated in the graphic below. Once that was achieved, impact looked like investing in their homes (moving from a rental or shared situation to their own property) and farms, as well as in their children’s advanced and/or private

tion of end goals, typically starting with meeting an essential need, such as put-ting enough food on the table, before progressing through a series of steps perceived as representing increased success. The figure below shows how a smallholder finance client might start out at a certain point along an axis of prior-ities and his/her priorities evolve over time.

For subsistence farmers we talked to, the most critical measure of success was the ability to feed their family and send their children to school. Farmers interviewed in Kenya were typically non-commercial smallholders and more fo-cused on subsistence. Their most import-ant priorities – and thus desired financial solution outcomes – were meeting basic needs such as food security and sec-ondary needs such as rudimentary edu-

Figure 5: Farmer measures of success – Ghana interviewees

Biggerfarmland

Build townhome

Higherincome

Cropdiversification

Livestockfarming

Qualityeducation


education.3 Eventually, they could con-sider diversifying their income sources through other kinds of farming.

While geographical factors play a role, we believe that the primary driv-er of the difference in priorities cited by interviewees in Kenya and Ghana is a farmer’s stage in farming rather than his/her location. Even though the differing profiles of farmers interviewed in Kenya and Ghana led them to measure success differently, we anticipate that measures of success would converge to some degree across the two countries if the farmer stage were similar.

Understanding the priorities of farm-ers is important not just for the design of financial solutions—which is beyond the scope of this report—but also for the measurement of impact. By under-standing where a particular customer is along the curve of household needs or outcome priorities, finance providers or researchers can design more effective evaluations of impact. First, this base-line will better enable providers to un-derstand what success looks like from the client’s perspective and to assess the degree to which they are helping a client achieve these goals. Second, this knowl-edge can improve the accuracy of impact assessment efforts. If research measures an outcome that the household is not pri-oritizing – for example cash profits for a household prioritizing food security – it may be less likely to find a statistically significant result, and could understate the impact of an intervention.

3 While it may appear odd that a Ghanaian farmer prioritizes a town home over an activity such as livestock raising, it is important to realize that farmers in Ghana typically do not live on their farms, but rather live in the nearest village/town, often renting their home or sharing with relatives.


We look at two categories of evidence that each provide insight into the impact of RAF: the academic literature and, as a complement to this, the internal mea-surement, evaluation, and learning (MEL) efforts of RAF providers.

BOX 1: Impact assessment of financial products versus zero-return interventions

RAF solutions, like most microfinance offer-ings, typically generate revenue for the pro-vider that decreases – and in some cases, fully eliminates – the need for donor support for the intervention. While not all RAF pro-viders are, or seek to become, fully financial-ly self-sustaining (particularly when the RAF provider’s solution also includes non-finan-cial services such as financial and/or agro-nomic training), the ability for this revenue to fund at least part of the intervention cost and lower reliance on donor funds is import-ant to consider when comparing the impact of a RAF intervention to that of other inter-ventions targeting smallholders (e.g., exten-sion services, unconditional cash transfers). In other words, since benefits (impact) must be weighed against costs, it’s important to keep in mind that net costs in a market inter-vention like RAF are meant to be decreasing over time. In the case where costs are elimi-nated, impact assessment is no longer about “aid effectiveness” but rather optimal use of investment given other opportunities.

SECTION II EVIDENCE TO DATE

Findings from the academic literature

A literature review conducted by EPAR offers a starting point for assessing the evidence base. The Learning Lab commissioned this report to identify all the methodologically rigorous studies on the impact of rural and agricultural finance in Africa on four outcome areas of interest: 1) production,4 2) income and wealth, 3) consumption and food secu-rity, and 4) resilience.5 The EPAR review makes a good companion to this report as it contains details on each paper re-viewed.

The empirical evidence base for rural and agricultural financial inclusion is limited. Ultimately, only 38 studies qual-ified for inclusion in the literature review based on EPAR’s filter criteria.6 The Dal-

4 Production includes measures of total agricultural production as well as measures of farmer productivity such as yield per hectare or technical efficiency.5 Resilience is defined as the ability to cope with shocks and/or smooth consumption.6 The researchers conducted a systematic search of academic and gray literature databases to identify a total of 1038 studies. They then filtered through: i) a focus on Sub-Saharan Africa—as the continent’s unique agricultural context makes it difficult to generalize the findings of studies from other regions—eliminated over 60% of the studies; ii) measuring a dependent variable signifi-cantly outside the four outcome areas of interest (e.g., measuring the impact of an intervention on access to finance, which is not a question of ultimate impact on the customer) eliminated a further 345, or nearly 90%, of the remaining studies; and iii) application of rigorous methodologies—experimental, quasi-experimental, and econometric methodologies with reasonable sample sizes and careful implementation—eliminated a final 41 studies.


Table 1: Mapping evidence to impact framework

Total number of studies

Ag credit

Use of inputs

OUTPUTS AND OUTCOMES IMPACTS

5

3

15**

7

4

5

Crop Insurance

Non-‐Ag. Credit

Savings

Payments

Insurance

Broader Rural Finance

Exclusively posi.ve results Mix of posi.ve and non-‐ significant results Mix of posi.ve and nega.ve results

Exclusively nega.ve results Exclusively non-‐significant results

Consump>on and food security

Income Non-‐material household well-‐being

Households assets and resilience

Produc>on

1 2

1

3

1

2 1

1

1 1

2

1 1 2 2 3 2

1 3

3

2 1 1 1 1

1

3

1 3

1

1 2 3 1 1

1 2 1

2 1 1 1

1

1 1


all investigate the impact of fairly sim-ilar input loans may choose to measure that impact in terms of agricultural reve-nue, consumption expenditure, or maize production, which are not easily compa-rable. We provide more information on this challenge in the “Measurement ap-proaches” section.

Nevertheless, the available studies suggest that financial products can make a real impact on smallholder and other poor rural clients, but that this impact is far from guaranteed. Of 39 studies, 21 record only positive, statis-tically significant impact and nine find a mix of significant and not significant, but still positive, results. Six papers fail to find any evidence of significant impact (either positive or negative). Only two studies—both of them focused on group-liability business loans—find any evidence of a negative impact. 9 Tarozzi et al. (2015) sus-pect the negative result is spurious since most other indicators changed in a posi-tive direction. Crépon et al. (2014) find a positive effect overall, but large hetero-geneity in findings, with 25% of borrow-ers experiencing negative returns. The remaining studies find a mix of positive and not significant results. However, as the Crépon el al (2014) study demon-strates, we need to be cautious about heterogeneous impacts concealed be-hind positive or neutral average effects and what those mean about the suitability of financial products to certain customer profiles. In addition, we acknowledge the possibility of publication bias having led to the underreporting of negative or not significant findings.

9 The 39th study offers an interesting insight about the interac-tion between different financial products—mainly insurance and credit—but does not directly measure impact on the household (Gine and Young, 2007).

berg team excluded one of these studies as being of limited relevance,7 but added a further two studies based on recom-mendations from interviews and a sup-plementary consultation of the literature. Twenty of those studies focus on credit, while other product categories remain relatively understudied: there are just seven studies focused on savings, eight on insurance, and four on payments products. This is not surprising in the case of insurance and payments prod-ucts: both are relatively new—especially digital financial services—and adoption rates remain low (with a few exceptions, like m-Pesa).

The available academic evidence to date is suited for examining the direct relationship between inputs and out-puts/outcomes/impact more than for exploring those factors and qualities outside of the linear results chain. In Table 1, we mapped the number of stud-ies and direction of their findings by type of product (input) and type of impact (or output or outcome) measured.8

It is notably difficult to aggregate lessons from the available studies because definitions of impact and product types vary significantly. For example, savings-related studies explore the impact of village savings and loan associations (VSLAs), rotating savings and credit associations (ROSCAs), sub-sidized savings accounts, individual sav-ings accounts, savings accounts with and without commitment mechanisms, etc., but rarely at once. Similarly, studies that

7 The study in question measures insurance adoption, rather than the impact of insurance (Norton et al., 2014).8 While we have generally complied with EPAR’s categorization of impact, we diverged in placing measures of asset accumulation un-der resilience instead of income. In reality, assets could easily serve both as income-generating and resilience tools.


The two most commonly studied types of impact are increases in income or consumption, which is often used as a proxy measure of income. Studies find that financial solutions generally have a positive—although not always significant—effect on indicators in these categories; in some cases, these effects accumulate over time. For example, Berhane and Garde-broek (2011) find that household per-cap-ita consumption increases by 5 – 12 per-cent for each additional year a household borrows, while Owuor (2009) finds that participation in group-liability loans by smallholder farmers in Kenya increases household incomes by USD 200 – 260 in a single production period and over the course of a year by USD 478 – 641. Produc-tion-related indicators, such as land under cultivation, yield, and cereal production, are the next most commonly studied cat-egory; the majority of evidence there is positive and statistically significant—in fact, slightly more so than income or consump-tion. This is not surprising, as the link be-tween an input such as an agricultural loan and farm yields is more direct than, for ex-ample, the link between that loan and farm income.

Although the number of studies is still very limited, evidence around the im-pact of agricultural credit is particu-larly mixed. Only five studies look at the impact of credit products that are specifi-cally targeted to farmers. These products are: loan and training bundles (2), balloon loans (2), and maize flour on credit. Bea-man et al. (2014) is the only paper to find an unequivocally positive, statistically sig-nificant impact from agricultural lending—on production from a group loan in Mali. Burke (2014), Ashraf et al. (2009) and Fink el al (2014) all find a mix of positive and not significant results depending on the im-

pact metric used. For example, Ashraf et al (2009) finds positive one year impacts of a credit, extension and marketing bundle on farmers’ production and marketing costs; however, the impact on household income is positive but not significant. Finally, Ade-bayo et al. (2012), measuring the impact of a bundled credit and training intervention on farmers in Nigeria, find that the inter-vention had no significant impact on the crop production, food security status, or monthly food expenditures of beneficia-ries.

The volume of research dedicated to exploring the other kinds of relevant financial products remains too small to be conclusive. Six out of seven10 stud-ies find positive and significant outcomes from insurance. Studying crop insurance specifically, Karlan et al. (2014) find that farmers in Ghana given free weather insur-ance invested 33% more in agricultural in-puts. Health insurance has been observed to increase income (Yilma et al. 2015) and reduce health related consumption shocks (Shimeles 2010). Despite these largely positive results in the few studies of the impact of insurance, uptake of avail-able microinsurance products remains a challenge.11 A majority of research around payments and savings also finds positive results, although these are mixed with non-significant results. For instance, in the case of group savings, BARA/ICA (2013) find no significant impact on value of live-stock, whereas Annan et al.(2013) find that treatment households had assets valued

10 While an earlier figure indicated that there were eight studies fo-cused on insurance, one of these (Gine and Young, 2007) studies the impact of insurance on loan access, and thus is not directly related to impact metrics. Of the six studies with positive, statistically significant results, two are focused on agricultural insurance and four on health insurance. 11 See Matul et al. 2013. “Why people do not buy microinsurance and what we can do about it.” ILO. This study reports that voluntary microinsurnace schemes rarely enroll more than 30% of a target pop-ulation, with even lower renewal rates.


at roughly the equivalent of one head of cattle more than did control households.12

Across the evidence base, studies with relatively short research timelines (i.e., two years or fewer between baseline and end line) are more common. While the effect of the research timeline on re-sults will vary by intervention type, these relatively short studies may not yield signif-icant findings for certain impact indicators that emerge over a longer period of time.

12 Note that BARA and ICA (2013) study rotating savings and credit associations (ROSCAs), whereas Annan et al. (2013) study village sav-ings and loan associations (VSLAs), which are slightly different group savings mechanisms.

The effect of the external and quality factors highlighted in Figure 2—custom-er characteristics, enabling environ-ment, and the product experience—on the realization of impact has been insuf-ficiently studied to generate aggregate insights. Only a handful of studies have considered one or more of these factors; we discuss the findings of those studies in Table 2, below:13

13 No studies were available for “Enabling environment.”

OVERVIEW EXAMPLES

Cus

tom

er c

hara

cter

istic

s

Customer profile• Studies have looked at: risk aversion and

understanding of contracts, gender, lev-el of initial income/wealth/profit

• One study finds that only male-headed households enjoy long-term benefits

• Four* studies find that customers with higher initial levels of income/wealth/ profits reap greater benefits from access to credit and health insurance

• Schaner (2015) finds that subsidizing savings increases household income, but the in-crease persists in the long run only for male participants

• Hill and Viceisza (2010) find that the impact of crop insurance was greater for more risk-averse clients, as well as clients with a better understanding of contracts

• Shimeles (2010) finds that the degree of healthcare utilization was much higher among non-poor than poor subscribers to a community-based health insurance scheme

Unobservable marginal returns• One study identified different out-

comes associated with customer in-terest in taking a loan, suggesting that clients have different marginal returns to borrowing, or at least inclination to invest in farming (this also highlights the importance of using control groups when measuring impact)

• In Beaman et al. (2014), the primary inter-vention was loan disbursal in treatment vil-lages, but researchers also offered grants to randomly selected households in control villages and to some households in treat-ment villages who did not take up a loan. They found that offering grants in no-loan villages generated statistically significant increases in farm profit from investments in cultivation, but such is not the case in loan villages. The study concluded that non-bor-rowers had different (smaller) marginal re-turns to borrowing but did not investigate what was behind this.

Table 2: Studies on the effect of external and quality factors


OVERVIEW EXAMPLESPr

oduc

t exp

erie

nce

Delivery model• Studies have looked at bundling cred-

it with training or insurance and at pro-portion of potential customers served, but evidence is generally scarce and inconclusive

• Two studies find mixed results with re-spect to the impact of training on credit outcomes

• One study finds some indication that choosing to make a financial solu-tion available to an entire community instead of only to a portion of house-holds may affect outcomes

• Kim et al. (2009) find no statistically sig-nif-icant difference between villages that re-ceived a microfinance-only intervention (non-agriculture specific) and those that re-ceived a combined microfinance/training in-tervention

• Ashraf et al. (2009) find that that the treat-ment group offered credit had higher in-come when compared with the treatment group offered training only, but outcomes were not significant for either group

• Gine and Young (2007) find that bundling credit with mandatory insurance decreases adoption of credit (and as a result, lower in-vestment in farm inputs)

• Fink et al. (2014) find that loans of maize flour were only effective when offered to 100% of eligible households instead of just 50%

Product design• Just two studies focus on the impact of

product design, particularly for savings• They find that customizing the design

of a savings solution can improve its effectiveness or alter its primary pur-pose

• Bruné et al. (2014) randomly assigned farm-ers to several different treatment groups; they found that the group assigned a savings account with commitment features experienced the greatest improvements in input use, farm output and farm profit.

• Dupas and Robinson (2013) assigned ROSCAs to four treatment conditions, which offered a combination of savings and commitment technologies; only the indi-vi-dual savings account increased a house-hold’s ability to afford medical care, while the lock box (for which the key was held by someone outside the group) and the health savings pool increased investments in pre-ventative health

While no studies have explicitly sought to measure the effect of different en-abling environments, one study does demonstrate their importance. Ashraf et al. (2009) examine the outcomes of an intervention conducted by DrumNet in Kenya that aimed to help farmers adopt and market export crops. One year after the study ended, the European Union changed its import requirements. Par-ticipating farmers could not comply with these new requirements, so the export-er refused to continue buying their cash crops, for which there was no local de-mand. Farmers defaulted on their loans

and DrumNet collapsed in that region.

In summary, rigorous studies high-light successes and some failures in delivering client impact, but are far from making sense of the range of exogenous and endogenous factors that drive impact. Going forward, more studies of how these factors affect RAF impact can inform practitioner approach-es. The key question is not what is the im-pact of RAF on clients, but what solutions and factors make financial solutions most impactful for rural clients.


BOX 2: Evidence from outside of Africa

The literature review conducted by EPAR also includes a scan of findings on RAF impact from beyond Africa, summarized here:

Credit: As with findings from Africa, evidence from other regions is quite mixed, typically finding some positive and some non-significant results (Mexico – Ange-lucci et al., 2013; Mongolia – Attanasio et al., 2011; India – Banerjee et al., 2015)

Savings: In general, the evidence suggests that expansion of formal savings opportunities to unbanked rural areas has positive impacts (India – Burgess and Pande, 2005; Mexico – Apportela, 1999 and Bruhn and Love, 2009)

Insurance: Just two relevant studies were identified looking at the impact of agricultural insurance, finding a positive impact on production in Peru (Carter, Galarza, and Boucher, 2007) and China (Cai et al., 2009). However, a forthcom-ing study from India (Tobacman et al., 2015) found little to no systemic long-term effect on farmer investment, revenues, or savings.

Source: Anderson et al. 2015. “Evidence on the Impact of Rural and Agricultural Finance on Clients in Sub-Saharan Africa.” EPAR Brief No. 307.

Results from practitioner evaluation efforts

Practitioner research generally shows positive effects but also highlights ar-eas to improve; this research is itera-tive and has been expanding in scope. Even where RAF providers’ internal mea-surement, evaluation, and learning (MEL) efforts do not meet the same standards of rigor as academic research, findings from these efforts are important and comple-mentary to the academic research base. For example, practitioners are well-po-sitioned and incentivized to measure im-pact early, adjust the intervention and evaluate again. We reviewed the impact measurement results of five organizations whose work was deemed relevant to this study. Table 3 summarizes those findings in a format similar to that employed for the literature review.

Practitioners primarily measure impact on agricultural production/yields and net income; they generally find that their programs have a positive effect on these measures. For example, MyA-gro’s 2014 internal monitoring and evalua-tion (M&E) reveals that treatment farmers, who access agricultural inputs on layaway via a mobile payment platform, earned a net profit 43 – 112% higher than control farmers. In 2014, Nuru International found that its Agriculture Program—which pro-vides farmers access to farm input loans, technical training, extension services, and group support structures—reduced the incidence of hunger among its clients in Kenya. The organization has also record-ed improved yields and revenues among customers, and found that customers in Kenya had higher profits relative to con-trol farmers.


Total number of RAF providers with studies reviewed*

Ag credit**

Consump)on and food security

Income Non-‐material household well-‐being

Households assets and resilience

Produc)on Use of inputs

OUTPUTS AND OUTCOMES IMPACTS

3

1

Ag sa

ving

s

1 2

1

1 2 1

1

Exclusively posi.ve results Mix of posi.ve and non-‐ significant results

Exclusively non-‐significant results Mix of posi.ve and nega.ve results

1 Other***

1 1

1 1 1 1

1 1

Broader Rural Finance

Table 3: Summary of practitioner evaluation results

However, not all impacts are positive, and providers can use disappointing findings to improve their programs. For example, in 2009, One Acre Fund—which offers farmers in-kind farm input loans, insurance, and training14—mea-sured increase in maize profits and found high heterogeneity in results. While they observed a 40% average increase in maize profits relative to control group, they discovered that some farmers actu-ally reported negative returns on inputs. This led the organization to implement a number of changes, including conduct-ing trials to reduce fertilizer recommen-dations, requiring a 10% prepayment of the loan, making crop insurance manda-tory, and revising farmer eligibility crite-ria.15 All of these changes have contrib-uted to improved results in subsequent rounds of measurement. Nuru Interna-

14 As well as a variety of add-on products, such as solar lamps. 15 For more detail around the lessons generated by the 2009 re-sults, please refer to “One Acre Fund: Randomized Control Trials,” available at https://www.oneacrefund.org/uploads/all-files/Radom-ized_Controlled_Trials_One_Acre_Fund__2014.pdf

tional offers another example. While Nuru farmers experienced positive results in 2014, the percentage change relative to the comparison group was lower than the previous harvest season. Moreover, not all farmers had adopted the diversified crop strategy and Maize Lethal Necrosis Disease (MLND) presented challenges to Nuru farmers. Given these findings, the Agriculture team included MLND preven-tion funds in its budget and revamped the crop package to make it more attrac-tive to farmers, while ensuring that farm-ers were still diversifying. Finally, they improved training and supervision to make sure that farmers were continuing to adopt appropriate agronomic practic-es. The Nuru team is optimistic about the next round of results measurement.


Fewer organizations have published impact findings on non-financial as-pects of household well-being such as education or health, but interviews suggest these impacts are increasingly being added to learning agendas. Op-portunity International (OI) gives farmers access to a broad suite of financial ser-vices. In its 2013 impact assessment, OI found that its clients reduced the number of school days missed as a result of late payment of school fees more than did comparison households, and were more likely to report that they were better able to meet educational related expenses than in 2009. They were also significantly more likely to report improved access to health-care and greater ability to meet healthcare expense than were non-borrowers. Village Enterprise (VE), which targets ultra-poor households in Kenya with a package that includes a business seed grant, savings group, and training, conducted a longitu-dinal impact assessment in 2010. Findings from that assessment indicate that, for the children of treatment households, enroll-ment in school increased from 68% to 95% and the number of households with cor-rugated iron roofs increased from 37% to 65%. Unfortunately, this study did not ben-efit from a comparison group (although this will change soon, as VE has launched a randomized control trial (RCT) described in the “Methodologies” section).

Evidence gapsSignificant research gaps remain across the board. First, although the bulk of the literature has focused on credit, the evi-dence around agricultural lending is incon-clusive and merits additional investigation. In addition, there are general gaps in evi-dence around the impact of savings, pay-ments, and insurance products. As some of these products continue to gain traction with customers—e.g., mobile money—de-veloping a better understanding of their impact will be critical. Finally, there is little quantitative evidence around the influence of such factors as delivery method, cus-tomer characteristics, or larger contextual considerations (e.g., ecosystem, policy, or culture). For example, interviewees have confirmed the need to develop a deeper, more nuanced understanding of how RAF outcomes are affected by the factors listed in Table 4 on the following page.

These factors can be included in fu-ture study designs – in some cases by mixing research methodologies – and practitioners should be working along-side researchers to document their ef-fects, as it has important implications for successful program scale-up. Fill-ing research gaps can also be made more manageable by prioritizing and then tar-geting research at the most relevant client segments, especially when initial evidence or marketing priorities point to a specific segment being the most likely user of a fi-nancial solution.


ing projects, given the gaps in evidence identified, include:

• Understudied products: Mercy Corps and Opportunity International will investigate the impact of digital payments (and other digital financial services);

• Product experience and bun-dling: Studies planned by Village Enterprise16 and Mercy Corps will look at the impact of delivery mode by designing a number of different treatments;

• Customer profile: IDinsight will measure the impact of farmer profile on outcomes;

• External validity: Innovations for Poverty Action (IPA) is replicating a study on VSLAs in Ghana, Malawi, and Uganda that will offer insights as to the generalizability of results.

The Learning Lab hopes to post live/edit-able data on upcoming and past studies at data.raflearning.org.

16 A two-page description of VE’s RCT is available here: http://vil-lageenterprise.org/wp-content/uploads/2011/05/RCT-2-Pager-Up-dated-2015.pdf

Prod

uct

expe

rienc

e• Product design features that reflect agricultural/rural needs versus general• Delivery of loans in kind versus in cash• Incentives and pricing of financial solutions• Bundling multiple financial products and services• Bundling financial products with non-financial services• Who or what is the primary client interface

Cus

tom

erch

arac

teris

tics • Initial level of wealth

• Value chain participation• Education/knowledge• Gender • Age

Enab

ling

envi

ronm

ent

• Cultural norms• Government policy

Interviews also reveal a set of addi-tional research questions that practi-tioners and donors are interested in pursuing about. These include:

• Impact chains: How are rural and agricultural financial solutions actu-ally generating impact, i.e., what are the underlying levers?

• Long-term outcomes: Does the impact on clients persist in the long run? Does it accumulate over time?

• Financial independence: Can RAF continue to have a positive impact while attaining financial sustainabili-ty?

• Farmer behavior: What aspects of farmer behavior are most important to the impact of a financial product? Can solutions be designed in a way that encourages farmers to behave in a way that promotes impact?

A pipeline of future studies will help to address some—though not all—of these gaps. Annex 3 lists in-progress studies highlighted during stakehold-er interviews. Some particularly excit-

Table 4: Product experience, customer and context factors to consider


• “Econometric Analysis” studies in-volve a cross-sectional multivariate regression with control variables.

A detailed description of these approach-es and evaluation of their respective mer-its is outside the scope of this report. Additional detail on these topics can be found in the full-length EPAR report.

Measurement methods observed in the academic literature

In its scan of the literature around the impact of financial products on smallholder/rural clients, EPAR en-countered a wide variety of research approaches. Only some of those, how-ever, were deemed sufficiently rigorous to warrant inclusion in the final literature review. These studies had to carefully im-plement experimental, quasi-experimen-tal, or econometric methodologies and draw on a reasonable sample size. EPAR defines the different methodology brack-ets as follows:

• “Experimental” studies are random-ized control trials or other experi-ments.

• “Quasi-Experimental” studies include a comparison group created by matching or other techniques includ-ing methodologies such as differ-ence-in-difference, propensity score matching, and regression discontinu-ity.

SECTION III MEASUREMENT APPROACHES


illuminating how a program is changing outcomes these stories can helporgani-zations formulate specific hypotheses to test in future quantitative research. Indi-vidual stories also offer a more holistic perspective on program impact, add-ing color to quantitative data. They can also raise flags in what is largely an aver-age-dominated field; as one interviewee described it, they allow the evaluation officer to “double click” on phenome-na they see reflected in the data. Final-ly, case studies play an important role in fundraising and awareness building, as they offer very tangible, personal exam-ples of success.

Participatory monitoring and evalu-ation is a qualitative approach that makes MEL more client-centric and can make research designs more rele-vant. As the name implies, in participato-ry evaluation, a broader set of stakehold-ers actively participate in developing and implementing the evaluation. Praction-ers/providers, program beneficiaries (or “clients” in the case of RAF), and funders all play a part, and participation occurs throughout the evaluation process in-cluding:

• identifying relevant research ques-tions;

• planning the evaluation design;

• selecting appropriate measures and data collection methods;

• gathering and analyzing data.17

In the case of RAF, participatory evalu-ation can help providers to incorporate the opinions of the farmer/client in pro-gram evaluation design and execution.

17 Zukoski, Ann and Mia Luluquisen. 2002. “Participatory Evalua-tion: What is it? Why do it? What are the challenges?” Partnership for the Public’s Health. Available at: https://depts.washington.edu/ccph/pdf_files/Evaluation.pdf

Practitioner approaches to measurement and evaluation

Practitioners and their funders are recognizing the importance of and in-creasing investment in measuring the impact of their programs. For some, this has meant collecting more data or identifying comparison groups. For oth-ers, internal impact assessments are be-ing supplemented with externally-led impact evaluations (IEs). These changes are often at the request of donors, who are eager to have evidence of the return on their investments. In fact, The Master-Card Foundation’s decision to create the Rural and Agricultural Finance Learning Lab is a key example of this move to gen-erate knowledge about RAF impact. Ta-ble 5, below, provides examples of new activities Lab partners and other organi-zations are adding to their impact mea-surement efforts.

Qualitative approaches to evaluation and learning are also valuable for prac-titioners, providing a more in-depth customer understanding and often guiding the design of future quanti-tative research efforts. For example, interviewees highlight that case studies serve multiple purposes. First, anecdotes can provide inspiration for and inform the design of quantitative research: many or-ganizations have success stories that they feel capture their impact, but may not al-ways have the data to back them up. By


RECENTLY ADDED IMPACT EVALUATION ACTIVITIES

RATIONALE

Opp

ortu

nity

In

tern

atio

nal In 2015, launched an impact evaluation of

their program in Ghana in collaboration with NORC at the University of Chicago. The evaluation will apply a matched differ-ence-in-difference quasi-experimental de-sign with a baseline sample of more than 2,000 customers.

OI’s 2013 internal impact assessment pro-duced encouraging results, but relied on a retrospective baseline using farmer re-call, and faced limitations in defining the comparison group. OI saw an opportunity to test further and achieve greater internal validity, with support from The Mastercard Foundation for the resources requirements of more rigorous techniques.

One

Acr

e Fu

nd

Conducted two randomized control trials (RCTs)—in 2009 and 2014.

One Acre Fund sought to confirm that on-going measurement methods were pro-ducing correct indications of impact.

Launched a Quality of Life Study in August 2015: • Has a sample size of 2,400 in each coun-

try; • Will employ a difference-in-difference

methodology with propensity score matching to measure impact on a wide range of financial and non-financial areas including education and health access, food security and nutrition, income, and assets over a period of 3 – 5 years.

One Acre Fund aims to:• Broaden its view on impact of financial

solutions and measure the impact of many add-on products it offers that are not financial (e.g., solar lamps);

• Determine where to invest more re-sources and target new interventions.

Mer

cy C

orps Just designed and is in the process of imple-

menting a MEL framework for AgriFin Accel-erate that enhances approaches from the previous program, and includes a significant investment in a rigorous impact study such as an RCT.

Mercy Corps aims to generate:• Lessons on product bundles and deliv-

ery models that are commercially via-ble, generally applicable, and scalable across other contexts;

• Evidence of how digital service provi-sion impacts incomes, productivity, and resilience of SHFs.

Villa

ge E

nter

pris

e (V

E)

Launched a three-year, independent RCT evaluation of its ultra-poor microenterprise development program. • Will include data from over 6,600

households in Uganda by BRAC and IPA.

• For the first time, will use comparison groups to explore differences in out-comes from several different treatments: full program, program without savings group formation, in-cash grant, in-kind grant, and control.

VE intends for findings to:• Guide the expansion and future pro-

grammatic refinements of the VE model;• Contribute new knowledge to sector

research.

Table 5: Selected examples of recent investments in enhanced impact evaluation


viewee on his/her farm, learning about his/her families, daily activities, experi-ences with financial products, and hopes and aspirations. The Appendix provides more information about the process we followed in each country and the inter-view guide we used.

The degree of comfort cultivated by embedding ourselves in the farmers’ settings allowed for an unparalleled level of depth and honesty in sharing experiences. This allowed us to garner a wide array of insights into their lives and circumstances and lent additional nuance to how we think about measur-ing impact by seeking to capture the multiple dimensions of the farmer. For instance, while many impact studies pri-oritize income measurement, it was clear that for the farmers interviewed in Kenya, paying school fees was much more im-portant—and this could be done through bartering. Focusing on standard income measures would likely miss a high-priori-ty outcome: an increased ability to cover school fees. Another key benefit was the sense of reward that farmers felt in con-tributing to the solution generation pro-cess.

Generally, practioners seek to balance different approaches in a MEL frame-work that is “just right” for their orga-nization. The goal is to balance the need for information on impact, i.e., how is your program affecting your client, with oper-ational data to inform management deci-sions, i.e., data on program rollout, effi-ciency, financial sustainability, etc. To do this effectively, RAF providers must also take into consideration 1) the target au-dience, 2) the available operational and financial resources, and 3) the potential

The “Most Significant Change” (MSC) technique can be a valuable partic-ipatory evaluation tool. To apply this technique, the evaluation team collects significant change stories (i.e., stories of impact) from the field; a panel of des-ignated stakeholders or staff then sys-tematically selects the most significant of these stories. The stories become the subject of frequent discussions and, if the technique has been implemented suc-cessfully, everyone in the organization or community begins to focus on program impact.18

While practitioners interviewed have not yet adopted participatory meth-ods at all stages of MEL, they are in-creasingly thinking about ways to include customer perspectives. All practioners interviewed expressed a commitment to ensuring that the cus-tomer’s voice is heard in program design and evaluation. Approaches vary among organizations, including customer expe-rience interviews, focus groups, custom-er satisfaction surveys, etc.

Dalberg utilized participatory meth-ods—in-depth, one-on-one farmer interviews—to inform this research effort. Our team spent two weeks im-mersed in rural Kenya and Ghana in or-der to gain more nuanced insights into farmers’ priorities and how RAF assists them in achieving these objectives. Tak-ing care to identify a diversified sample of target interviewees based on charac-teristics such as gender, age, experience with RAF, type of crop farmed, etc., we spent about two hours with each inter-

18 Davies, Rick and Jess Dart. 2005. “The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use,” available at: http://www.mande.co.uk/docs/MSCGuide.pdf


burden on customers. RAF sponsors and providers interviewed noted that mon-itoring, evaluation and learning plans typically undergo frequent revisions to remain relevant to the organization and build on lessons from early rounds of data gathering. This is true regardless of whether the MEL plan was developed in-house or by external advisors.

Nuru International provides a useful illustration of a MEL framework that incorporates five different, but com-plementary, kinds of analysis, as de-scribed in Table 6 below. Each approach has been chosen to answer a specific research question. For instance, Barrier Analysis helps the MEL team figure out what may be precluding customers from adopting behaviors promoted by Nuru’s programs and thus hampering impact. Nuru has developed the framework over time to ensure that it continues to capture the complexities of its work. The last col-umn in Table 6, which describes efforts to measure the organization’s integrated impact (rather than the impact of individ-ual programs), is the most recent addi-tion. Nuru indicates that it would consid-er using an RCT in future to complement its existing MEL efforts.


STRENGTHS AND NEEDS ASSESSMENT (SNA)

BARRIER ANALYSIS

PERFORMANCE MONITORING

EVALUATING IMPACT OF PROGRAMS

EVALUATING NURU’S INTEGRATED IMPACT

PURP

OSE

AN

D

DES

CRI

PTIO

N O

F D

ATA

COLL

ECTI

ON

A participatory effort that clarifies the needs and the strengths of a community. Contributes to defining the problem statement and pro-gram goal of each Impact Program.

A discrete data collection process that helps Nuru identify why recommended behaviors are or are not adopted. Contributes to targeted program design.

A systematic data collection process used to measure progress toward stated pro-gram inputs, outputs, out-comes, and goal.

Strategic data collection that enables Nuru to make decisions about future programming (i.e., replication, scaling) and addresses whether the goal of the program has been achieved.

To determine if Nuru is achieving its goal of ending extreme pover-ty in remote rural areas.

PRIM

ARY

LE

ARN

ING

Q

UES

TIO

N

What are the needs and strengths of the community?

What are the barriers to adoption of recommended behaviors?

How effective and efficient is the rollout of the intervention?

What is the impact of the program?Where applicable, is the impact attributable to Nuru?

What is the impact of Nuru’s holistic model on poverty?

TIM

ING

At country start-upprogram design phase

Program design phase or program iteration

Ongoing – usually quarterly or annually

Prior to the start of each intervention (baseline) and at project mid and end points

Prior to the start of any intervention and at project mid and end points

SAM

PLE

SIZE

A

ND

DES

IGN

• Estimated at 60 primary• Purposeful sample• Primary triangulated

with secondary data

• 90 (45 doers and 45 non-doers) for each behavior

• Purposeful sample

• Depends on population size

• Purposeful sample (all - Nuru farmers/program participants)

• Depends on popula-tion size

• Random and repre-sentative sample

• Depends on popu-lation size

• Random and repre-sentative sample

DAT

A M

ETH

OD

Mixed methods Qualitative Mixed methods Quantitative Quantitative, MPI

Table 6: Illustration – Nuru’s MEL Strategy and Process for Designing Country-Specific Learning Agendas


ple, in helping to calibrate in-house MEL or determine whether or not to scale up—and in justifying future investments by funders and donors.

Others consider the design and imple-mentation challenges insurmountable and the results not sufficiently gen-eralizable in many cases. The external validity of RCTs in particular is a source of ongoing debate, with Cartwright’s (2007) article “Are RCTs the Gold Stan-dard?” offering a well-known critique of the method. As Cartwright explains, “the formal methodology puts severe constraints on the assumptions a target population must meet to justify export-ing a conclusion from the test popula-tion to the target.” Our prior discussion of external and quality factors affecting impact illustrates the problem of taking findings from a study in western Ghana and applying them elsewhere: RCTs typi-cally measure the impact of the interven-tion itself while holding external factors constant. This works well for internal va-lidity, but it is impossible to hold every-thing else constant when replicating the intervention elsewhere. And while qual-ity factors (like product experience) may be endogenous, they are usually not iso-lated and measured in RCTs. Two inter-esting examples from the literature are:

• “How much can we generalize from impact evaluation?” Vivalt (2015): The study demonstrates that specific pro-gram characteristics—such as who is implementing them, where they are being implemented, the scale and the methods being used—all affect the potential success of a replicated program.

• “Pitfall of Scaling-up Proven Educa-tion Interventions: Evidence from

Overcoming challenges to measurement

Rigorous measurement by practitioners is challenging, and while many of the dif-ficulties exist in other fields as well, the rural/smallholder farmer context often makes them even more acute. Table 7, at the end of this section, outlines the main challenges to measurement practitioners and researchers describe, along with some possible solutions. Some obstacles are more broadly applicable across MEL approaches, while others are more close-ly associated with IEs, including random-ized control trials (RCTs) and other forms of rigorous quantitative evaluation that involves a comparison group.

An important theme that emerges from the long list of challenges is the complexity of implementing and gen-erating meaningful insights from rig-orous impact evaluations, including randomized control trials. All of the stakeholders we interviewed acknowl-edge that IEs can be expensive, difficult to implement, and disruptive to oper-ations. However, they diverge in their opinions as to whether navigating these difficulties is worthwhile. Views on IEs are further complicated by the fact that these studies vary significantly in terms of time, cost and operational requirements to im-plement, as well as in the technical rigor achieved.

Some note that IEs offer a valuable, unbiased measure of their impact. This is useful both internally—for exam-


an RCT in Kenya,” Bold et al. (2012): The study demonstrates that there are obstacles to successfully scaling proven education interventions in Kenya due to the structural differenc-es in capacity between the imple-menting institutions.

Practitioners can also explore new ways to collaborate with impact eval-uation experts to obtain relevant find-ings or reduce costs. Some research providers offer more flexibility than the traditional setup, in which the external researcher largely owns both the design and implementation of the evaluation. For instance, in 2014, One Acre Fund conducted a mini-RCT on its Kenya op-erations in-house, but worked with IDin-sight to ensure accurate interpretation of results. This kind of collaboration can be enhanced to include expert consultations on the up-front design of a study prior to its launch. This approach allows the RAF provider to maintain greater control over the IE implementation, while drawing on research providers’ expertise in the de-sign and/or analysis stages.

In addition to insights on quantitative research, our team’s field work also highlights lessons around incorporat-ing qualitative methods in evaluation and learning. In demonstrating a partic-ipatory evaluation technique, we note six key insights for undertaking qualitative field research or client-centered design processes.

• Use a proven methodology/tech-nique: Qualitative research can and should be rigorous. Pre-established research processes of higher quality will generate more valid results. Our methodology was drawn from hu-

man-centered design approaches, but these certainly are not the only or most rigorous options available.

• Speak to a purposive sample of beneficiaries: Purposive sampling is a type of non-representative sam-pling focused on client characteris-tics of interest to the research aim. In our case we prioritized variation in our sample because we wanted to understand a range of “definitions of success” and what themes were com-mon across geographic, gender, and farming profiles. However, especially when linked to a specific financial solution, purposive sampling may mean focusing in on similar charac-teristics that comprise a targeted client segment to understand their particular perspective more deeply. Practically speaking, we (i) worked with partner organizations on the ground to recruit farmers based on pre-established criteria and (ii) inde-pendently recruited farmers through local leaders and at local markets.

• Utilize local experts: A team con-ducting interviews in a foreign coun-try can encounter cultural barriers arising from differing social norms. To moderate this, the team (i) inter-viewed experts on the ground (e.g., field officers) to better understand local norms, and (ii) worked with translators to ensure that they had sufficient training to ask accurate questions and do so in a way that was sensitive to farmers.

• Preserve transparency of the re-spondent’s voice: Interviewees should be able to articulate their views with honesty and transpar-ency, in order for the participatory approach to be successful. The po-


context and tone). In addition, review sessions with translators can ensure that their approach to asking ques-tions is appropriate. Finally, the team should be flexible enough to recruit a wide range of respondents and change its recruiting approach based on what is emerging as effective.

Practitioners and providers face a daunting task in measuring impact; luckily, there is a growing number of initiatives designed to collect and disseminate best practices in impact measurement. For example, it is clear from the challenges described earlier that RCTs are not appropriate in all situ-ations and organizations must carefully consider the tradeoffs involved before embarking on one. But how does one go about evaluating those tradeoffs and deciding how much measurement is enough? One recently launched initiative to help stakeholders answer this ques-tion is Innovations for Poverty Action’s Goldilocks Project. The project was con-ceived to help organizations build “right-fit” MEL systems that balance the need for information on impact with the need for operational data to inform manage-ment decisions. Acumen is spearheading another helpful project called the Lean Data Initiative, which is experimenting with more cost- and time-effective ways of collecting data.19 Finally, D-Lab and Root Capital’s Lean Research Initiative provides guidance in utilizing client-cen-tric approaches to field work (see Box 3 for more information). The briefing note on best practices in RAF impact mea-surement that accompanies this report suggests other useful resources.

19 For readers looking to learn more, it is worth mentioning that the Lean Data Initiative will be publishing a paper of findings and recommendations in the very near future.

tential exists for community lead-ers, partner organizations, or even spouses to influence the process through their presence at meetings or by making the farmer introduction (i.e., the farmer might feel an implied expectation to respond in certain ways). We attempted to temper this influence by: (i) communicating with farmers directly once given their contact information, (ii) conducting the interview without a local leader or partner organization staff present, (iii) spending the initial part of the conversation clarifying the purpose of the interview, and (iv) providing opportunities for the farmers to be in the “position of power” in interviews by allowing them to lead us, show us, take us around, and ask us questions.

• Mitigate distractions to the partic-ipant’s narrative: The team should pay attention to factors that may distract farmers from presenting their views accurately, including their past experience interviewing, to ensure responses best reflect the farmers’ true circumstances. In addition, the form of media capture chosen should not distract from the purpose above. For example, if video recording is necessary but puts farmers on edge, the team can use an approach that includes writing key insights from the farmers during the interview and have the farmer repeat these into the camera at the end of the interview.

• Revise approaches on an ongoing basis: Working in the field requires flexibility to adapt the interview pro-cess and content based on emerging findings. A field team should have daily check-ins to gauge the rele-vance and sensitivity of questions in the interview guide (in terms of both


Evaluation organizations are also helping practitioners with customized support and external evaluations. We were able to speak to two of them while collecting information for this report:

• The International Initiative for Impact Evaluation (3ie) funds impact evalua-tions of development programs. The organization brings together aca-demics and practitioners and offers a unique model that lets practitioners outsource the entire impact study process, with 3ie staff responsible for scoping and overseeing the studies done by external researchers.

• IDinsight brings together a team of highly qualified researchers who tai-lor rigorous impact evaluation meth-odologies to the decision-making priorities and constraints of practi-tioners in international development. Among other services, IDinsight regularly helps design and conduct impact evaluations, including RCTs.


CHALLENGE POTENTIAL SOLUTIONS

Des

ign

• Little consensus on what the best met-rics are for capturing impact - RAF interventions are often com-

plex with multiple inputs (e.g., agricultural loan and agronomic training) and intended impact* along multiple dimensions

- Metrics need to be adapted to the relevant context

• Look to the practices of peers and start a delib-erate movement toward harmonization**

• Consider the profile of your customer and what success looks like for him/her (i.e., at what stage in the “household priorities” chart is he/she?)

• Collect data on a number of variables, both proxy and direct

• Include metrics tailored for internal deci-sion-making needs

• Ethical concerns regarding use of con-trol groups• Ethical considerations around

depriving a community of what practitioners believe is a beneficial financial solution

• Provide compensation to excluded households (although this may skew results, so should be carefully designed)

• Use a phased roll-out approach

• Challenges to setting up and maintain-ing an effective control group, includ-ing:

• Spillover effects as treatment farm-ers share what they have learned with neighbors

• Drastic differences in agroclimatic con-ditions even a short distance away

• Frequently update the control group to ensure ongoing relevance (although this may intro-duce bias, so should be designed with care)

• Assign treatment at the village-level (or other similar level) instead of the household-level so that within-village spillovers do not bias results

• Capture spillover effects so that they can be ac-counted for in analysis

• Consider designs that compare different inter-vention arms without a pure control group, us-ing control groups only when truly required to determine whether intervention effectiveness is sufficient to justify implementation

• Misalignment in research agendas between providers and researchers, for example: - Researchers seek to measure

outcomes that are not priorities for providers

- Providers seek information on outcomes that require more time or resources than is feasible

- Providers desire results faster than research timelines allow

• Have frank discussions up front with all parties involved to align on a common set of goals and priorities (e.g. to inform provider actions in spe-cific context versus contribute to global body of evidence), and to clarify tradeoffs between design rigor and feasibility of implementation

• Try to build flexibility into the design and imple-mentation processes in order to be able to sat-isfy varying needs (e.g., option for study man-agement to transfer from external evaluator to the RAF provider after a certain period of time)

Table 7: Common challenges to measurement


CHALLENGE POTENTIAL SOLUTIONSIm

plem

enta

tion

• Low accuracy of self-reported metrics such as income - Recall typically generates unreliable

results (and most poor households do not maintain budget records)

- Complexity of a smallholder farm or rural household typically means there are multiple sources of in-come, that a significant portion of production is consumed on the farm, etc.

• Test accuracy of self-reported data by respon-dent, recall period and metrics requested be-fore implementing at scale

• Triangulate measurement of key outcomes us-ing different means

• Calibrate the reliability of metrics by taking multiple measurements (e.g., survey the same household at end of program and one month later)

• Review existing literature to understand which metrics and techniques (including new innova-tions) have worked and not worked previously for researchers in other contexts

• Consider alternative metrics and data sources including: - Proxy variables, e.g., using consumption ex-

penditure or a poverty indicator such as the Progress out of Poverty Index (PPI) instead of income

- Alternative data collection approaches, such as financial or farm diaries or methods to measure production directly

- Administrative data sources - Emerging digital sources of automatic data,

e.g., mobile phone top ups, digital transfers

• Data collection errors contribute to low-quality data - When data collection managed

internally, field staff often lacks train-ing in enumeration techniques

- Data often lost and/or entered in-correctly when enumerators during paper-based data collection

• Consider using external enumerators• Implement strict screening criteria for enumer-

ators• Incorporate enough time for enumerator

training (including field-based training) into the study timeline

• Utilize digital data collection methods

• IE constraints on operations - IE design locks the RAF provider

into the same treatment assignment and measurement methodology for the duration of the study

• Bring funders and other stakeholders on board early on to ensure that they are aware of how the study is likely to affect operations (e.g., program scale up may be slower than intended if it infringes on control areas)

• Find a way to limit the scope of the IE so it is not affecting the entire operation (e.g., by con-fining it to a specific area)

• During implementation, carefully monitor for and document any deviations to original treat-ment plan in order to incorporate into analysis plan

• Use staged roll-out plan that eventually pro-vides intervention to all participants

Table 7: Common challenges to measurement (cont'd)



plem

enta

tion

• High cost of data collection - Driven by lower population density

and poor roads in rural settings• Perception among many providers that

the cost of rigorous impact evaluations – in particular, RCTs - is prohibitive - External RCTs sometimes quoted in

the range of USD 1 – 2 million, and potential for other impact evaluation approaches (e.g., difference-in-dif-ference, propensity score matching) to be similarly or more expensive

- Levers for reducing costs are not widely known

• Utilize existing staff and systems for data collec-tion

• Implement new approaches to data collection including:

• Mobile surveys (SMS and/or IVR) - Leveraging existing data collection efforts,

e.g., partner with agro-dealers/agro-pro-cessors who already collect client data for their own business purposes (however it is important to invest in validating these existing data collection efforts, for example through spot-checking)

• Take steps to decrease data collection costs (see above)

• Consider changes to IE design to bring down cost, for example: - Reduce sample sizes by limiting number of

treatment arms (albeit with tradeoffs) - Target outcome variables with large effect

sizes—a longer time horizon can mean lower costs if effects sizes are greater

• Consider alternative implementation formats, e.g., get external help on designing the IE, but run certain components (e.g., data collection and entry) in-house

• Clearly define anticipated uses of evidence and limit IE scope to what is needed for these uses

• Potential to generate respondent fa-tigue - Surveys may eventually start to feel

excessively frequent and/or intru-sive to RAF clients

• Employ participatory evaluation approaches to engage respondents in MEL design process

• Limit survey length and frequency, surveying only as much as is absolutely necessary (in-cludes leveraging existing data collection ef-forts as described above)

• Design survey approach and timing to make participation as convenient as possible for re-spondents (e.g., consider phone surveys, visit households during non-work hours)

• Provide tokens of thanks to respondents for participation

• Train enumerators to establish a good report with respondents so that they are more likely to participate in follow-up surveys

• Respondent attrition over course of study - Numerous drivers of participant at-

trition including respondent fatigue, changes in contact information, changes in physical address, etc.

• Take steps to minimize respondent fatigue (see above)

• Collect sufficient contact information (cell phone numbers, email addresses, GPS coordi-nates of households, etc.) to facilitate follow-up

• Attempt multiple follow-up contacts• If repeated follow-up becomes expensive, sam-

ple a subset of dropouts and re-weight their responses during analysis to obtain a represen-tative sample




plem

enta

tion

• Competition for scarce field staff time - Field staff are often stretched thin

and juggling multiple responsibil-ities, so may be unwilling to spend time on MEL, which detracts from operations activities

- Can be exacerbated further with an externally-run IE, which requires field staff support but rarely factors it into the cost

• Get the buy-in of the field staff by explaining why data collection is important

• Align MEL closely with program implementa-tion, e.g., collect data during “off-cycles” for program tasks

• Estimate the extra human resources needs of an external evaluation and factor them into the cost of the study

• Use external enumerators

Resu

lts

• Discrepancies in ideal time horizon: - Program staff typically want results

quickly so that they can continuous-ly improve the design of their pro-grams, but some IEs (including, but not limited to RCTs) can, depending on scope and design, take several years to generate insights

• Design an MEL framework that combines data collection useful to both operations and impact evaluation

• Optimize for ideal time horizon during IE de-sign

• Establish an upfront plan to generate and com-municate interim findings

• Errors in data cleaning and analysis negatively affect results - Small errors can occur at many

points during the cleaning and anal-ysis process (such as an incorrect-ly-coded variable)

• Document all steps in the cleaning and analysis process

• Ensure that all scripts and documents are care-fully scrutinized

• Misinterpretation of results can lead to inaccurate recommendations. For ex-ample: - Variance in impact estimates (due to

sampling error) can be larger than the magnitude of the impact

- If multiple outcomes are evaluated, some of them will be “significantly different” across treatment and con-trol groups due to random chance

• Draft a ‘pre-analysis plan’ and submit it to a public registry for peer review (for example, 3ie’s free Registry for International Develop-ment Impact Evaluations)

• Low generalizability of findings: - A host of variables beyond the con-

trol of researchers can influence out-comes, e.g., implementation quality, the factors described in Figure 2

- It is very difficult to replicate the find-ings of a study in a different context

• Undertake study replication and look for pat-terns in findings

• Strive to understand the key success factors of an intervention, e.g., through qualitative re-search

• Don’t replicate an intervention/solution in a new context at scale—start small, test early, test often



BOX 3: Impact Evaluation That Creates Value for Participants

Some actors in the space are looking to take participatory evaluation fur-ther and actually create value for customers through the evaluation pro-cess. Practitioners are increasingly aware of the burden that evaluation places on survey participants and the effect that this has on the quality of responses. Hence, they are starting to think about ways to make survey participation useful for the customer.

In an effort to share ideas and encourage conversation around this top-ic, in June of this year, Root Capital published a working paper called “A Client-Centric Approach: Impact Evaluation That Creates Value for Partic-ipants.” The paper describes Root Capital’s experience with farmer skepticism and outright refusal to participate in impact evaluations, then goes on to lay out some principles of evaluation that create value for participants. A few interesting examples include:

Visit prospective research partners in person at their place of business, to listen to and understand their priorities.

Explore the possibility that research paradigms other than impact evaluation—such as customer feedback, market assessment, needs assessment, etc.—might better address clients’ priorities. These can be added to an evaluation to increase the value of the data-collection exercise for participants.

Note that these principles were developed in the context of Root Capital’s model, in which the “client” is not an individual farmer, but rather a producer organiza-tion or agribusiness that then on-lends to farmers. We believe that considerations around customer value creation are still relevant in a direct-to-farmer setting, but RAF providers may have to tailor the principles put forth by Root Capital to suit their operating environment.

(*) The paper is available at: http://www.rootcapital.org/sites/default/files/down-loads/2015-june_-_client_centric_approach_wp_final.pdf


A PATH FORWARD

As we have seen from the evidence, financial products certainly can but do not always generate impact for smallholders and other rural clients; there is still much to learn about what works, for whom, and in what setting. In particular, future research should fo-cus on understanding: 1) the impact of products/services that are relatively new or relatively understudied, but are rapid-ly gaining traction, such as digital trans-actions; 2) financial solutions, which may bundle multiple products and services, rather than just standalone products; and 3) additional nuance around the im-pact of customer profile, deployment approach, and contextual factors for all kinds of products and solutions.

Both external researchers and inter-nal MEL teams can contribute to filling these gaps in evidence. At its core, mea-surement helps practitioners learn more about how their programs work. Much of this measurement can be done in-house by qualified MEL staff and used to fine tune program design. In some cases, practitioners will benefit from partner-ing with external researchers. As external evaluators offer an additional degree of neutrality and expertise, they are often engaged for more complex evaluations and/or when the objectives of measure-ment extend to fundraising and/or policy change.

Funders also have an important role to play by dedicating more funds to RAF impact measurement. By doing so, they will not only foster more learning in this space, but also prevent diversion of ex-isting, valuable programmatic funding to MEL.

To be relevant and effective, this ad-ditional research should take into ac-count emerging best practices from measurement efforts to date. Specif-ic recommendations are laid out in the “Overcoming challenges to implemen-tation” section. Some more general sug-gestions for future research—equally val-id for all actors in this space—include:

1. Maintain a balance of rigor and rele-vance – measure what matters, rather than measuring for the sake of mea-surement, and do not let measure-ment supersede high quality service delivery;

2. Be flexible – consider what you can learn given tradeoffs involved and or-ganizational constraints, rather than simply focusing on what you want to learn;

3. Create value for all parties involved – clients should understand how the data they are asked to provide will be used to help them, field staff need to have a vision for how measurement will improve operations, etc.;

4. Reflect client priorities – identify the


customer segments you are target-ing, consult with clients to under-stand what success looks like from their perspective, and incorporate this into the MEL approach;

5. Look beyond the short-term impact – look to understand how intended and unintended effects unfold and endure over time.

The rural and agricultural finance com-munity will benefit from greater col-laboration around a common set of re-search goals going forward. By reading this report, every reader has taken the first, and possibly most important, step in that direction: becoming familiar with the evidence and knowledge accumulat-ed to date around the impact of RAF. In order to grow that knowledge and fill in the evidence gaps, the community must maintain its momentum. To that end, RAF stakeholders should explore formal and informal mechanisms for ongoing sharing and collaboration that will help to:

• Keep RAF actors up to speed on the most recent research findings,

• Identify more opportunities for col-laboration between researchers and practitioners,

• Improve aggregation of future find-ings by standardizing metrics,

• Disseminate emerging best practic-es, and

• Maintain a healthy dialogue around the merits of various approaches.


ANNEX I

Sample theories of change related to rural and agricultural finance

Source: ISF website (http://www.globaldevincubator.org/current-initiatives/smallholder-impact-risk-metrics/#Theory of Change)

INITIATIVE FOR SMALLHOLDER FINANCE


CARE PATHWAYS (MORE AGRICULTURE THAN FINANCE- ORIENTED)

Source: CARE Pathways to Empowerment website (http://www.carepathwaystoempowerment.org/theory-of-change/)

OPPORTUNITY INTERNATIONAL

Source: Opportunity International Agricultural Finance Program Impact Evaluation Design Report, July 2015


ASIAN DEVELOPMENT BANK

Source: ADB, Finance for the Poor: Microfinance Development Strategy, 2000 (available at: http://www.adb.org/sites/default/files/institutional-document/32094/financepolicy.pdf)

Source: Kim Siegal, One Acre Fund Director of Monitoring and Evaluation, Dec. 2015

ONE ACRE FUND


ANNEX II

Definitions of products and impact variables

PRODUCT DEFINITIONSPRODUCT DESCRIPTION

Insurance

Livestock insurance Covers a wide range of animals such as cows, goats, and poultry, primarily against mortality and injuries

Traditional crop insurance Covers farmers against various risks that may lead to crop loss such as nat-ural disasters and negative price fluctuations, among others

Index based insurance Covers farmers against shared rather than individual risk—such as risk as-sociated with weather fluctuations, disease, or price loss—and is tied to an objective parameter

Personal insurance Includes a range of insurance products such as health, life, and funeral in-surance; while not an agri-finance product, it has become an important add-on to typical agri-finance packages

Asset/Property insurance Provides financial compensation to the owner or renter of a structure and its contents in the event of damage or theft

Accident insurance Covers loss from accidental body injury

Savings

Regular current and sav-ings accounts

Typical bank accounts available to SHF’s; in some instances farmers are required maintain a minimum deposit or to deposit harvest money in their accounts

Compulsory/Mandatory savings

Savings accounts into which farmers are required to contribute money on a regular basis, allowing them to obtain loans at an agreed-upon interest rate

Informal savings Saving products offered through informal channels—village savings and lending associations (VSLAs), rotating credit and savings associations (ROSCAs), chamas, susus, etc.

Time/certificate/fixed de-posit

Savings accounts available to farmers that do not allow the withdrawal of money before a set date, typically during the planting season

Credit

Seasonal based loan Agricultural loans with repayments aligned to farmers’ cash flows, thereby taking into consideration the seasonal nature of farming


Asset financing Loans used to purchase agricultural assets, which then serve as the primary collateral for the loan; these loans cover a variety of assets including live-stock, farming equipment, and infrastructure such as greenhouses

Trade finance Loans provided to bridge the trade cycle funding gap between time of purchase and time of delivery (typically used by exporters, middlemen, ag-gregators, etc.)

Working capital loan Loans provided to farmers to help them finance their day-to-day operations

Warehouse receipt sys-tems

Act as a form of alternate collateral to enable farmers to access agricultural financing; warehouse receipts typically certify the quantity and quality of stored goods or commodities

Non-agriculture general loans

General loans that are also available to farmers, such as school loans and emergency health loans; in many cases, farmers turn to informal money-lenders to cover immediate needs

Transactional Finance

Money transfer servicesBill pay servicesCash collection

Money transfer products and services available to various players along the agricultural value chain; they can be categorized as business-to-cus-tomer (B2C), customer-to-business (C2B), customer-to-customer (C2C), government-to-business (G2B) and person-to-government (P2G)

SAMPLE METRICS OF IMPACT USED BY RESEARCHERS

INCOME CONSUMPTION

HH Income (log) Calories consumed/calories required

Total HH income Total monthly per capita consumption

Total HH Income from farming, self-employment, or labor

Meal consumption

Total average monthly income Log of total per capita HH expenditure over the 20 days prior to each survey

HH agricultural income Per capita consumption expenditure

Total household income from agricultural produc-tion (USD)

Annual HH consumption expenditure

Total HH Income from farm and non-farm activities, excluding remittances.

HH caloric availability / seasonality in consump-tion

Maize net revenue: value of all maize sales minus value of all maize purchases and interest payments

Percentage of HHs w/per capita expenditures un-der $1.25 per day

Non-farm income Per capita HH consumption

Business income Total HH consumption expenditure in last 30 days

Hypothetical ROI from fertilizer purchased

Business profits

Net sales (yearly revenue - input purchases)


ANNEX III

Pipeline of notable rural and agricultural finance impact studies

ORGANIZATION(S) STUDY METHODOLOGY INTERVENTION TYPE DATE

Village Enterprise, BRAC Research, IPA

Cash or Cash-Plus? Com-ponents and Variants of an Ultra-Poor Microenter-prise Development Pro-gram

RCT Group savings, grants, training

2013 – 2017

One Acre Fund One Acre Fund Quality of Life Study

Quasi-experimental Input credit, training, insur-ance

2015 – 2018 (ap-prox.)

Opportunity Inter-national, NORC

Agricultural Finance Pro-gram Impact Evaluation

Quasi-experimental Credit, sav-ings, insurance, non-financial in-terventions

2015 – 2017

Opportunity Inter-national, IPA

Research Study 7: RCT on mobile money and savings behavior

RCT Payments

Opportunity Inter-national

Research Study 8: Review effectiveness of AgroSave, ExtraSave, EduSave and other savings innovations

Savings 2015 (to be con-firmed)

Mercy Corps Mercy Corps Digital Fi-nancial Services RCT

RCT Payments and other mobile FS

tbc

MyAgro Expanding scope of M&E to look at broader impact of access to a layaway / savings program and im-pact on client financial ca-pacity

Quasi-experimental Savings 2015 on-wards

IDinsight Impact of access to land and tractor rental on credit

Credit


IDinsight Farmer profiles and the impact of cotton input credit

Credit

Oxfam R4 Rural Resilience Initia-tive Impact Evaluations in Senegal, Malawi, and Zambia

Mixed methods Credit, savings, insurance

N/A

IPA, GAIP, Ghanaian Ministry of Food and Agricul-ture, IFPRI, SARI1

Disseminating Innovative Resources and Technolo-gies to Smallholder Farm-ers in Ghana

RCT I n s u r a n c e , non-financial in-terventions

2014 – 2016

IPA, CARE Evaluating Village Savings and Loan Associations in Ghana

RCT Group savings 2008 – 2012

IPA, CARE Evaluating Village Savings and Loan Associations in Malawi


IPA, CARE Evaluating Village Savings and Loan Associations in Uganda


AGRA / 3ie Two forthcoming impact studies of agricultural fi-nance, linked to FISFAP program

RCT or Quasi-ex-perimental

TBD 2015 on-wards

Fund for Rural Pros-perity / RAF Learn-ing Lab

The Learning Lab will com-mission 1 – 3 rigorous im-pact studies on projects from the Fund for Rural Prosperity

RCT or Quasi-ex-perimental

Multi-product fi-nancial solutions

2015 on-wards


ANNEX IV

APPROACH TO FIELDWORKOur approach in each country was organized into four stages:

Develop a research plan: This includes developing a clear definition of the research ques-tion, working with field teams to recruit farmers from diverse backgrounds, and creating detailed interview guides tailored to each farmer group.

Kick-off meeting with the Learning Lab’s partner organization: The team met with field officers working with specific Learning Lab partner organizations. In Kenya, this was with One Acre Fund and in Ghana with Opportunity International. The RAF partners in each country provided insights into the local context, introductions to key support staff such as interpreters and agriculture extension officers, and the necessary cultural grounding to ensure that our engagement with farmers was appropriate.

Participatory field interviews: The team spent two days conducting in-depth interviews with farmers recruited through partner organizations, and two days with independently recruited farmers. These interviews included a set of questions based on interview guides, a home and a farm visit aimed to ensure a high degree of collaboration, and rich feedback throughout the process.

Development of farmer landscape: The team synthesized the findings from the field, them in a farmer landscape. The landscape articulated the farmers’ priorities by the stage of farm-ing in which they were engaged and also captured their future aspirations. This landscape provided an appropriate framing for how to understand farmers’ priorities and, in turn, refine how we think of impact.

IN-DEPTH INTERVIEW GUIDE

Introduction: (Why we’re here—what we’re trying to learn)

• Introduce yourself, the team members, and their roles

• The project and what you’d like to achieve from the interview

• Ask if it’s ok to document the session (pictures, video, and notes) and obtain video con-sent

Participatory research work plan and questionnaire


Farmer Introduction (Understanding farmer’s context)

• Introduction: Name, what do you do for a living, fun activities, level of education.

• Explore family & home: Number of household? Who else lives in the house and does what in the household? How long have you lived there?

Household and Lifestyle Context

• Day’s activities: What have you done today from AM to PM? How about your daily/week-end activities? (Probe: activities such as farmer group meetings, community based meet-ings/activities)

• Farm Tour (capture activities around their farm) - Learn about the home and farm: size, crop type, any animals, what they like about the home and farm

• Explore community engagement: Involvement in groups (Probe: farming, non-farming, religious), dynamics with neighbors, community leaders/influencers?

• How has this month been overall? This year? Compared to last year? Are things getting better/worse? Would your family / neighbors / community agree? (Probe: what’s changing, how things have changed, time frames)

Financial Context

• Income: Do you earn money from farming alone? Any other sources (animals, crafts, fami-ly money)? How often do you get paid? How do you receive the money? (Probe: regularity/irregularity, seasonality)

• Expenses: What are your major expenses? (Probe: total amount, types, regularity) How do you normally pay for them? How do you organize your money to pay major expenses? When do you pay them? Do you wish something could be easier when you pay your ex-penses or when you get paid?

• Portfolio: Are you using any products and services (formal and informal) that relate to managing your finances? (Probe: saving, borrowing, lending, investing, insurance, remit-tances, etc.) Why?

• Providers: How do you access your finances? (Probe: types, providers, first awareness and first use, access, frequency of use) How has it impacted your day-to-day life? What are your perceptions of formal institutions, costs vs. benefits, challenges in understanding services, communication, brand perception and loyalty? Do you have past experiences, motivators/triggers to start using a formal institution? Could you give a recent scenario?

• Mobile: Are you using any mobile money or banking services? Why did you pick the one they are using? Explore technology/services used: Sources of information, services used, literacy (Probe: farming related, finance related)?

• Ag finance: Have you ever used specialized ag finance products (e.g., input loans, crop insurance, etc.)? What is your take? Any improvements you have already seen in your day-to-day lives?

Impact:


• Have you experienced any impact/improvement within your life after you’ve accessed/used/received financial services?

• How about in your self/health/mind-state? family? Home? Community? Business? Farm? (Probe in depth for each of these, for all major services identified in the previous section)

Farm Context

• How was the last year of farming, best & worst seasons?

• Details of the best season that you’ve ever had? Who made/helped make that a success?

• Details of the worst season you’ve ever had? What’s one thing you wish you had during that bad time? What would you do to prevent it from happening again?

• When you had the above setback, how did you get/look for advice for a better result?

• Market: How do you learn about the best places to sell? Who usually buys your harvest? Do you have a payment plan?

• Workforce, equipment, and inputs: When do you use farming equipment? How old? How were they bought? Do you borrow or inherit farming equipment? What ways do you pay for the farming equipment? Where, when do you get fertilizers & seeds? How do you buy them? Borrow money? With savings? With government subsidies? Money from relatives? With loans? Do you buy them alone or in groups?

• Training and support: Have you ever received ag training? From where/who? How was it? What was good/bad about it?

Expectations (future executions and requirements for RAF)

• What are your plans for the next year in general? What are your hopes or expectations?

• What are your plans for your farm for next year?

• What do you want changed in the current RAF you are using? (Probe: using RAF cards)

• Now that you know how RAF works, which financial services will you desire to use for next year’s farm and household activities? (Probe: Insurance, microfinancing options, lending from SACCO or family)

• What issues should the financial services you will use address or solve?

• How do you prioritize your needs now that you have experienced both positive and nega-tive impacts on how you manage your finance?

• Given a chance to change mindset of other farmers about RAF services, what would you tell them? (Probe: a quote to other farmers)

• Looking back over your career/life, what’s one thing you wished you knew / were taught? How did you figure it out?


BIBLIOGRAPHY

Adebayo, C. O., Sanni, S. A., & Baiyegunhi, L. J. S. (2012). Microcredit scheme impact and food security status of beneficiaries in Kaduna State, Nigeria: A propensity score matching ap-proach. African Journal of Agricultural Research, 7(37), 5191-5197.

Anderson, Leigh, Chris Clark, Katie Panhorst Harris, Pierre Biscaye, Mary Kay Gugerty. (2015). “Evidence on the Impact of Rural and Agricultural Finance on Clients in Sub-Saharan Africa: a Literature Review,” Rural and Agricultural Finance Learning Lab Brief No. 1 / EPAR Brief No. 307.

Anderson et al. (2015) “Evidence on the Impact of RAF: Study Review Framework and Results Coding”

Angelucci, M., Karlan, D., & Zinman, J. (2013). “Win some lose some? Evidence from a random-ized microcredit program placement experiment by Compartamos Banco.” Institute for the Study of Labor (IZA) IZA Discussion Papers 7439.

Annan, J., Armstrong, M., & Bundervoet, T. (2011). “Urwaruka Rushasha: A randomized impact evaluation of village savings and loans associations and family-based interventions in Burun-di.” Evaluation Brief, International Rescue Committee, New York.

Aportela, F. (1998). Effects of financial access on savings by low-income people. MIT, mimeo.

Ashraf, N., Giné, X., & Karlan, D. (2009). Finding missing markets (and a disturbing epilogue): Evidence from an export crop adoption and marketing intervention in Kenya. American Jour-nal of Agricultural Economics, 91(4), 973-990.

Attanasio, O., Augsburg, B., De Haas, R., Fitzsimons, E., & Harmgart, H. (2011). “Group lending or individual lending? Evidence from a randomised field experiment in Mongolia.” Institute for Fiscal Studies IFS Working Papers W11/20.

Banerjee, A., Duflo, E., Glennerster, R., & Kinnan, C. (2015). The miracle of microfinance? Evi-dence from a randomized evaluation. American Economic Journal: Applied Economics 7(1): 22-53.

Bureau of Applied Research in Anthropology (BARA) and Innovations for Poverty Action (IPA) (2013). Final impact evaluation of the Saving for Change program in Mali, 2009–2012. Bureau of Applied Research in Anthropology, University of Arizona, and Innovations for Poverty Ac-tion. Evaluation commissioned by Oxfam America and Freedom from Hunger.

Beaman, L., et al. (2014). Self-Selection into Credit Markets: Evidence from Agriculture in


Mali (No. w20387). National Bureau of Economic Research.

Berhane, G., & Gardebroek, C. (2011). Does microfinance reduce rural poverty? Evidence based on household panel data from northern Ethiopia. American Journal of Agricultural Economics, aaq126.

Bruhn, M., & Love, I. (2009). The economic impact of banking the unbanked: evidence from Mexico. World Bank Policy Research Working Paper Series, Vol. 4981

Brune, L., Giné, X., Goldberg, J., & Yang, D. (2015). “Facilitating Savings for Agriculture: Field Experimental Evidence from Malawi.” (No. w20946). National Bureau of Economic Research.

Burgess, Robin, and Rohini Pande. (2005). Do Rural Banks Matter? Evidence from the Indian Social Banking Experiment. American Economic Review, 95(3): 780-795.

Burke, M. (2014). “Selling low and buying high: An arbitrage puzzle in Kenyan villages.” Re-trieved from http://web.stanford.edu/~mburke/papers/

Cai, H., Y. Chen, H. Fang, and L.-A. Zhou. (2009). “Microinsurance, trust, and economic devel-opment: Evidence from a randomized natural field experiment.” Working paper, Cambridge, MA: National Bureau of Economic Research.

Carter, M.R., Barrett, C.B., Boucher, S., Chantarat, S., Galarza, F. et al. (2008). “Insuring the Never Before Insured: Explaining Index Insurance through Financial Education Games”, BASIS Brief 2008–07: 1–8.

Cull, Robert, Tilman Ehrbeck and Nina Holle. (2014). “Financial Inclusion and Development: Recent Impact Evidence.” CGAP Publications. Crépon et al. (2014)

Davies, Rick and Jess Dart. 2005. “The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use,” available at: http://www.mande.co.uk/docs/MSCGuide.pdf

Dupas, P., & Robinson, J. (2013). Why don't the poor save more? Evidence from health savings experiments (No. w17255). National Bureau of Economic Research.

Fink, G., Jack, B. K., & Masiye, F. (2014). Seasonal credit constraints and agricultural labor sup-ply: Evidence from Zambia (No. w20218). National Bureau of Economic Research.

Gine, X., & D. Yang (2007). Insurance, credit, and technology adoption: Field experimental ev-idence from Malawi. World Bank Policy Research Working Paper 4425.Washington, DC: World Bank.

Hill, R. V., & Viceisza, A. (2010). An experiment on the impact of weather shocks and insurance on risky investment (No. 974). International Food Policy Research Institute (IFPRI).

Karlan et al. (2014)


Kim, J., Ferrari, G., Abramsky, T., Watts, C., Hargreaves, J., Morison, L., ... & Pronyk, P. (2009). “Assessing the incremental effects of combining economic and health interventions: the IM-AGE study in South Africa.” Bulletin of the World Health Organization, 87(11), 824-832.

Laker-Ojok, Rita and Godfrey Kayobyo. (2013). “An Impact Assessment of Opportunity Inter-national’s Agricultural Lending Program in Uganda, Malawi, and Ghana.” Opportunity Inter-national. Available at: https://opportunity.org/content/News/Publications/Knowledge%20Exchange/2014%20Opportunity%20International%20Agricultural%20Finance%20Impact%20Assessment.pdf

Matul, M. Dalal, A. De Bock, O. and Gelade, W. (2013). “Why People Do Not Buy Microinsur-ance and What We Can Do About It. Briefing Note 17.” Geneva: Microinsurance Innovation Facility.

McCreless, Michael. (2015.) “A Client-Centric Approach: Impact Evaluation That Creates Val-ue for Participants.” Working paper. Root Capital. Available at: http://www.rootcapital.org/our-approach/publications

MyAgro. (2014). Internal Impact Data. Obtained directly from the organization.

Norton, M., Osgood, D., Madajewicz, M., Holthaus, E., Peterson, N., Diro, R., ... & Gebremi-chael, M. (2014). Evidence of demand for index insurance: experimental games and commer-cial transactions in Ethiopia. Journal of Development Studies, 50(5), 630-648.

Nuru International. (2014). “Ethiopia Agriculture Program Impact Assessment.” Available at: http://www.nuruinternational.org/wp-content/uploads/2014/06/2014-Nuru-Ethiopia-Agricul-ture-Program-Impact-Assessment.pdf

Nuru International. (2014). “Kenya Agriculture Program Impact Assessment.” Available at: http://www.nuruinternational.org/wp-content/uploads/2014/06/2014-Nuru-Kenya-Agricul-ture-Program-Impact-Assessment-20141219.pdf

Nuru International. (2014). “Kenya Financial Inclusion Program Impact Assessment.” Available at: http://www.nuruinternational.org/wp-content/uploads/2014/06/2014-Nuru-Kenya-Finan-cial-Inclusion-Program-Impact-Assessment-20141219.pdf

One Acre Fund. (2014). Annual Impact Country Report. Available at: https://www.oneacre-fund.org/uploads/all-files/2014_Annual_Impact_Country_Report_One_Acre_Fund_.pdf

One Acre Fund. (2015). Program Evaluation: Analysis and Results. Available at: https://www.oneacrefund.org/uploads/all-files/One_Acre_Fund_Program_Evaluations_IDInsight.pdf

One Acre Fund. (2015). One Acre Fund Evaluation: Assessment of Evaluation Limitations. Available at: https://www.oneacrefund.org/uploads/all-files/One_Acre_Fund_Assessment_of_Evaluation_Limitations_IDInsight.pdf

One Acre Fund (2014). Randomized Control Trials. Available at: https://www.oneacrefund.org/


uploads/all-files/Randomized_Controlled_Trials_One_Acre_Fund__2014.pdf

Owuor, G. (2009). “Is micro-finance achieving its goal among smallholder farmers in Africa? Empirical evidence from Kenya using propensity score matching”: paper presented at the XXV11 International Conference of Agricultural Economists, Beijing, China. August, 16-22, 2009.

Schaner, S. (2015). “The persistent power of behavioral change: Long-run impacts of tempo-rary savings subsidies for the poor.” Working paper. Dartmouth College: Hanover, NH.

Shimeles, A. (2010). Community based health insurance schemes in Africa: The case of Rwan-da. Working paper No. 120. African Development Bank.

Tarozzi, A., Desai, J., & Johnson, K. (2001). The impacts of microcredit: Evidence from Ethio-pia. American Economic Journal: Applied Economics, 7(1), 54-89.

Tobacman et al., 2015

Village Enterprise. (2010). “Kenya Longitudinal Impact Assessment”. Available at: http://vil-lageenterprise.org/wp-content/themes/village/pdf/village-enterprise-2010-longitudinal-im-pact-assessment-report-final.pdf

Village Enterprise. (2008). “East African Microenterprise Development: A Longitudinal Impact Assessment of Village Enterprise Fund.” Available at: http://villageenterprise.org/wp-content/themes/village/pdf/vef-impact-assessment.pdf

Village Enterprise. (2011). “Randomized Control Trial: Cash or Cash-Plus? Components and Variants of an Ultra-Poor Microenterprise Development Program”. Available at: http://villa-geenterprise.org/wp-content/uploads/2011/05/RCT-2-Pager-Updated-2015.pdf

Yilma, Z., van Kempen, L., & de Hoop, T. (2012). A perverse ‘net’ effect? Health insurance and ex-ante moral hazard in Ghana. Social Science & Medicine, 75(1), 138-147.

Zukoski, Ann and Mia Luluquisen. (2002). “Participatory Evaluation: What is it? Why do it? What are the challenges?” Partnership for the Public’s Health.

WWW .RAFLEARNING .ORG

Documents

agricultural finance - RAFLearning · and agricultural finance 2. Assess the current evidence base for the impact of financial products on smallholder and rural households, particularly