Beatrice Rogers and Jennifer Coateswith:

Johanna Andrews, Alexander Blau, Ameya Bondre, Jamie Fierstein, Kathryn Houk, Tina Galante, Carisa

Klemeyer, Elizabeth Kegode, Leslie Sanchez

“BUT WE WILL ALWAYS BE HERE!”ASSURING SUSTAINABLE BENEFITS

AFTER FOOD AID PROGRAMS

Exit Strategy Study Overview2

Study Rationale3

Title II programs closing in non-priority countries

Little systematic knowledge of whether program impacts are maintained and how to maintain them

Effectiveness of Title II programs depends on both short-term impact and long-term sustainability

Immediate and long-term impact may be trade-offs

Study Objectives4

1. Determine the extent to which activities, outcomes, and impacts of Title II programs were sustained.

2. Identify program characteristics that make it possible to sustain program activities and effects after the program shuts down.

3. Assess how the process of “exiting” affects sustainability.

4. Provide guidance to future programs on how to exit while ensuring sustainability.

Key Concepts5

Sustainability Sustainability of

impacts among program participants

Diffusion of impacts to new populations

Sustainability/ expansion of activities

Exit From specific

activities By the Title II

Awardee from the area

Key Concepts: Exit Strategy/Sustainability Plan

6

Explicit plan for transition from program support to exit

Often called “sustainability plan” in agency documents

Specifies approach to exit: phase over (to whom), phase out (of resources)

Should include timeline and allocation of responsibility for ensuring progress

Study Methods Overview7

1. Qualitative picture of exit strategy implementation around time of agency exit

2. Qualitative inquiry one year after exit: to explore the success of exit strategy in leading to sustainable service provision (where applicable) and beneficiary uptake

3. Qualitative and quantitative data two years after exit to assess sustainability of impacts and outcomes

4. Compare quantitative follow-on data to agency endline survey data to determine sustainability and link to exit strategies

Analytic Strategy: Associating Exit Strategy with Sustainability of Impacts

8

Agency evaluations did not include control groups for attribution of impact

Associations based on: Testing Implementation Pathways

Sustainability of desired behaviors and impacts Continuation of service provision and utilization Implementation of an exit strategy

Use of Secondary Data To serve as a point of comparison for survey results

Study Locations• Kenya• Bolivia• Honduras• India

Selection Criteria• Programs achieved key

impacts on food security and malnutrition

• Programs implemented exit strategies

• Close-out coincided with study time frame9

Conceptualizing Sustainability and Exit

10

Sustained Impact

Sustained Behaviors and/or Service Utilization

SUSTAINED

SERVICE DELIVERY

SUSTAINED

ACCESS

SUSTAINED

DEMAND

SUSTAINED

RESOURCES

SUSTAINED

CAPACITY

SUSTAINEDMOTIVATIO

N

SUSTAINED

LINKAGES

Program Exit StrategiesProgram Exit StrategiesEXTE

RN

AL F

ACTO

RS

11

Critical combination of resources, motivation, and technical and managerial/administrative capacity

Programs should be designed with exit in mind Gradual transition to independence with a period of

independent operation before exit Importance of linkages - variable Impact at exit is not the same as sustained benefit later

on Provision of free resources poses challenges for

sustainability Depending on government linkages is risky, and

depending on newly entering NGOs poses questions Different technical sectors face different challenges

applying these principles

Emerging Conclusions

RESULTS - BOLIVIA12

13

Title II Program Areas in Bolivia

14

2002 20112008-9

Agencies collect baseline

quant data

Tufts collects

post-program

qualitative data

2010

Tufts collects post-program qualitative and

quantitative datafollow-up

quantitative survey replicates

agencies’ endline surveys

2004

Agencies collect mid-term quant and qual

data

Agencies collect endline

qual and quant data

Study Methods Overview

15

1. Maternal and Children Health and Nutrition (MCHN)

2. Water and Sanitation (WatSan)

3. Agriculture/Rural Income Generation (RIG) (after mid-term evaluation change in focus from production to commercialization)

4. Natural Resource Management (NRM)

Program Technical Sectors

16

Motivate CHWs to continue providing GMP, health promotion and home visits through social prestige and respect; credentialing.

Link CHWs to decentralized health services to assure supervision, training, resources

Teach mothers to substitute local foods for Title II provided foods

MCHN Exit StrategyMCHN Exit Strategy

17

Prevalence of stunting declined substantially during Title II and generally remained low after exit: ADRA, CARE, SC: No significant

change in stunting since exit FH: Stunting continued to decline

significantly after exit While overall stunting in the

Departments declined during the period of Title II implementation, the Awardee declines were larger than regional declines and Awardees worked in areas with much higher initial rates of stunting

MCHN Results: Stunting

18

Departmental statistics (< 5y)Title II Communities (3–35 mo)

Percentage of children stunted (HAZ)

Stunting

19

GMP participation and prenatal care visits remained high Success of linkages to national government programs:

Zero Malnutrition (ZM): nutritional supplements for children under 2; Comprehensive Nutrition Units (UNIs) at healthcare centers

Bono Juana Azurduy (BJA) conditional cash transfers for completing prenatal/postpartum care and growth monitoring visits

Awardees with both a significant increase in GMP and the strongest malnutrition impacts, CARE and FH, continue to operate after exit

GMP participation declined in SC communities, where fewer governmental and NGO resources may be available

MCHN Results: Demand for ServicesGrowth Monitoring and Prenatal Care

20

Percentage of households taking child < 5y to GMP

Growth Monitoring Participation

21

Percentage of mothers reporting any prenatal care during last pregnancy

Prenatal Care

22

Percentage of mothers reporting prenatal care in first 5 months during last pregnancy

Prenatal Care (< 5mo)

23

Presence of a community health worker remains high

Coherent governmental and NGO initiatives likely sustaining CHW presence: Intercultural Family and Community

Health Program (SAFCI) Tarija’s “Health Guards” NGOs (Esperanza Bolivia, Plan

International) adapt CHWs for their own program activities

MCHN Results: Service Delivery

24

FH collaborates with UNICEF in some former Title II communities in malnutrition prevention and GMP, with funding from Spain and USAID (Proyecto Integrado de Seguridad Alimentaria, PISA)

In some cases local incentives may sustain CHWs Incorporation into water committee;

profit-generating women’s groups

MCHN Results: Service Delivery

25

Percentage of communities reporting CHW presence

Community Health Workers (2011)

26

EBF increased during the program and has been sustained No cost; reinforced bygov’tprograms

(BJA, ZM) All other practices have declined since exit

Proper treatment of diarrhea, hygiene Importance of reinforcing behavior change

messages: Behaviors not reinforced through home

visits/trainings since exit Not prioritized since health system targets

services like prenatal visits/GMP

Results: MCHN Practices

27

Title II Communities National Statistics

Percentage of mothers reporting exclusive breastfeeding for child < 6mo

Exclusive Breastfeeding

28

Diarrhea Treatment (food and liquid)

Percentage of mothers offering same/more food to child during diarrhea

Percentage of mothers offering same/more liquids to child during diarrhea

29

Percentage of mothers offering same/more food , liquid, or ORT to child during last diarrhea episode

Diarrhea Treatment (food, liquid, and ORT)

30

Percentage of children 12–23 months of age receiving the third dose of DPT or pentavalente vaccine

Note: No data available for ADRA in 2002 and 2004

Vaccinations

31

Sustainability of impacts and CHW service use due to alignment with government health programs and presence of NGOs for continued support ANC and GMP remain high since exit

Behavior change harder to sustain once food rations removed Rations had been provided children < 35 months and

pregnant women participating in GMP/health talks Use of health services remains high since exit, but the

lack of resources for community training and CHW supervision appears to limit the quality of information provided or the incentive to continue household-level behaviors requiring time/resources

MCHN Summary

32

Piped water and latrinesAwardee provided inputs for construction of piped water and latrinesElected water committees trained in system maintenance and repair, and in financial management and administration.Beneficiaries provided labor for construction, pay a connection fee and a monthly fee for water use; water cut off for lack of payment;User fees cover maintenance and repairsSome water committees were operating prior to Title II but received training

WatSan Exit Strategy

33

Vertical linkages to municipal government and horizontal linkages among water committees were part of sustainability plan, not implemented

Water quality testing Water committees would take over water quality testing

once awardees left

Hand washing and latrine use Hygiene behaviors promoted by community health

workers in home visits would continue

WatSan Exit Strategy

34

Infrastructure continues to be available in Awardee areas Percentage of communities reporting piped water and

latrine availability generally maintained Immediate tangible benefit; responds to priority need;

successful water committee model; infrastructure coverage is government priority

Sanitation and handwashing practices were not sustained after exit Water quality testing also not maintained

Results: WatSan

35

Percentage of households with piped water

Title II Communities Departmental Statistics

Piped Water

36

Percentage of communities maintaining their own water system (2011)

Water System Maintenance

37

Percentage of households with latrine

Latrine Access

38

Percentage of population using hygienic sanitation facilities (with signs of use, Awardee criteria)

Latrine Use

39

Percentage of caregivers (responsible for food preparation) with proper handwashing practices (awardee criteria)

Handwashing Practices

40

Percentage of caregivers using soap/detergent

Note: No data available for CARE

Soap Use

41

Percentage of households using any water treatment (boiling, chlorination, SODIS)

Note: No data available for FH

Water Treatment

42

Water and sanitation infrastructure demonstrated greater sustainability than hygiene behaviors and water purification Infrastructure: Tangible benefit, self-financing user

fees, community capacity to operate system and maintain accountable administration; not dependent on linkages

Behaviors: No immediate, tangible benefit, no financial incentives to motivate implementation, no national program focused on such behaviors, since national priorities are more concerned with visible infrastructure coverage

Water quality: no independent operation before exit

WatSan Summary

WatSan Sustainability!

43

44

Rural Income Generation (RIG) Exit Strategy

Model farmers were given free agricultural inputs and training in return for training other farmers

Model farmers would continue to serve as technical resources

Focus on improved production and diversification shifted to commercialization after MTE

Farmers associations formed, trained in production and in management, contracting

Profits from increased sale would motivate sustained use of practices taught in the program, and profits would cover purchase of inputs

45

Results Rural Income Generation (RIG)

Agricultural income of farmers increased throughout the program, but fell substantially at follow-up, though remaining substantially higher than baseline Inputs provided free during program now must be

purchased

Farmers who are members of producer associations (PA) have maintained substantially higher incomes than those who received training but are not PA members

National GDP per capita continues to rise according to World Bank indicators

46

Mean Annual Income from Agricultural Sales by Farmers Trained in the Program

(adjusted for inflation, 2011=100)

Agricultural Income

47

Mean Annual Agricultural Income: Producer Association Members vs. Non-Members

(2011)

48

Percentage of trained farmers belonging to a producer association

Note: No endline data available for ADRA and FHCARE participation data are only for farmers in promoted value chains

Producer Association Membership

49

Percentage of trained farmers adopting improved agricultural practices (Awardee criteria)

Improved Agricultural Practices

50

Percentage of farmers in producer associations is declining

Members of producer associations have substantially higher incomes than non-members

Qualitative evidence indicates that farmers may drop out of associations because they are unable to meet quality standards

More successful farmers may be those able to afford inputs through the profits from marketing

Commercialization model is successful for those who can take advantage of it

RIG Results Summary

51

Use of inputs and promoted practices declining Inputs provided free during life of program now

must be purchased; farmers may have become accustomed to receiving these inputs without charge

Practices sustained by trained farmers are those returning noticeable benefit and low cost (organic fertilizer, crop rotation)

Proportion of farmers adopting improved practices similar among trained and un-trained farmers Individual producers likely copying improved

production methods

RIG Results Summary

52

Sales through an association more likely where associations continue to receive support from external sources (NGOs, foundations, government)

Government partners have high rates of turnover, changing priorities, stretched budgets, and shortages of technical staff; partnerships with buyers are more reliable

Sustained operation of PAs is more likely when responsibility of negotiating contracts is transitioned gradually, with a period of independent operation

RIG Results Summary

53

Practices have declined significantly since exit Food for work and free inputs no longer available Motivation is low if tangible benefit is not perceived

Continued practices are those that produce tangible benefit and do not require purchased inputs

Municipal Natural Resource units are underfunded and understaffed

Some activities produced lasting change during the program, but were not continued (forestation, terracing)- phase out

Results: Natural Resource Management

54

Trained Farmers Adopting at Least Three NRM Practices

55

NRM activities largely not sustained after exit

Technical capacity was strong; resources and motivation lacking

Municipal support was weak or lacking

Withdrawal of food and free inputs jeopardized sustainability of activities

NRM Results Summary

Conclusions56

Conclusions57

Impact at exit does not consistently predict sustained impact two years later.

There are specific ways to increase the likelihood of sustainability.

Provision of free resources poses risks to sustainability.

Impact Assessment at Exit Can Be Misleading

58

Impact assessment at exit does not consistently predict impact two years later.

Many activities, practices, and impacts across sectors declined over the two years after exit.

These declines are related to inadequate design and implementation of sustainability strategies and exit processes.

Sustaining service provision and beneficiary utilization of services and practices depends on three critical factors:1. Resources 2. Technical and Management Capacity3. Motivation

There are often synergies among these three elements.

Best practice models have emerged for each – and they are often sector-specific.

The relevance of linkages is sector-specific. Gradual exit and independent operation are

necessary (though not sufficient) for sustainable results.

Specific Ways to Increase the Likelihood of Sustainability

59

Provision of Free Resources Poses Risks to Sustainability

60

Withdrawal of food rations or any other free input (as incentive) jeopardizes sustainability without consideration of substitute incentives.

Provision of free food rations risks creating unsustainable expectations.

Consideration of alternative incentive structure must be incorporated into program design.

Beneficiaries receiving free inputs to support program activities may not be willing or able to replace them once project ends.

MCHN: Exit Strategy Models Lacked Sustained Sources of Resources, Capacity, and Motivation61

Motivation: Withdrawal of food was a disincentive for participation in and provision of growth monitoring. In Kenya, withdrawal of food resulted in

reduced participation in growth monitoring. NGO return also reinstated the incentive.

In Honduras, withdrawal of food reduced demand participation in growth monitoring, while in Bolivia, the government implemented a cash incentive for growth monitoring and promotion.

Overall community health worker (CHW) service provision declined in Kenya, Bolivia, and Honduras with decline in material incentives and demand.

MCHN: At least four types of resources must be considered in order to sustain CHW service

delivery62

1. Resources that helped CHWs do their job, such as weighing scales, report forms, and bicycles, as well as training and supervision from the health sector

2. Resources that they offered the community as an incentive to participate in activities

3. Benefits accrued to the CHWs that incentivized their service, such as access to goats distributed to women’s goat groups

4. CHW time and its converse, the opportunity cost of time spent on CHW activities rather than on other productive labor

• No fee for service models were observed in the health sector. • Linkages to government did not fulfill all resource needs..

MCHN: Success of linking to government depended on government capacity and resources63

Bolivia: Some CHWs continued linkages to health system; public health system provided services.

In Honduras, linkages with health system worked until government resources ran out.

In Kenya, government did not/could not take the responsibility for supervision, training, or provision of resources resulting in declines in CHW service quality, motivation, and capacity when they were not being reinforced.

Water system exit strategy demonstrated that motivation, capacity, and resources are all critical

to sustainability64

Beneficiaries are motivated to pay for water services that are reliable, convenient, and abundant.

Technical and management capacity of water committees permits system to continue operation post-project.

Fees paid by consumers provide ensured resources for system maintenance.

WatSan: Linkages to Government Not Always Needed

65

Linkages were actively avoided by most water committees; independence from government entities that could be unreliable sources of support

Sustainability of Water Qua lity Provides a Counterexample….

66

Motivation for water quality testing is low because benefits are not visible.

Therefore willingness to pay was low.

Capacity-building activities during the DAPs did not emphasize water quality.

Water quality testing was not operating independently at exit.

No linkages to support water quality testing were in place at exit.

Agriculture Sector Exit Strategies Yielded a Mixed Sustainability

Story67

Model farmers (extension farmers) lacked motivation, resources, and reinforcement of capacity to continue providing technical assistance after incentives were withdrawn. Service provision largely declined.

Producer associations participation rates were variably low/declined. PAs provided concrete benefits in some countries but not all.

Commercialization activities produced sustainable benefits for those engaged in them.

Improved agricultural practices generally declined when requiring resources provided during the

DAP68

Free inputs were provided during program life

and

Use of these inputs declined when they needed to be purchased after the program ended

Engagement in NRM activities and practices declined dramatically after

program exit69

Food as pay (FFW) was withdrawn.

Resources in the form of inputs (seedlings, etc.) was no longer provided free of charge.

Motivation was lacking unless costs were low and direct benefits to farmers were high.

Linkages to local government environmental groups were ineffective as those groups lack resources.

Positive changes (reforestation, terracing, etc.) were maintained, though activities to expand these changes were not.

COSAMO in Kenya Was a Model of Sustainability

70

Sustained capacity, motivation, and resources. Sustained beneficiary utilization and impact.

Self-financing: No outside seed money and no other external resources needed.

Modular training program built solid technical and managerial capacity.

There was a gradual withdrawal of NGO after a period of COSAMO group independent operation

Linkages were not necessary – could pay for technical assistance of trained resource persons.

Recommendations71

Program Design72

Sustainability should be built into the design of programs from the beginning.

Plans must include: decision about approach (phase out, phase over); explicit benchmarks for progress; timeline; clear allocation of responsibility, graduation criteria

Each element — capacity, motivation, and resources — must be considered, with redundancies and contingencies for external shocks built into the plan.

Project Cycles Should Be Longer and Incorporate Sustainability Benchmarks

73

Phased Program Implementation

74

Sustainable design and initial service delivery, demand creation, and partnership formation

Strengthening capacity, developing ensured resources, ensuring appropriate linkages, promoting independent operation

Independent operation of program activities and gradual agency withdrawal

Program Monitoring and Evaluation

75

Emphasis on assessment of impacts (e.g., reduction in stunting) can undermine focus on sustainability.

Progress toward sustained capacity, motivation, and resources should be measured as process indicators during monitoring.

Evaluations should be done at each phase; criteria for assessment will differ by the objectives of the phase.

Program Withdrawal (Exit)76

Phase-over of responsibility must be gradual.

Groups should be operating independently (with agency backstopping) before full withdrawal.

Community organizations and individuals should be aware of their post-exit roles and responsibilities from the outset.

Philo s o phy of sustainability should be embedded throughout implementation to withdrawal.

Be aware that communities may not commit to independence, in the expectation of new sources of external support.

This study is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases, and Nutrition, Bureau for Global Health, and the Office of Food for Peace, Bureau for Democracy, Conflict and Humanitarian Assistance, U.S. Agency for International Development (USAID), under terms of Cooperative Agreements GHN-A-00-08-00001-00, AID-OAA-A-11-00014, and AID-OAA-A-12-00005 through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360.

The contents are the responsibility of Tufts University and do not necessarily reflect the views of USAID or the United States Government.

77

Acknowledgments and Disclaimer

THANK YOU!