Interim Report Final Draft: August 22, 2012search.oecd.org/countries/elsalvador/report-102012-evaluation-slv... · Interim Report Final Draft: August 22, 2012 Randall Blair Larissa

Impact Evaluation Findings after

One Year of the Productive and

Business Services Activity of the

Productive Development Project,

El Salvador

Interim Report

Final Draft: August 22, 2012

Randall Blair

Larissa Campuzano

Lorenzo Moreno

Seth Morgan

Contract Number:

MCC-10-0114-CON-20/TO03

Mathematica Reference Number:

06918.800

Submitted to:

Millennium Challenge Corporation

875 Fifteenth Street NW

Washington, DC 20005-2221

Project Officer: Rebecca Goldsmith

Submitted by:

Mathematica Policy Research

1100 1st Street, NE

12th Floor

Washington, DC 20002-4221

Telephone: (202) 484-9220

Facsimile: (202) 863-1763

Project Director: Larissa Campuzano

Impact Evaluation Findings after

One Year of the Productive and

Business Services Activity of the

Productive Development Project,

El Salvador

Interim Report

Final Draft: August 22, 2012

Randall Blair

Larissa Campuzano

Lorenzo Moreno

Seth Morgan

iii

ACKNOWLEDGMENTS

We greatly appreciate the hard work of many people whose efforts contributed to this report. We especially thank our monitoring and evaluation colleagues at MCC and FOMILENIO: Rebecca Goldsmith, Lola Hermosillo, Claudia Argueta, Xenia Panameño, and Moisés Díaz. Their input and support at all stages of the project have been invaluable. FOMILENIO employee Isabel Rodríguez also deserves special recognition for her excellent assistance with data collection and monitoring. We also thank Chemonics employee William Mejía for his prompt assistance with administrative data and information requests.

Our colleagues at Mathematica have provided guidance and suggestions throughout the project, particularly Ken Fortson, Hanley Chiang, and John Deke, who have offered valuable insight on several methodological issues. We also thank Jackie McGee for her diligence and patience in formatting this report. Last but definitely not least, Laurie Schulte does a wonderful job tracking and managing the project staffing and resources.

All of the above organizations and individuals have contributed tremendously to this report and to the project more generally, but any remaining errors are the fault of the authors alone.

iv

CONTENTS

EXECUTIVE SUMMARY ................................................................................................... x

I BACKGROUND ............................................................................................... 1

A. Summary of the PDP and PBS Activity ....................................................... 1

B. Design of the PBS Activity ........................................................................ 2

C. Implementation of the PBS Activity .......................................................... 3

D. Costs of the PBS Activity .......................................................................... 7

E. Interim Findings on PBS Implementation .................................................. 8

F. Evaluation Rationale ................................................................................ 9

II EVALUATION DESIGN ................................................................................... 10

A. Design Summary ................................................................................... 10

B. Evaluation Questions ............................................................................. 11

C. Evaluation Design Implementation ......................................................... 12

D. Outcomes and Data Sources .................................................................. 15

E. Design Limitations ................................................................................. 20

III STUDY SAMPLE ............................................................................................ 21

A. Description of the Evaluation Sample ..................................................... 21

1. Demographic Characteristics: Producers .......................................... 24

2. Demographic Characteristics: Households ....................................... 26

B. Baseline Equivalence .............................................................................. 26

C. Participation in the PBS Activity .............................................................. 27

IV IMPACTS OF PBS ASSISTANCE ....................................................................... 34

A. Impacts of PBS on Productive Practices .................................................. 34

B. Impacts of PBS on Producer-Level Income, Employment, and

Production ............................................................................................. 37

1. Dairy Value Chain ............................................................................ 37

2. Horticulture Value Chain ................................................................. 40

3. Handicrafts Value Chain .................................................................. 45

Contents Mathematica Policy Research

v

4. Comparison of Intent-to-Treat and Treatment-on-the-

Treated Impact Estimates ................................................................ 46

C. Impacts of PBS on Household Income, Consumption, and

Poverty .................................................................................................. 48

D. Economic Rates of Return ...................................................................... 51

V CONCLUSIONS ............................................................................................. 53

A. Key Findings .......................................................................................... 53

B. Discussion ............................................................................................. 53

C. Next Steps ............................................................................................. 54

REFERENCES ............................................................................................................... 55

APPENDIX A SELECTION CRITERIA FOR PBS ASSISTANCE

APPENDIX B CONTEXTUAL INFORMATION, SENSITIVITY TESTS, AND

EXPLORATORY ANALYSES

APPENDIX C QUALITATIVE FINDINGS FOR THE DAIRY CHAIN

APPENDIX D ADDITIONAL BASELINE AND IMPACT TABLES

vii

TABLES

I.1. Comparison of Phase I and Phase II PBS Assistance, by Value Chain ............... 6

I.2. PBS Service and Outcome Targets and Results ............................................... 7

I.3. Costs of the PBS Activity (In U.S. Dollars) ........................................................ 8

II.1. Key Dates in the PBS Impact Evaluation ........................................................ 13

II.2. Definitions of Main and Intermediate Outcomes, PBS Impact

Evaluation .................................................................................................... 16

II.3. Producers in the PBS Impact Evaluation, by Value Chain ............................... 17

III.1. Baseline Information on Producers, by Value Chain (Averages in U.S.

Dollars Unless Otherwise Indicated) ............................................................. 24

III.2. Baseline Information on Households, by Value Chain (Averages in

U.S. Dollars Unless Otherwise Indicated) ...................................................... 26

III.3. Crop Cycle Durations and Production/Sales Potential ................................... 28

III.4. Treatment Group Participation in PBS Activities During the Study

Period, by Value Chain and Implementation Phase ....................................... 33

IV.1. Impact of PBS on Participants’ Productive and Marketing Practices,

by Value Chain (Percentages unless Otherwise Indicated) ............................. 36

IV.2. Impact of PBS on Dairy Producers’ Investment, Income,

Employment, and Production (in U.S. Dollars Unless Otherwise

Indicated) .................................................................................................... 38

IV.3. Impacts of PBS in Phase I and Phase II for Dairy Producers (In U.S.


IV.4. Impact of PBS on Horticulture Producers’ Investment, Income, and

Employment (in U.S. Dollars Unless Otherwise Indicated) ............................. 41

IV.5. Impact of PBS on Horticulture Producers’ Agricultural Production

(Averages Unless Otherwise Indicated) ......................................................... 43

IV.6. Impacts of PBS in Phase I and Phase II for Horticulture Producers (in

U.S. Dollars Unless Otherwise Indicated) ...................................................... 44

IV.7. Impact of PBS on Handicraft Producers’ Investment, Income,

Employment, and Labor (in U.S. Dollars Unless Otherwise Indicated) ............ 45

IV.8. Impacts of PBS at the Producer Level, by Value Chain (in U.S. Dollars

Unless Otherwise Indicated) ......................................................................... 47

IV.9. Impact of PBS on Annual Household Income, Consumption, and

Poverty, by Value Chain (in U.S. Dollars Unless Otherwise Indicated) ............ 49

Tables Mathematica Policy Research

viii

V.10. Impacts of PBS at the Household Level, by Value Chain (in U.S.


B.1. Description of Sensitivity Tests, Dairy Chain ............................................... B.4

B.2. Estimated Impact of PBS on Net Productive Income, by Baseline Net

Income (in U.S. Dollars) ............................................................................... B.4

D.1. Producers’ Baseline Demographic Information, by Value Chain

(Averages in U.S. Dollars Unless Otherwise Indicated) ................................ D.2

D.2. Baseline Annual Household Income, Consumption, and Poverty, by

Value Chain (Averages in U.S. Dollars Unless Otherwise Indicated) ............. D.3

D.3. Additional Impacts of PBS on Dairy Producers’ Practices and

Technology Adoption (Percentages), 2011 ................................................. D.4

D.4. Impact of PBS on Dairy Producers’ Investment, Income, and

Employment (Averages Unless Otherwise Indicated) ................................... D.5

D.5. Additional Impacts of PBS on Horticulture Producers’ Practices and

Technology Adoption (Percentages), 2011 ................................................. D.6

D.6. Impact of PBS on Horticulture Producers’ Investment, Income, and


D.7. Impact of PBS on Horticulture Producers’ Production and Sales

(Averages Unless Otherwise Indicated) ....................................................... D.8

D.8. Impact of PBS on Handicraft Producers’ Investment, Income, and


D.9. Impact of PBS on Female- versus Non-Female-Headed Households in

the Handicraft Value Chain (in U.S. Dollars) .............................................. D.10

D.10. Additional Descriptive Statistics on Handicrafts Producer Groups’

Production and Sales (Averages Unless Otherwise Indicated) .................... D.10

D.11. Impact of PBS on Annual Household Income, Consumption, and

Poverty, by Value Chain (in U.S. Dollars Unless Otherwise Indicated) ........ D.11

ix

FIGURES

I.1 PBS Logic Model ............................................................................................. 2

I.2 Timeline of PBS Implementation, 2007-2012 .................................................. 4

II.1 Periods Covered in the Interim Impact Analysis of PBS Activity ..................... 12

II.2 Randomization Results for the PBS Evaluation, by Value Chain ..................... 14

II.3 Timeline of PBS Implementation for Treatment and Control Groups,

by Value Chain ............................................................................................ 14

II.4 Geographic Distribution of Producers in the Evaluation Sample, by

Value Chain ................................................................................................. 17

III.1 Number of PBS Participants Versus PBS Participants in the

Evaluation, by Value Chain ........................................................................... 21

III.2 Proportion of PBS Participants in Each Value Chain ....................................... 22

III.3 Proportion of PBS Participants Residing in Each Department, by

Value Chain ................................................................................................. 23

III.4 Number of Randomized Producers Who Received Assistance, by

Month and Treatment Group: Dairy Chain, 2010-2011 ................................. 30


Month and Treatment Group: Horticulture Chain, 2010-2011 ...................... 31


Month and Treatment Group: Handicrafts Chain, 2009-2010

(Phase I) ....................................................................................................... 32

IV.1 Annual Employment Generated by PBS Participants, by Value Chain

(in Full-time Jobs) ......................................................................................... 47

IV.2 Net Annual Productive Income of PBS Participants, by Value Chain

(in U.S. Dollars) ............................................................................................ 48

IV.3 Net Annual Household Income of PBS Participants, by Value Chain

(in U.S. Dollars) ............................................................................................ 50

IV.4 Annual Household Consumption of PBS Participants, by Value Chain

(in U.S. Dollars) ............................................................................................ 50

B.1 Consumer Prices for Milk in El Salvador, 2009-2011 (Averages in

U.S. Dollars per Bottle) ................................................................................ B.2

B.2 Consumer Prices for Vegetables and Basic Grains in El Salvador,

2009-2011 (Averages in U.S. Dollars per Pound) ......................................... B.7

x

EXECUTIVE SUMMARY

Funded by the Millennium Challenge Corporation (MCC) and implemented by El Salvador’s Millennium Challenge Account (known as FOMILENIO in Spanish) from 2008 to 2012, the main objective of the Productive Development Project (PDP) was to assist in the development of profitable and sustainable business ventures for poor individuals in El Salvador’s Northern Zone. Over approximately four years, the PDP used nearly $72 million in allocated funds to provide over 13,500 participants with technical and material assistance and create more than 11,000 full-time equivalent jobs. The PDP comprised three activities: Production and Business Services (PBS), Investment Support, and Financial Services. The PBS Activity offered training and technical assistance, in-kind donations, and other business development services to small farmers and business owners. The Investment Support Activity offered investment capital (in the form of long-term loans of over $50,000) for viable business proposals. Lastly, the Financial Services Activity supported two loan guarantee programs targeting micro-, small, and medium enterprises, as well as a small technical assistance program to financial institutions.

The largest of the PDP’s three activities was the Production and Business Services (PBS) Activity, which provided technical and material assistance to farmers and small-scale producers to support the Northern Zone’s dairy, fruit, horticulture, handicrafts, tourism, forestry, and coffee sectors. As stated in the 2006 MCC-El Salvador compact in which it was established, the goal of the PBS Activity was to ―help poor farmers, organizations and micro-, small, and medium enterprises that benefit poor inhabitants of the Northern Zone successfully transition to higher-profit activities, generating new investment, expanding markets and sales, and creating new jobs in ways that stimulate sustainable economic growth and poverty reduction.‖

PBS assistance began with a pilot phase in 2008, and general implementation began in September 2009 and ended in July 2012. As defined in the compact, the PBS Activity’s total funding of $57 million was originally allocated to finance technical assistance to poor farmers, in-kind donations, business development services, as well as pre-investment studies to develop and implement viable business plans related to the activity’s target value chains. Following modifications to program activities in early 2010, PBS funding was allocated to a wider array of investments, including in-kind donations, technical assistance and training, demonstration plots, technical and financial support for enterprises created and supported by FOMILENIO, and investments in innovative productive projects. Table 1 provides a summary of the key characteristics of the PBS Activity, including its objective, funding, and primary forms of assistance.

Executive Summary Mathematica Policy Research

xi

Table 1. Key Characteristics of the PBS Activity of the Productive Development Project

Objective Help producers successfully transition to higher-profit activities, generate new

investment, expand markets and sales, and create new jobs

Target Population Poor farmers, organizations and micro-, small, and medium enterprises that

benefit poor inhabitants of the Northern Zone

Total Funding $56.9 million

Implementing Parties

FOMILENIO and Chemonics (all value chains); TechnoServe, Zamorano,

Proleche, and CARE (dairy); CLUSA, IICA, CARE, CATIE, and Zamorano

(horticulture); Aid to Artisans, Swisscontact, and BERDAL de CV (handicrafts)

Time Frame 2008 to 2012

Activities/Assistance

Technical assistance and training

In-kind donations (such as agricultural inputs)

Demonstration plots and group training sessions

Technical and financial support for enterprises created and supported by

FOMILENIO

Investments in innovative productive projects related and unrelated to

agricultural sectors

Source: PDP Operations Manual, December 2010.

Figure 1 provides a visual representation of how the PBS Activity was designed to achieve its key objective of economic development. First, PBS service providers offer technical assistance and in-kind donations to farmers and artisans. In addition, small-scale producers receive training through use of demonstration plots (in the case of the horticulture and dairy chains), which allow for observation and hands-on exposure to new crops, production technologies, and irrigation techniques. A portion of PBS participants also have access to investment capital through Activity 2 of the PDP (Investment Support), as well as small loans through Activity 3 of the PDP (Financial Services).1 This capital can help producers transition to high-value crops and finance new production technologies such as greenhouses and irrigation systems.

As a result of training and assistance, farmers develop stronger business and technical skills. With these new skills as well as donated inputs and increased investments, farmers generate increased and more diversified production. In addition, enterprises supported by FOMILENIO provide participating farmers with cheaper inputs and pay farmers a higher price for their production. This leads to increased sales, income, and profits. As defined in the compact, PBS participants were expected to increase their annual income by 15 percent, on average, as a result of assistance.

1 Although a strong intersection between PBS and Activity 2 was originally envisioned, only 15 PBS participants

secured credit through Activity 2 during the compact period.


xii

Figure 1. PBS Logic Model


Note: A broken line is used for the box portraying access to Activities 2 and 3 because PBS participants must

complete an application process to qualify for capital and assistance through these activities.

Summary of PBS Implementation

The preparation phase of the PBS Activity began in September 2007. Managed by the Salvadoran Multi-Sector Investment Bank (known as BMI for its initials in Spanish) and SNC-Lavalin, an engineering firm based in Canada, this phase served to establish basic PBS operations and conduct diagnostic studies of investment opportunities in the Northern Zone. From July 2008 to September 2009, BMI and SNC-Lavalin oversaw the pilot phase of the PBS. During this phase, newly contracted PBS service providers initiated 13 pilot projects to provide training to small producers in the dairy, horticulture, fruit, handicrafts, and tourism chains.

In September 2009, FOMILENIO assumed direct supervision of the PBS Activity from BMI. In addition, Chemonics International, a development consulting firm, began coordinating and managing the various components of the PBS Activity under the PDP’s general implementation phase. This coordination involved various subcontracts with service providers related to assistance in horticulture, dairy, handicrafts, tourism, and forestry value chains. During this general implementation phase (as well as during the pilot phase), the PBS Activity focused on increasing and diversifying farmers’ and artisans’ production. As such, PBS assistance during this time period—later referred to as Phase I—was oriented toward decreasing input costs, promoting new technologies, and improving productive practices. For each value chain, one service provider covered all PBS assistance to beneficiaries in the Northern Zone.

In September 2010, the PBS Activity was reorganized and a new phase of PBS implementation began. During this phase, called Phase II, the focus of PBS expanded to include production, access to markets, and business capacity. Organized by geographic regions in the Northern Zone, service providers began training beneficiary groups in production and irrigation techniques through the use of demonstration plots (in the case of dairy and horticulture value chains). With one exception, service providers in this new implementation phase were different from those that provided technical assistance in Phase I. Modifications to the PBS Activity under Phase II also included a

Increased income

and profits

Technical

assistance

and training

Increased private

investment

Stronger business

and technical skills

Increased

employment

Expected Results

Improved market

access

In-kind

Donations

Increased production

and sales

Assistance Potential access to

investment capital and

financial services through

Activities 2 and 3

Demonstration

plots

Support for

enterprises


xiii

commercial alliance between FOMILENIO, USAID, and a major Salvadoran supermarket chain, Súper Selectos, as well as the creation and strengthening of several producer-owned enterprises in the dairy and horticulture value chains. These producer-owned enterprises provided an opportunity for PBS participants to buy inputs and sell production in bulk. Assistance under Phase II continued until the expiration of field-staff’s contracts in July 2012.

Table 2 provides a summary of PBS service targets and outcome targets, as well as final results of the PBS Activity by August 2012 (according to Chemonics’s administrative records). As illustrated, the activity exceeded all service and outcome targets. Notably, Chemonics exceeded its training target of 13,000 individuals by over 2,300 producers, and surpassed its target of 10,500 new permanent equivalent jobs by over 2,400 jobs. In addition, Chemonics reported over $27 million in counterpart investments from PBS participants.

Table 2. PBS Service and Outcome Targets and Results

Projected Final Results

Service Target

Individuals Served with Technical

Assistance and Training

13,000 15,319

Outcome Targets

Number of Permanent Equivalent Jobs

Created

10,500 12,929

Number of Hectares Under Cultivation

with PDP Assistance

17,554 22,207

Counterpart Investments $10.1 million $27.1 million

Source: Chemonics, August 2012.

Note: Targets and achievements are aggregated across all value chains in the project.

Research Questions and Evaluation Design

In 2007, MCC contracted with Mathematica Policy Research to design and conduct the impact evaluation of the PBS Activity. In this interim evaluation report, we analyze the following primary research questions related to this activity:

What impact did FOMILENIO’s offer of PBS assistance have on employment creation and producers’ investment and income?

What impact did FOMILENIO’s offer of PBS assistance have on household income?

What impact did FOMILENIO’s offer of PBS assistance have on intermediate outcomes, such as production levels, business practice adoption, technology adoption, and product diversification?

The first question relates to income at the producer level, whereas the second addresses income at the household level. Tracking income at these two levels is important because income at the producer level provides the most direct measure of the economic impact of PBS assistance, whereas income at the household level provides a measure of the ultimate effect of assistance on the well-being of producers’ households. In this interim evaluation, we also attempt to determine whether there were differential impacts of PBS assistance in Phase I versus Phase II.


xiv

The design for the impact evaluation of the PBS Activity is a randomized rollout design, in which some producers were offered PBS assistance several months before other producers were offered similar assistance. The goal of this design is to attain the highest level of rigor possible without significantly altering existing implementation plans. The evaluation is limited to three value chains: (1) handicrafts, (2) dairy, and (3) horticulture. Among all value chains in the PBS, these three chains were most amenable to random assignment, and were expected to yield impacts in the one-year interim timeline specified for the evaluation.

Implementation plans required that random assignment be done at the group level for the dairy and horticulture chains, and at the municipality level for the handicraft value chain. In all three value chains, groups (or municipalities in the case of handicrafts) were randomized into treatment and control. Treatment groups were offered PBS assistance in the first implementation phase (beginning in late 2009 for handicrafts and mid-2010 in the case of the dairy and horticulture chains) and control groups in the dairy and horticulture chains were offered PBS assistance roughly one year after the treatment group.2 Table 3 provides a summary of the number of groups and individuals randomized into treatment and control in each chain, as well as the number of randomized individuals who completed both baseline and follow-up surveys, and were thus included in the evaluation. As over 15,000 individuals had participated in the PBS Activity by August 2012, this evaluation sample of 1,736 individuals (518, 593, and 625 in the dairy, horticulture, and handicrafts chains, respectively) includes only 11 percent of the individuals served under the activity.

Table 3. Producers in the PBS Impact Evaluation, by Value Chain and Treatment Group

Dairy Horticulture Handicrafts Total

Treatment Control Treatment Control Treatment Control Treatment Control

Number of Randomized

Groups

14 14 15 16 9 10 38 40

Number of Randomized

Individuals

295 300 324 323 337 337 956 960

Number of Individuals

in the Evaluation

260 258 297 296 303 322 860 876

Percentage of

Randomized Individuals

in the Evaluation

88 86 92 91 90 96 90 91

Source: Mathematica administrative records.

The evaluation design chosen for the PBS evaluation offers the key advantage of randomized studies: random assignment usually leads to the creation of two virtually identical groups at baseline, the sole difference being that only one group (the treatment group) is offered the intervention, while the other group (the control group) is not. As a result, any changes observed between the two groups following intervention activities can be attributed to the intervention, or defined as the impact of the intervention. To ensure that these estimates are not biased, however, we must verify that no differential or overall attrition occurred during the evaluation period. Given the high proportion of randomized individuals who completed both surveys in all three chains, the PBS evaluation is unlikely to yield biased estimates (see Table 3). We also did statistical tests to verify that treatment

2 The original design was that the producers assigned to the control group in the handicrafts chain would be

offered services beginning in September 2010. However, due to modifications to the PBS activity in 2010, Phase II services were not offered to the control group at any point in 2010, 2011, or 2012.


xv

and control groups were equivalent at baseline. In general, we find the groups are similar, but some differences were found for the dairy and handicrafts value chains. Our analysis controls for these initial differences.

In this report, we analyze the impact of PBS after the first implementation period. This can be interpreted as the effect of being offered PBS assistance for approximately one year, compared to not being offered this assistance during the same period. (This first implementation period should not be confused with the phases of PBS implementation, as this one-year period includes a mix of Phase I and Phase II services in the dairy and horticulture chains.) The data sources for this analysis are the baseline and follow-up Productive Development Surveys (PDS), which were administered before and after the first implementation period. These surveys were developed by Mathematica and administered by the Salvadoran Ministry of the Economy’s General Office of Statistics and Census (known as DIGESTYC for its initials in Spanish) to all producers in the study sample. With baseline and follow-up data for each producer in the study, we can estimate the impact of PBS after controlling for producers’ baseline levels of production and income.

For each of the three value chains in the evaluation, we present two types of impact estimates: intent-to-treat (ITT) estimates and treatment-on-the-treated (TOT) estimates. Intent-to-treat estimates capture the impact of the offer of PBS, regardless of whether the producers in the treatment group accepted this offer. To construct these estimates, we compare the entire sample of producers in the treatment group to the entire sample of producers in the control group, regardless of whether producers participated in PBS. However, another important question is the effect of PBS on the producers who actually received PBS assistance. This is particularly important for this evaluation, given the heterogeneity in participation rates among the three chains: treatment group participation rates were 79, 33, and 65 percent for the dairy, horticulture, and handicrafts chains, respectively. To determine the effect of assistance among those who received it, we also calculate treatment-on-the treated estimates. In the next section, we first present intent-to-treat findings, and then compare and contrast intent-to-treat and treatment-on-the-treated estimates.

Key Impact Findings

A. Producer-level Findings

The offer of PBS had a positive effect on employment in the handicrafts chain, but not in the dairy or horticulture chains. Under an intent-to-treat approach, artisans in the treatment group generated, on average, 0.13 more full-time equivalent jobs than artisans in the control group during the follow-up period (Figure 2).3 This is equivalent to over one month of additional full-time labor. This difference is statistically significant at the 1 percent level. Although employment created in the other two value chains was higher among treatment group producers than their control group counterparts, these differences were not statistically significant. In particular, the lack of statistical significance of the large treatment-control difference in the dairy chain likely reflects the study’s low level of statistical precision, primarily caused by the small number of randomized producer groups.

3 Full-time equivalent jobs are jobs that require 250 days of labor per year.


xvi

Figure 2. Annual Employment Generated by PBS Participants, by Value Chain (Full-time Jobs)

Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from September

2009 to July 2011.

Notes: Estimates reflect treatment-control differences at follow-up, after controlling for baseline

differences. Employment in the horticulture chain reflects all employment related to agricultural

production. Treatment group participation rates were 79, 33, and 65 percent for the dairy,

horticulture, and handicrafts chains, respectively.

The offer of PBS increased net productive income in the dairy value chain. In our intent-to-treat analysis of dairy production, the average annual productive income of treatment group producers was $1,849 higher than control group producers (p-value of 0.01, Figure 3). Our analysis of intermediate outcomes found that treatment group producers in the dairy chain sold a larger volume of milk, as well as more secondary dairy products than control group producers, even after controlling for baseline sales. The positive impact of PBS assistance on producers’ net income is likely related to both of these dairy products.

Figure 3. Net Annual Productive Income of PBS Participants, by Value Chain (in U.S. Dollars)


2009 to July 2011.


differences. Net income in the horticulture chain reflects all income related to agricultural



0.58

0.22

0.15

0.40

0.14

0.02

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Dairy Horticulture Handicrafts

Treatment

Control

3,608

-109

679

1,759

191

654

-500

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000


Treatment

Control

Impact = $1,849

p-value = 0.01

No Significant

Impact

No Significant

Impact

No Significant

Impact

Impact = 0.13

p-value < 0.01

No Significant

Impact


xvii

In the horticulture and handicrafts chains, we found no significant impacts of PBS on productive income. In the horticulture chain, we found some evidence that treatment group farmers were more likely to produce fruits and vegetables—particularly cucumbers—than control group farmers. However, this behavior change did not produce the higher income that would be expected to accompany increased high-value crop production. In the handicrafts chain, we found some evidence that treatment group artisans invested more in production inputs than control group artisans, particularly in paid labor. However, there is no evidence that this additional investment resulted in higher net income (Figure 3).

B. Household-level Findings

We found no statistically significant impact of PBS on net household income in any value chain. Under an intent-to-treat approach, PBS had a positive but statistically insignificant effect on annual household income for the dairy value chain. Households in the treatment group reported average annual incomes that were around $2,600 higher than households in the control group (Figure 4). Because this impact is not significant at the 5 percent level, we cannot conclude with certainty that PBS had a positive impact on net household income for the dairy chain. The lack of statistical significance of this treatment-control difference likely reflects the study’s low level of statistical precision. In addition, no noteworthy treatment-control differences in household income were found for the horticulture or handicrafts chains.

Figure 4. Net Annual Household Income of PBS Participants, by Value Chain (in U.S. Dollars)

Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from

September 2009 to July 2011.


differences. Among households, treatment group participation rates were 73, 33, and 67

percent for the dairy, horticulture, and handicrafts chains, respectively.

We found no impact of PBS on household consumption in any value chain. Under an intent-to-treat approach, PBS assistance did not lead to a statistically significant increase in household consumption in any of the evaluation’s three value chains.

9,231

2,869

2,464

6,622

2,927 2,615

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000


Treatment

Control

No Significant

Impact No Significant

Impact

Difference = $2,608

p-value = 0.12


xviii

C. Comparison of Intent-to-Treat and Treatment-on-the-Treated Estimates

Table 4 summarizes this report’s main findings at the producer level under both the intent-to-treat (ITT) and treatment-on-the-treated (TOT) approaches. As mentioned, ITT estimates can be interpreted as the effect of the offer of PBS assistance, regardless of whether the services were accepted, and TOT estimates can be interpreted as the effect of PBS on those who accepted these services.4 Under both ITT and TOT approaches, we find that the offer of PBS assistance had a statistically significant and positive impact on employment generation among artisans. Among all artisans in the treatment group, PBS assistance resulted in 0.13 additional jobs (ITT), and among artisans who participated in the intervention, PBS assistance resulted in 0.19 additional jobs (TOT). This last difference is equivalent to nearly 50 days of full-time employment.

As discussed above, we find evidence that the offer of PBS had a significant positive impact of around $1,850 on dairy producers’ annual productive income (ITT). At over $3,000, this impact is even larger among individuals who participated in PBS assistance (TOT).

Table 4. Summary of Impacts of PBS at the Producer Level, by Value Chain


Intent-to-Treat Analysis (ITT)

Investments and Input Costs No statistically

significant impact

No statistically

significant impact

No statistically

significant impact

Number of Full-time Equivalent Jobs

Generated

No statistically

significant impact

No statistically

significant impact

Statistically

significant impact

of 0.13 FTEs

Net Productive Income Statistically

significant impact

of $1,849

No statistically

significant impact

No statistically

significant impact

Treatment-on-the-Treated Analysis (TOT)

Investments and Input Costs No statistically

significant impact

No statistically

significant impact

No statistically

significant impact

Number of Full-time Equivalent Jobs

Generated

No statistically

significant impact

No statistically

significant impact

Statistically

significant impact

of 0.19 FTEs

Net Productive Income Statistically

significant impact

of $3,114

No statistically

significant impact

No statistically

significant impact




differences. Treatment group participation rates were 79, 33, and 65 percent for the dairy,


4 To calculate TOT estimates, we used an instrumental variable approach, in which the assignment to treatment

was used as an instrument for participation. Another approach is to divide ITT estimates by the participation rate of the treatment group in each chain, minus the participation rate of the control group. Bloom (1984) describes this method in detail. This technique assumes that the effect of the program on non-participants is zero, so it re-scales the ITT estimate to reflect only the effect on participants. Both approaches yielded similar impact estimates.


xix

Table 5 summarizes this report’s main findings at the household level under both the ITT and TOT approaches. As illustrated, both approaches yielded no statistically significant impacts of PBS assistance on net household income or consumption.

Table 5. Summary of Impacts of PBS at the Household Level, by Value Chain



Net Household Income (from all

sources)

No statistically

significant impact

No statistically

significant impact

No statistically

significant impact

Household Consumption No statistically

significant impact

No statistically

significant impact

No statistically

significant impact


Net Household Income (from all

sources)

No statistically

significant impact

No statistically

significant impact

No statistically

significant impact

Household Consumption No statistically

significant impact

No statistically

significant impact

No statistically

significant impact




differences. Among households, treatment group participation rates were 73, 33, and 67

percent for the dairy, horticulture, and handicrafts chains, respectively.

Conclusions

Analyzing all impact results, we find two particularly promising findings. First, the offer of PBS services had a statistically significant and positive impact on employment generation for the handicrafts value chain. In particular, the treatment-on-the-treated estimate of 0.19 full-time equivalent jobs is a promising and substantive finding, as it provides evidence that artisans who participated in training and technical assistance made a substantial investment in additional labor related to handicraft production.

The second noteworthy finding is that PBS assistance to dairy producers significantly raised these individuals’ productive income. This finding suggests that interventions of this type, particularly in the dairy chain, can generate significant effects on income within a relatively short time period of one year. This result coincides with expectations set by Chemonics and FOMILENIO regarding the timeframe in which benefits of assistance would accrue. However, the concentration of positive impacts in the dairy value chain underscores the importance of sector-specific factors—including market conditions and technical assistance packages—in influencing participants’ productive and economic outcomes.

Analyzing impact estimates in light of program implementation findings, perhaps the relatively high frequency of assistance provided to most dairy producers in the study—compared to the lower frequency of assistance provided to a minority of horticulture producers in the study—was instrumental in generating positive economic impacts in the dairy chain as opposed to the horticulture chain. The quality of such assistance and its relevance to producers’ needs may have also influenced differential impacts in the dairy and horticulture chains. The absence of statistically significant impacts in the handicrafts chain, despite a moderate level of services to most artisans in the study, suggests that structural obstacles to marketing and selling handicrafts may have inhibited positive impacts on artisans’ sales and income.


xx

Limitations

While this rigorous evaluation provides an unbiased estimate of the impact of PBS assistance, its primary limitation is the lack of generalizability of its impact estimates. With an analysis sample of less than 2,000 producers who are not representative of all 15,319 PBS participants, this evaluation’s results cannot be extrapolated to the entire population of producers assisted under PBS. In addition, the evaluation’s one-year time frame precludes a rigorous evaluation of the impact of PBS assistance over the full multi-year compact period.

This evaluation also has limited statistical precision, largely due to the small number of randomized producer groups in each value chain. At its inception, the impact evaluation was designed to produce impact estimates for the sample of producers in each value chain, as well as the combined sample of producers in all value chains. This combined sample offered the potential for a higher level of statistical precision than the value chain-specific samples. However, large heterogeneity in producers and activities among the three chains precluded an impact analysis for the combined sample. In addition, the total number of randomized groups of producers in each value chain was lower than originally planned. As a result, minimum detectable differences in the dairy, horticulture, and handicrafts value chains were relatively large, at over 30 percentage points for all key individual-level outcomes and over 20 percentage points for all key household-level outcomes in each of the value chains.

Next Steps

In September 2011, data collectors completed the second follow-up survey for the handicrafts value chain. A final round for the handicrafts survey is scheduled for August 2012. Scheduled for submission in January 2013, the final report of the PBS impact evaluation will present two- and three-year impact findings for the handicrafts chain. In addition, Mathematica will complete a performance analysis of PBS assistance in the dairy, horticulture, and handicrafts value chains for submission in September 2012.

1

I. BACKGROUND

In this report, we estimate the impact of the Production and Business Services (PBS) Activity of the Productive Development Project on small producers’ investments, employment, and income. In this chapter, we provide a summary of the PBS Activity. In Chapter II, we present the evaluation’s methodology, including the study’s goals, research questions, design, and data sources. In Chapter III, we provide a description of the study sample. In Chapter IV, we present the impacts of PBS after one year of services for producers in the dairy, horticulture, and handicrafts value chains. Chapter V provides a summary of the findings and presents the report’s conclusions.

A. Summary of the PDP and PBS Activity

The Productive Development Project (PDP) was one of three large-scale projects financed under the 2006 compact between the Millennium Challenge Corporation (MCC) and the Government of El Salvador. Administered under the supervision of the Millennium Challenge Account (known as FOMILENIO in Spanish), the main objective of the PDP was to assist in the development of profitable and sustainable business ventures for poor individuals in El Salvador’s Northern Zone. Between 2008 and 2012, the PDP used nearly $72 million in allocated funds to serve over 13,500 participants and create more than 11,000 full-time equivalent jobs.5 The PDP comprised three activities: Production and Business Services (PBS), Investment Support, and Financial Services. The PBS Activity offered training and technical assistance, in-kind donations, and other business development services to small farmers and business owners. The Investment Support Activity offered investment capital (in the form of long-term loans of over $50,000) for viable business proposals. Lastly, the Financial Services Activity supported two loan guarantee programs targeting micro-, small, and medium enterprises, as well as a small technical assistance program to financial institutions.

The largest of the PDP’s three activities was the Production and Business Services Activity, which provided technical and material assistance to farmers and small-scale producers to support the Northern Zone’s dairy, fruit, horticulture, handicrafts, tourism, forestry, and coffee sectors. As stated in the 2006 MCC-El Salvador compact in which it was established, the goal of the Production and Business Services (PBS) Activity was to ―help poor farmers, organizations and micro-, small, and medium enterprises that benefit poor inhabitants of the Northern Zone successfully transition to higher-profit activities, generating new investment, expanding markets and sales, and creating new jobs in ways that stimulate sustainable economic growth and poverty reduction.‖

As defined in the compact, the PBS Activity’s total funding of $57 million was originally allocated to finance technical assistance to poor farmers, in-kind donations, business development services, as well as pre-investment studies to develop and implement viable business plans related to the activity’s target value chains. Following modifications to program activities in early 2010, PBS funding was allocated to a wider array of investments, including technical assistance, in-kind donations, demonstration plots and group training sessions, technical and financial support for enterprises created and supported by FOMILENIO, and investments in innovative productive projects related and unrelated to agricultural sectors.

5 According to FOMILENIO and MCC, a PBS participant is an individual who takes part in PBS training or

technical assistance, and or receives donations as part of the PBS activity. This is distinct from a beneficiary, who is defined as an individual who experiences improved outcomes as a result of PBS assistance.

I. Background Mathematica Policy Research

2

B. Design of the PBS Activity

Figure I.1 provides a visual representation of how the Production and Business Services Activity was designed to achieve its key objective of economic development. First, PBS service providers offer technical assistance and in-kind donations to farmers and artisans. In addition, small-scale producers receive training through the use of demonstration plots, which allow for observation and hands-on exposure to new crops, production technologies, and irrigation techniques. A portion of PBS participants also has access to investment capital through Activity 2 of the PDP (Investment Support) as well as small loans through Activity 3 of the PDP (Financial Services).6 This capital can help producers transition to high-value crops and finance new production technologies such as greenhouses and irrigation systems.

As a result of training and assistance, farmers develop stronger business and technical skills. With these new skills as well as donated inputs and larger investments, farmers generate increased and more diversified production. In addition, enterprises supported by FOMILENIO provide participating farmers with cheaper inputs and pay farmers a higher price for their production. This leads to increased sales, income, and profits. As defined in the compact, PBS participants were expected to increase their annual income by 15 percent, on average, over the course of the multi-year activity as a result of assistance.

Figure I.1. PBS Logic Model


Note: A broken line is used for the box describing access to Activities 2 and 3 because PBS participants

must complete an application process to qualify for capital and assistance through these activities.

6 Although a strong intersection between PBS and Activity 2 was originally envisioned, only 15 PBS participants

secured credit through Activity 2 during the compact period.

Increased income

and profits

Technical

assistance

and training

Increased private

investment

Stronger business

and technical skills

Increased

employment

Expected Results

Improved market

access

In-kind

Donations

Increased production

and sales

Assistance Potential access to

investment capital and

financial services through

Activities 2 and 3

Demonstration

plots

Support for

enterprises


3

C. Implementation of the PBS Activity

In September 2007, the preparation phase of the PBS Activity began. Managed by the Salvadoran Multi-Sector Industrial Bank (known as BMI for its initials in Spanish) and SNC-Lavalin, an engineering firm based in Canada, this phase served to establish basic PBS operations and conduct diagnostic studies of investment opportunities in the Northern Zone. From July 2008 to September 2009, BMI and SNC-Lavalin oversaw the pilot phase of the PBS. During this phase, $5 million was disbursed to 13 productive projects. Projects included technical assistance for dairy farmers, technical and material assistance for artisans, and training for small farmers related to fruit and vegetable production. Pilot projects benefited 155 groups and 3,625 people with technical and material assistance.7 No impact evaluation of the pilot phase of the PBS Activity was completed.8

In September 2009, FOMILENIO assumed direct supervision of the PBS Activity from BMI. In addition, Chemonics International, a development consulting firm, began coordinating and managing the various components of the PBS Activity under the PDP’s general implementation phase. This coordination involved various subcontracts with service providers related to assistance in horticulture, dairy, handicrafts, tourism, and forestry value chains. During this general implementation phase (as well as during the pilot phase), the PBS Activity focused on increasing and diversifying farmers’ and artisans’ production. As such, PBS assistance during this time period—later referred to as Phase I—was oriented toward decreasing input costs, promoting new technologies, and enhancing productive practices. For each value chain, one service provider covered all assistance to beneficiaries in the Northern Zone. The service provider for the handicrafts value chain, Aid to Artisans, worked strictly with groups of producers, whereas the other service providers (CLUSA for horticulture and TechnoServe for dairy) worked with individual producers as well as organized groups. Due to changes in some service providers and contract renewals during late 2010, very few beneficiaries in any value chains received services from September 2010 to mid-January 2011. (See timeline of PBS implementation in Figure I.2)

Starting in September 2010, the PBS Activity was reorganized and a new phase of PBS implementation began. During this phase, called Phase II, the focus of PBS Activity expanded to include production, access to markets, and business capacity. Organized by geographic regions, service providers in the dairy and horticulture chains began training beneficiary groups in production and irrigation techniques.

7 These figures are based on monitoring reports submitted by pilot phase implementers.

8 However, a process analysis of the pilot was conducted by FOMILENIO.

Artisans supported by PBS sell their merchandise

at a handicrafts fair (photo courtesy of

Chemonics, 2011).


4

Figure I.2. Timeline of PBS Implementation, 2007-2012

With the exception of TechnoServe in the dairy value chain, service providers in Phase II were different from those that provided technical assistance in Phase I. Swisscontact was the primary provider for the handicraft value chain;9 IICA, CARE, CATIE and Zamorano were the providers for the horticulture value chain; and TechnoServe, Zamorano, Proleche, and CARE were the providers for the dairy chain. Each of these providers offered assistance to small producers in their assigned area and facilitated commercial linkages between actors in the region. Also under Phase II, technical assistance for the horticulture chain was delivered in tandem with assistance for the fruit value chain. Assistance under Phase II continued until the expiration of field-staff’s contracts in July 2012.

In Phase II, all training in the horticulture and dairy chains was conducted through Business Development Demonstration Centers (called CDEDs for their initials in Spanish), which had demonstration plots that were used for ―field school‖ training sessions. Multiple farmers participated

in these sessions, which featured instruction in crop management, market access, and business development. These centers were equipped with the necessary resources to facilitate farmers’ understanding and adoption of new production technologies and practices.

Modifications to the PBS Activity under Phase II also included a commercial alliance between FOMILENIO, USAID, and a major Salvadoran supermarket chain, Súper Selectos, as well as the creation or strengthening of several producer-owned enterprises. Under Phase II, these producer-owned businesses began specifying production levels and buying production from productive groups at a fair price. The producer-owned enterprise in the horticulture chain was El

Salvador Produce, and the primary producer-owned enterprises in the dairy chain were Lácteos de Morazán and Lácteos Zona Norte. These enterprises negotiated contracts with Súper Selectos and other large buyers, and then organized and aggregated production through use of collection and supply

9 Another service provider, BERDAL, also provided technical services in the handicrafts chain. As of early 2012,

however, BERDAL was no longer associated with the PBS Activity.

Dairy farmers in Sensuntepeque, Cabañas,

participate in a PBS training session on cattle feed

(photo courtesy of Chemonics, 2011).


5

centers. This included 17 collection centers for the dairy chain, 10 workshops for the handicrafts chain, and 5 collection centers and 2 supply centers for the horticulture chain.

In addition, eligibility criteria for assistance under PBS changed from Phase I to Phase II for each of the three value chains in the evaluation: dairy, horticulture, and handicrafts. Notably, Phase II participants in the dairy chain were required to own at least 10 cows and work primarily in cattle farming, whereas these requirements were not explicit in Phase I. Similarly, Phase II participants in the handicrafts chain were required to have experience in handicraft production, whereas an interest in handicraft production was sufficient in Phase I. (See Appendix A for a full list of Phase I and Phase II eligibility criteria.)

Table I.1 provides a summary of assistance provided in Phase I and Phase II of the PBS for the dairy, horticulture, and handicrafts value chains. As shown, the major differences between the two phases of PBS implementation are the larger number of contractors in Phase II, as well as the higher intensity of assistance in the second phase compared to the first. Notably, training and assistance in the horticulture chain under Phase I occurred every six weeks to two months, whereas training and assistance under Phase II was designed to occur on a weekly basis for all three of the value chains.

The PBS Activity had several funding components in Phase II: $21 million was allocated for general implementation (which included $3 million to finance collection centers); $13 million was allocated for in-kind investments; $8 million was allocated for the Fund to Support Productive Development (known as FADEP due to its initials in Spanish), which included $500,000 to strengthen the capacity of El Salvador Produce; and $4 million was allocated to the Fondo de Iniciativas Productivas (FIP), which financed ongoing productive projects similar to pilot projects. Examples of projects funded by the FIP include a project to strengthen fruit production systems in Cuscatlán and Cabañas (funded at $341,000), a project to link dairy farmers in La Unión (funded at $344,000), and a project linking chipilín and loroco producers to markets in the United States (funded at $670,000).10

Table I.2 provides a summary of PBS service targets and outcome targets, as well as final results of the PBS Activity by August 2012 (according to Chemonics’s administrative records). As illustrated, the activity exceeded all service and outcome targets. Notably, Chemonics exceeded its training target of 13,000 individuals by over 2,300 producers, and surpassed its target of 10,500 new permanent equivalent jobs by over 2,400 jobs. In addition, Chemonics reported over $27 million in counterpart investments from PBS participants.

Throughout Phase I and Phase II implementation, tropical storms and economic trends affected PBS implementation. In 2009 and 2010, tropical storms Ida, Agatha and Mathew caused losses to Salvadoran agricultural production estimated at $96.3 million, of which $68.9 million were losses related to the production of basic grains (CENTA 2010). Regional departments most affected by the storms were San Salvador, San Vicente, Cuscatlán, La Paz, Usulután, and La Libertad, and crops most affected were corn, rice, fruits and vegetables, and sugarcane.

10 These FIP-funded projects do not fall within the scope of the PBS impact evaluation.


6

Table I.1. Comparison of Phase I and Phase II PBS Assistance, by Value Chain

Dairy Horticulture/Fruit Handicrafts

Phase I Implementation

Key Actors Contractors: TechnoServe Contractors: CLUSA Contractors: Aid to Artisans (ATA)

Assistance

Provided

Start-date: May 2010

End-date: End of August 2010

Frequency: Between every 15

days and once per month

Technical staff: Groups were

served by technical staff and

assistants.

Assistance: Groups were trained

in herd maintenance, irrigation

techniques, vaccines, de-

worming, fertilizers, and

fumigation. Technicians formed

field schools and gave courses to

participant groups. TechnoServe

staff also donated machinery as

well as genetic material to some

beneficiary groups.

Start-date: April/May 2010

End-date: End of August /

beginning of September 2010

Frequency: The service provider

planned to conduct monthly

visits. However, visits were every

six weeks to two months for

some participant groups.


served by teams made up of one

trained technician and two

community promoters.

Assistance: CLUSA installed

irrigation systems in some land

owned by beneficiaries, and

distributed machinery to some

beneficiaries. CLUSA technicians

also built small greenhouses and

distributed plants.

Start-date: September 2009 for

most groups

End-date: End of August 2010

Frequency: Weekly. ATA staff also

responded to pressing issues as

necessary.


served by teams of specialists,

including a design specialist and a

marketing specialist.

Assistance: Topics covered in

trainings were design, marketing,

quality control and accounting.

ATA had a designer and a trainer

in charge of helping groups track

costs and keep financial records.

ATA also introduced design molds

in order to reduce costs and

standardize quality.

Phase II Implementation

Key Actors Contractors: TechnoServe,

Zamorano, Proleche, and CARE

Businesses: Lácteos de Morazán

and Lácteos Zona Norte bought

participants’ production.

Contractors: Contractors were

IICA, CARE, CATIE and Zamorano

(with 2 regions). Each contractor

ran 1 of 5 collection centers

(with Zamorano running two

centers).

Businesses: El Salvador Produce

handled all sales.

Contractors: Swisscontact and

BERDAL de CV

Businesses: Two handicrafts

retailers bought production from

beneficiary groups.

Assistance

Provided

Start-date: Assistance was

scheduled for early November

2010, but the actual start-date

was in early 2012.

End-date: July 2012

Frequency: On a weekly basis

Technical staff: Teams of 3

technical staff were assigned to

serve 200 beneficiaries. One

technical person in each team had

several years of experience, and

supervised the other 2 junior staff

members.

Assistance: Similar to the

horticulture value chain,

assistance was centered upon

demonstration plots, which

served as the forum for

theoretical and practical

instruction.

Start-date: Assistance was

scheduled for September 20,

2010, but actual start-date was

in early 2012. Collection centers

were operational by January

2011.

End-date: July 2012

Frequency: Field schools

convened at least once a week.

Technical staff: The technician

in charge of the school was

accompanied by 2-3 assistants.

Assistants had to hold a relevant

post-secondary degree.

Assistance: Technical assistance

was centered upon field schools,

comprised of a demonstration

plot donated by a beneficiary.

For each plot, 20 to 25

beneficiaries were trained on

new technologies and business

practices.

Start-date: Meetings with

Swisscontact officially started on

October 1, 2010.

End-date: July 2012

Frequency: On a weekly basis

Technical staff: Swisscontact staff

ran most training sessions.

According to FOMILENIO, this

organization had a strong

knowledge base.

Assistance: Similar to training

model used by ATA, with a focus

on design, business skills, and

marketing/sales. More material

assistance, including raw materials

for production, was offered in

Phase II.

Source: Chemonics program manuals and in-person interviews conducted by Mathematica staff in October 2010.


7

Table I.2. PBS Service and Outcome Targets and Results

Projected Final Results

Service Target

Individuals Served with Technical

Assistance and Training

13,000 15,319

Outcome Targets

Permanent Equivalent Jobs Created 10,500 12,929

Hectares Under Cultivation with PDP

Assistance

17,554 22,207

Farmers who applied improved

techniques

7,000 12,661

Enterprises assisted 110 602

Counterpart Investments $10.1 million $27.1 million

Source: Chemonics, August 2012.

Note: Targets and achievements are aggregated across all value chains in the project.

In October 2011, tropical depression 12E caused substantial losses to Salvadoran agricultural and dairy production, including the production of numerous PBS participants. Based on an internal assessment, Chemonics found that 775 of the 3,072 hectares under production with assistance from PBS had experienced damages. Total losses related to PBS assistance were estimated at $800,000 in investments and more than $1.8 million in potential sales (FOMILENIO 2011). To mitigate the effects of the storm, FOMILENIO financed a recovery effort in late 2011 and early 2012. The effort consisted of providing inputs and technical assistance to producers who experienced the largest damages.11

From late 2008 to 2010, the global financial crisis also had a detrimental effect on the Salvadoran economy. Specifically, the crisis caused contractions in the Salvadoran credit market, reduced demand for Salvadoran exports, and generated a decline in remittances from the U.S. Chemonics and FOMILENIO stakeholders reported that reduced consumer demand at the local, national, and international level had a negative effect on the production and sales of PBS participants, particularly in 2009 and 2010. However, the full effect of the crisis on producer outcomes is impossible to quantify.12

D. Costs of the PBS Activity

As of early 2012, total investments in the PBS Activity were nearly $46 million (Table I.3). Among the PBS Activity’s value chains, the average cost per participant ranged from around $1,600 in the forestry value chain to over $3,200 in the dairy value chain. At over $3,100, the average cost

11 Because this storm occurred after this analysis’s study period, its overall effect on horticulture and non-

horticulture production is not reflected in impact findings. Furthermore, the effect of significant tropical storms and other natural disasters on farmer productivity and net income is generally consistent across treatment and control groups, and thus rarely compromises key estimates of rigorous impact analyses.

12 Similar to weather conditions, the effect of the financial crisis is assumed to be generally consistent across treatment and control groups in the PBS evaluation, and thus not a threat to the evaluation’s validity.


8

per participant in the horticulture/fruit chain was nearly as high as the dairy chain, and average handicraft costs per participant were lower at around $2,000.

Table I.3. Costs of the PBS Activity (In U.S. Dollars)

Value Chain

Technical

Assistance

and

Training Donations

Equipment and

Infrastructure

Administrative

Costs

Total

Investment Participants

Cost per

Participant

Dairy 8,763,171 4,007,812 1,028,434 359,646 14,159,062 4,390 3,225

Horticulture/

Fruit

14,137,807 7,505,883 1,076,052 545,384 23,265,126 7,479 3,111

Handicrafts 2,095,075 140,486 80,297 61,013 2,376,871 1,176 2,021

Tourism 1,170,662 127,543 57,392 46,076 1,401,672 555 2,526

Fisheries 944,819 222,301 0 30,504 1,197,625 530 2,260

Apiculture 748,335 317,196 126,365 22,386 1,214,282 415 2,926

Coffee 1,274,942 477,824 122,604 62,372 1,937,742 610 3,177

Forestry 148,249 47,459 0 0 195,708 120 1,631

All Chains 29,283,059 12,846,504 2,491,144 1,127,381 45,748,088 15,275 2,995

Source: Chemonics monitoring data, 2012. Current as of April 2012.

E. Interim Findings on PBS Implementation

In 2010, FOMILENIO contracted an outside evaluator, ADEPRO, to document the experiences of FOMILENIO beneficiaries, including PBS participants. ADEPRO’s report, published in January 2011, provides insight into PBS implementation from 2008 to 2010. According to the report, around 80 percent of all 255 interviewed PBS participants—most of whom reported primary employment in the agricultural sector—reported that they had received training associated with the activity.13 The majority of individuals who received training stated that it was very useful. This was corroborated by additional focus groups conducted by ADEPRO with several PBS participants, who noted that technical assistance had helped them improve their production. However, focus group participants also noted that long periods of time would elapse between visits and training sessions. Participants also noted that the transition in service providers from Phase I and Phase II resulted in a discontinuity of services (ADEPRO 2011).

With respect to the activity’s desired outcomes of stronger business and technical skills and increased production, the majority of PBS participants interviewed for the ADEPRO study stated that PBS assistance did not help them acquire new business skills. However, over 80 percent of interviewed participants said they had applied new production techniques to their crops. Approximately seven in ten interviewed participants said that FOMILENIO assistance had helped them improve their monthly income. (However, it should be noted that only one-third of

13 Based on a proportional representation of all FOMILENIO beneficiaries and stratified by municipality, this

sample of 255 PBS participants was somewhat representative of all FOMILENIO beneficiaries when combined with results from other FOMILENIO activities. However, as all PBS interviewees were participants in the horticulture chain, these results are not representative for all PBS participants across all the activity’s value chains.


9

interviewees responded to this question.) Regarding employment generation, only 14 percent of interviewed participants noted that their employment of workers increased following assistance.

Interviews with FOMILENIO staff and PBS participants conducted in late 2010 by Mathematica revealed key findings that were similar to the 2011 ADEPRO report. As reported in these interviews, a large barrier to successful implementation during Phase I was a lack of resources devoted to each producer, particularly in the horticulture and fruit value chains. According to FOMILENIO, each technical field-staff member in the horticulture chain was assigned up to 150 beneficiaries, and this resource constraint resulted in very infrequent visits to some beneficiary groups. Similarly, each technical field-staff member in the dairy chain was assigned around 100 beneficiaries. In the handicraft value chain, FOMILENIO staff and participants stated that the lack of in-kind donations was a major obstacle to improved outcomes, as such donations could have motivated participants and boosted initial production. Regarding factors that facilitated PBS implementation, participants in each of the dairy, horticulture, and handicrafts chains stated that technical staff was very knowledgeable about the subject matter covered in training sessions.

F. Evaluation Rationale

MCC contracted Mathematica in 2007 to design and conduct evaluations of the first two PDP activities—PBS (Activity 1) and Investment Support (Activity 2). Given MCC’s goal of evaluating interventions with the most rigorous methods available, Mathematica researched both activities and determined that a rigorous evaluation of the PBS Activity was feasible. Developed and refined by Mathematica, MCC, FOMILENIO, and other stakeholders, the evaluation of the PBS Activity uses random assignment, which allows us to assess the impact of PBS on producers’ income and employment in a rigorous manner. The results of this evaluation can provide MCC and other stakeholders with unbiased estimates of the overall effectiveness of PBS assistance during a specified timeframe. In addition, the evaluation is intended to provide MCC staff with information that could inform future funding decisions or project designs.

Because it was not possible to estimate the counterfactual for the Investment Support Activity of the PDP, or how loan recipients would have fared in the absence of the activity, our evaluation of Activity 2 employed a case study design. Using this design, we analyzed the implementation of the Investment Support Activity and summarized key results.14 Although this design does not provide unbiased estimates of the impact of the Investment Support Activity, it is intended to help MCC staff better understand program implementation, lessons learned, and best practices.

14 The results for the case study of the Investment Support Activity were released in the implementation study

report ―The Investment Support Activity of the Productive Development Project, El Salvador‖ in November of 2011.

10

II. EVALUATION DESIGN

A. Design Summary

For the impact evaluation of the PBS Activity, Mathematica, MCC, FOMILENIO and other stakeholders chose a randomized rollout design, in which some producers were offered PBS assistance several months before other producers were offered similar assistance. The evaluation design exploited two key characteristics of PBS implementation in 2009 and 2010. First, there was insufficient capacity to serve all participants simultaneously given personnel and funding constraints. Second, there were enough people in the Northern Zone who were eligible and interested to participate in the PBS Activity to form a valid comparison group. In 2009, Mathematica, MCC, FOMILENIO, and Chemonics agreed upon a randomized rollout design. Under this agreement, Chemonics committed to identify and recruit potential beneficiaries who could be served in either a first or a second cycle of services. Then Mathematica staff randomly assigned potential beneficiaries into two groups: the treatment group, which would receive PBS assistance in the first cycle of services, and the control group, which would receive PBS assistance in the second cycle. This design allows us to assess the impact of PBS by comparing outcomes of the treatment group to outcomes of the control group after the first cycle of services.

Given the diversity of productive sectors served under PBS, Mathematica, MCC, and FOMILENIO agreed that the PBS impact evaluation should be limited to three value chains: (1) handicrafts, (2) dairy, and (3) horticulture. All stakeholders agreed that these three value chains were likely to yield impacts within one to two years. The randomized rollout design is common across all three value chains, but the evaluation covers slightly different time periods for each chain, based on pre-established implementation timelines. Another difference across chains is the unit at which random assignment occurred. Because implementation plans required that Chemonics serve entire groups of producers during the same cycle, we were unable to randomly assign individual participants into treatment and control groups. Consequently, random assignment was done at the group or municipality level for all three chains. Groups such as dairy cooperatives and agricultural producer associations were randomly assigned to either the treatment or control group for the dairy and horticulture value chains, respectively. For the handicraft value chain, implementation plans required that random assignment be done at the municipality level.

While this evaluation design is rigorous and likely to provide an unbiased estimate of the impact of PBS assistance, its primary limitation is the lack of generalizability of its impact estimates. With an analysis sample of less than 2,000 producers in only three value chains, this evaluation’s results cannot be extrapolated to the entire population of over 15,000 producers assisted under PBS.15 Similarly, given the particular characteristics of producers in this evaluation and the political and economic conditions in the Northern Zone, this evaluation’s results cannot be readily applied to similar populations of primarily agricultural producers outside of El Salvador’s Northern Zone.

15 Unfortunately, statistical weights cannot be used to make the evaluation results truly representative of all PBS

participants. This is because the study population was not drawn randomly from the full range of geographic locations served under PBS. Furthermore, the study population was deemed to be eligible only under Phase I selection criteria, as opposed to a substantial portion of PBS beneficiaries, who met stricter Phase II eligibility standards.

II. Evaluation Design Mathematica Policy Research

11

B. Evaluation Questions

In this interim evaluation report, we analyze the following primary research questions related to the Production and Business Services Activity:

What impact did FOMILENIO’s offer of PBS assistance have on employment creation and producers’ investment and income?

What impact did FOMILENIO’s offer of PBS assistance have on household income?

What impact did FOMILENIO’s offer of PBS assistance have on intermediate outcomes, such as production levels, business practice adoption, technology adoption, and product diversification?

The first question relates to income at the producer level, whereas the second addresses income at the household level. Tracking income at these two levels is important because income at the producer level provides the most direct measure of the economic impact of PBS assistance, whereas income at the household level provides a measure of the ultimate effect of assistance on the well-being of producers’ households.16 In this evaluation, we also attempt to determine whether there were differential impacts of PBS assistance in Phase I versus Phase II.

In this interim report, we analyze the impact of PBS after the first implementation period for each value chain. This can be interpreted as the effect of being offered PBS assistance for approximately one year, compared to not being offered this assistance during the same timeframe. (This first implementation period should not be confused with the phases of PBS implementation, as this one-year period includes a mix of Phase I and Phase II services in the dairy and horticulture chains; see Figure II.1.) In 2013, we will submit a final impact report of PBS assistance. Focusing on the handicrafts chain, the final report will examine the impact of offering PBS for two years (as compared to not offering services during this timeframe), as well as the effect of offering PBS for three years (as compared to not offering services during the same period).17 In addition, the final report will include an in-depth implementation (or performance) analysis of PBS assistance in the dairy, horticulture, and handicraft chains.

16 FOMILENIO and Chemonics define productive income at the producer level to be the key economic outcome

of interest, as opposed to income at the household level.

17 Impact estimation for the handicrafts chain is distinct from the other two chains because implementation plans for Phase II did not include providing services to the control group (that is, providers who would have received services in Phase II). Because FOMILENIO and MCC stakeholders are interested in determining the effect of offering three years of PBS assistance, DIGESTYC will conduct an additional round of data collection in 2012. The results regarding the impact of three years of assistance in the handicrafts chain will be included in the final report.


12

Figure II.1. Periods Covered in the Interim Impact Analysis of PBS Activity

C. Evaluation Design Implementation

The implementation of the randomized rollout design for the PBS evaluation consisted of the following eight steps.18

1. Identification of potential beneficiaries. In this first and critical step, Chemonics recruited enough producers to fill PBS service slots for the first cycle of implementation and enough of the second cycle to populate the study sample in each of the three value chains.

2. Compilation of lists of potential beneficiaries. For each value chain, a few weeks prior to the start of the first cycle, Chemonics provided Mathematica with lists of potential beneficiaries required for each value chain. The number of potential beneficiaries on these lists was determined by Chemonics’ target number of participants for each implementation cycle, as well as Mathematica’s preliminary calculations of the size of the impacts that the evaluation would be likely to detect with those sample sizes. Within the lists, producers were organized by groups—either the groups in which they already worked or new groups organized by Chemonics for training and assistance purposes.

3. Randomization. For each value chain, Mathematica randomized the set of potential beneficiaries into two groups: the treatment group, which would be served in the first cycle, and the control group, which would be served in the second cycle. Groups and individuals designated as exceptions by Chemonics were excluded from randomization.19 Mathematica sent the randomized lists of assigned potential beneficiaries to Chemonics, and Chemonics informed producers of the start-date of their services.

18 Here we present the steps required for all the value chains, but the schedule for each value chain varies. Each

schedule was defined in collaboration with all the stakeholders involved in order to respect Chemonics’ implementation plans as much as possible.

19 Exceptions included producers who were well organized and had strong potential links with buyers. Chemonics viewed these producers as too strategic to be subject to randomization, which could result in no access to services for several months if these producers were assigned to the control group.


13

4. Baseline data collection. Under a data collection contract with FOMILENIO, the Ministry of Economy’s General Office of Statistics and Census (known as DIGESTYC for its initials in Spanish) collected baseline data from all potential beneficiaries close to the start of the first intervention cycle. Survey dates (and other key dates) are presented below in Table II.1. In the dairy and horticulture chains, baseline data collection occurred shortly after the treatment group began services.

5. PBS assistance was offered to treatment group producers. During each chain’s first cycle of services, Chemonics offered PBS assistance to the treatment group, but not to the control group. This provided a comparison between receiving one year of intervention services (treatment group) and receiving no intervention services for one year (control group).

6. First follow-up data collection. For each value chain, DIGESTYC collected follow-up data close to the end of first cycle. The specific dates for the first follow-up survey vary by chain, as described in the data collection section.

7. PBS assistance was offered to control group producers. Next, the treatment group received a second year of PBS assistance and the control group received its first offer of PBS assistance (for a period of approximately one year). The exception is the handicrafts chain, in which the control group did not receive services during the second cycle of services.

8. Second and third follow-up data collection. For the handicrafts value chain, close to the end of the second and third implementation cycles, DIGESTYC will collect a second and third round of follow-up data, respectively. These data will be used to determine the impact of two and three years of assistance to artisans, respectively.

Table II.1. Key Dates in the PBS Impact Evaluation

Value Chain


Compilation of Lists January–March

2010

January–March

2010

July–September

2009

Randomization March 2010 March 2010 October 2009

Baseline Data Collection May 2010 June 2010 October 2009

Assistance Offered to

Treatment Group

April 2010 April 2010 October 2009

Follow-up (Round 1) Data

Collection

June 2011 May 2011 November 2010

Assistance Offered to Control

Group

May 2011 April 2011 NA


Collection

NA NA September 2011


Collection

NA NA August 2012


By the end of 2011, the first seven activities had been completed for the three value chains. Noteworthy was the randomization process, which occurred in late 2009 for the handicrafts chain and in early 2010 for the dairy and horticulture chains. The randomization results are provided in


14

Figure II.2. As mentioned above, producers were randomized at the group level for dairy and horticulture chains, and at the municipal level for the handicraft chain.

Figure II.2. Randomization Results for the PBS Evaluation, by Value Chain

Dairy Chain: 28 groups with

595 producers

Horticulture Chain: 31 groups

with 647 producers

Handicrafts Chain: 19

municipalities with 674 producers

Treatment Control

Treatment Control

Treatment Control

14 groups

with 295

producers

14 groups

with 300

producers

15 groups

with 324

producers

16 groups

with 323

producers

9 municipalities

with 337

producers

10

municipalities

with 337

producers


Figure II.3 illustrates the unique implementation and data collection timeline for each value chain in the evaluation. Under the randomized phase-in design, treatment group producers in the dairy chain were eligible to begin services in April 2010, whereas control group producers did not have access to services until May 2011. Similarly in the horticulture chain, treatment group producers were eligible to begin services in April 2010, whereas control group producers did not have access to services until April 2011. In both these value chains, participating treatment and control group producers were expected to continue services until the end of the activity in July 2012. In contrast, treatment group producers in the handicrafts chain were eligible to receive assistance starting in September 2009, whereas control group producers did not have access to services during the course of the compact period. The blue boxes in Figure II.3 illustrate the period covered in this interim evaluation report: approximately one year of services for treatment group producers, compared to no services for control group producers.

Figure II.3. Timeline of PBS Implementation for Treatment and Control Groups, by Value Chain

Note: Blue boxes indicate the evaluation’s interim follow-up period.


15

D. Outcomes and Data Sources

The PBS impact evaluation will assess both main and intermediate outcomes resulting from the offer of PBS assistance. To create these outcome indicators, we designed the Productive Development Surveys (PDSs), which are in-person baseline and follow-up surveys that focus on productive activities and household income and consumption. To accurately capture these measures, we developed a distinct PDS survey instrument for each of the dairy, horticulture, and handicraft chains in the evaluation. Under a contract between FOMILENIO and DIGESTYC, data collectors administered baseline and follow-up PDSs to all eligible producers in the three value chains featured in the evaluation, which includes all producers in the treatment group as well as the control group. For all three value chains, data collection plans included baseline data collection (right before the first cycle of services), the first follow-up data collection (right after the end of the first cycle and in some cases a month after the start of the second cycle), and second follow-up data collection (right after the end of the second cycle). MCC and FOMILENIO have added a third follow-up round of data collection for the handicrafts value chain, which will occur right after beneficiaries have received three years of services (See Figure II.2). In addition, MCC and FOMILENIO have cancelled second-round follow-up data collection for the dairy and horticulture chains.20

Main Outcomes. The ultimate goal of PBS is to increase the employment and household income of producers in El Salvador’s Northern Zone. Using the PDSs, we collected information on sources of income that were most directly affected by the training programs, specifically income from handicraft, dairy, and horticulture production. We also tracked employment information, measured by the number of people contracted by producers in the study sample to perform productive activities. Given that MCC is ultimately interested in measuring economic outcomes at the household level, we collected income at the producer level as well as the household level, and asked a series of questions about household consumption. Data on these outcomes was collected at baseline and follow-up. For handicrafts, we will capture these outcomes in a second and third follow-up survey.

Intermediate Outcomes. In addition to employment and income outcomes, we examined intermediate outcomes through which the training programs were intended to improve household income, including production and sales, adoption of new practices and technologies, as well as enhanced product diversification and value chain integration. As with the study’s main outcomes, data on these intermediate outcomes were collected at baseline and first follow-up. For handicrafts, we will also collect this data in the second and third follow-up survey. In Table II.2, we define the evaluation’s main and intermediate outcome measures.

20 These surveys were cancelled primarily because determining the impact of one year of services versus two years

of services was no longer a priority for stakeholders in early 2012. In addition, MCC had concerns that given the original study population and changes in program implementation, the evaluation’s results would not be generalizable to the larger population of PBS participants.


16

Table II.2. Definitions of Main and Intermediate Outcomes, PBS Impact Evaluation

Measure Definition

Main Outcomes

Full-time Employment Generation The number of full-time equivalent (FTE) jobs generated by the PBS

participant (one FTE is 250 days of labor per year) in the past year.

Annual Investment and Input Costs The sum of respondents’ annual short- and medium-term

productive investments and costs, including inputs, labor,

transportation, packaging, and marketing expenses.

Net Annual Productive Income Respondents’ annual income from productive sales after

subtracting the costs of all investments and inputs.

Net Annual Household Income The sum of all household members’ annual productive income,

wages, business income, and additional income (including

remittances), after subtracting agricultural and business expenses.

Net Annual Household Consumption The sum of all annual household expenses on food, household

items, utilities, health care, transportation, and education, among

others.

Intermediate Outcomes

Production, sales, and prices Number of items produced and sold, as well as the price per unit

sold in the past year.

Technology Adoption Binary measures of whether the producer used new information

technologies and production techniques in the past year.

Diversification Binary measures of whether the producer tried new crops or

products in the past year.

Business Practices Binary measures of whether the producer conducted quality

control measures, developed an inventory, composed a business

plan, or took measures to cut costs in the past year.

Value Chain Integration Binary measures of whether the producer looked for new clients in

the commercial sector, as well as the number of clients in the past

year.

The number of producers originally randomized and the number of producers in the evaluation

sample for each value chain are found in Table II.3, in addition to survey response rates. The analysis sample for this evaluation includes all respondents for which data were available at baseline and at follow-up. As shown, baseline and follow-up surveys had response rates above 89 percent, and only treatment and control groups in the dairy chain had slightly less than 90 percent of randomized producers in the final evaluation sample. Maps in Figure II.4 illustrate the geographic dispersion of treatment and control producers in the final study sample.


17

Table II.3. Producers in the PBS Impact Evaluation, by Value Chain

Dairy Horticulture Handicrafts Combined

T C T C T C T C

Number of Randomized Individuals 300 295 324 323 337 337 961 955

Number of Completed Baseline Interviews 282 282 310 308 331 327 923 917

Baseline Interview Completion Rate (%) 94 96 96 95 98 97 96 96

Number of Completed Follow-up Interviews 273 270 300 296 303 322 878 888

Follow-up Interview Completion Rate (%) 91 92 93 92 90 96 91 93

Number of Individuals Who Completed Both

Baseline and Follow-up Interviews

260 258 297 296 303 322 860 876

Percentage of Randomized Individuals in

the Evaluation Sample (%)

87 87 92 92 90 96 90 92


Note: T = Treatment, C = Control

Figure II.4. Geographic Distribution of Producers in the Evaluation Sample, by Value Chain

Note: T = Treatment; C = Control



18

Figure II.4. Geographic Distribution of Producers in the Evaluation Sample, by Value Chain (continued)

Note: T = Treatment; C = Control



19

The key advantage of implementing a study design based on random assignment is that, on average, producers in the treatment group and producers in the control group are balanced on all characteristics before the beginning of the intervention. Hence, after the first cycle of services, the difference between the mean of the outcome of interest for the treatment group and the mean of that same outcome for the control group is an unbiased estimate of the impact of the offer of PBS. To improve the precision of impact estimates, we use a regression model that controls for baseline characteristics. Furthermore, regression adjustments allow us to control for any differences between the treatment and control groups in baseline characteristics due to chance or to differential attrition.

The impact analysis relies on a core regression specification for each value chain. In this specification, we take into account that randomization occurred at the group (or cluster) level for each value chain.21 The econometric specification compares how groups (or clusters) in the treatment group changed over time with how groups (or clusters) in the control group changed over time, controlling for idiosyncratic differences in the two groups. The basic model can be expressed as follows:

where yigt is the outcome of interest for producer i in group or cluster g at time t; xigt-1 is a vector of baseline characteristics of producer i in group or cluster g; yigt-1 is the baseline value of the outcome indicator of producer i in group or cluster g; Tg is an indicator equal to one if group or cluster g is in

the treatment group and zero if it is in the control group;22 and igt is a random error term for

beneficiary i in group or cluster g observed at time t. The parameter estimate for is the estimated impact of the program for each value chain.

We also must verify that differential or overall attrition does not threaten the study’s internal validity. Given the low attrition rates for all three value chains, the PBS evaluation does not face this threat. Attrition at the level of random assignment (group or municipality) was non-existent, and attrition at the producer level was low: for dairy, overall attrition was 13 percent (with no treatment-control differential); for horticulture, overall attrition was 8 percent (with no treatment-control differential); and for artisans, overall attrition was 7 percent (with a 6 percent treatment-control differential). We also did statistical tests to verify that the groups were equivalent at baseline; these tests are discussed in the next section. Few differences were found at baseline, and we control for these differences in the regression analysis.

For each of the three value chains in the evaluation, we produce two types of impact estimates: intent-to-treat (ITT) estimates and treatment-on-the-treated (TOT) estimates. ITT estimates capture the impact of the offer of PBS, regardless of whether the producers in the treatment group accepted this offer. To construct these estimates, we compare the entire sample of producers in the treatment group to the entire sample of producers in the control group, regardless of whether producers participated in PBS. However, another important question is the effect of PBS on the producers who actually received PBS assistance. This is particularly important for this evaluation, given the

21 A key aspect of this model is that standard errors are clustered at the level of random assignment. This is the

group level for the dairy and horticulture chains, and the municipal level for the handicrafts chain.

22 A fixed effect specification was not feasible in this analysis because the main variable of interest, treatment, had no variation within group or cluster. While feasible, and random effects specification was not used due to the sensitivity of this specification to baseline controls.


20

heterogeneity in participation rates among the three chains: treatment group participation rates were 79, 33, and 65 percent for the dairy, horticulture, and handicrafts chains, respectively. To answer this question, we calculate TOT estimates, which can be interpreted as the effect of PBS on the producers who accepted the PBS offer. To calculate TOT estimates, we use an instrumental variable approach, in which the assignment to the treatment group is used as an instrument for participation.23

E. Design Limitations

While this rigorous evaluation provides an unbiased estimate of the impact of PBS assistance, its primary limitation is the lack of generalizability of its impact estimates. With an analysis sample of less than 2,000 producers in only three value chains, this evaluation’s results cannot be extrapolated to the entire population of 16,500 producers assisted all value chains of the PBS activity. In addition, the evaluation’s one-year time frame precludes a rigorous evaluation of the impact of PBS assistance over the full multi-year compact period.

This evaluation also has limited statistical precision, largely due to the small number of randomized producer groups in each value chain. At its inception, the impact evaluation was designed to produce impact estimates for the sample of producers in each value chain, as well as the combined sample of producers in all value chains. This combined sample offered the potential for a higher level of statistical precision than the value chain-specific samples. However, large heterogeneity in producers and activities among the three chains precluded an impact analysis for the combined sample. In addition, the total number of randomized groups of producers in each value chain was lower than originally planned. As a result, minimum detectable differences in the dairy, horticulture, and handicrafts value chains were relatively large, at over 30 percent for all key individual-level outcomes and over 20 percent for all key household-level outcomes in each of the value chains.

23 Another approach is to divide ITT estimates by the participation rate of the treatment group in each chain.

Bloom (1984) describes this method in detail. This technique assumes that the effect of the program on non-participants is zero. As a sensitivity test, we also calculated TOT estimates using this method.

21

III. STUDY SAMPLE

In this chapter, we provide a description of the beneficiary population and evaluation sample, and address whether treatment and control group producers were equivalent at the time of randomization. We also analyze producers’ participation in the PBS intervention, and document the basic character of this assistance.

A. Description of the Evaluation Sample

Numbering over 15,000 by August 2012, the total population of PBS participants received a wide array of services spanning eight value chains.24 Unfortunately, information on the number of participants served under each value chain was not available for the full number of 15,319 individuals served by August 2012. Instead, we used FOMILENIO monitoring data on 14,410 of these individuals to describe the distribution of participants among the project’s value chains. As illustrated in Figures III.1 and III.2, most participants received assistance related to horticulture and fruit production, followed by dairy and handicrafts production. However, over 400 individuals received PBS assistance related to fisheries, apiaries, tourism, and coffee production. In contrast, the evaluation sample is limited to three value chains: handicrafts, horticulture and dairy. As illustrated in Figure III.1, our evaluation sample includes only a fraction of the individuals served in each of these three chains.

Figure III.1. Number of PBS Participants Versus PBS Participants in the Evaluation, by Value Chain

Source: FOMILENIO, Office of Monitoring and Evaluation, December 2011.

Notes: Sample sizes are 14,410 (all participants) and 1,736 (participants in the 2011 evaluation sample). It

should be noted that the 322 artisans in the control group will not be offered PBS assistance.

24 This does not include the number of jobs created, which was an additional 12,900 by August 2012 according to

Chemonics’s calculations.

4,285

6,962

1,005

519 405 573

120

541 518 593 625

Dairy Horticulture

and Fruit

Handicrafts Fisheries Apiary Tourism Forestry Coffee

All Participants

Participants in Evaluation

III. Study Sample Mathematica Policy Research

22

As illustrated in Figure III.2, the evaluation sample also represents a different distribution of value chains than the total population receiving PBS assistance. In the evaluation sample, the proportion of handicrafts participants is much higher than the proportion of handicrafts participants in the entire project. However, the proportion of horticulture participants in the evaluation is lower than the proportion of horticulture participants in the entire project. Given the different distribution of value chains represented by the evaluation’s sample frame—as well as the different activities and services featured in each value chain—impact estimates for the combined sample of producers in the evaluation are not presented in this report.

Figure III.2. Proportion of PBS Participants in Each Value Chain

Source: FOMILENIO, Office of Monitoring and Evaluation, October 2010.

Note: Sample sizes are 14,410 (all participants) and 1,736 (participants in the 2011 evaluation

sample).

Examining the proportion of PBS participants residing in each department of the Northern Zone, we see that handicraft producers in the study sample have a similar geographic dispersion to the total population of handicraft producers served under PBS (Figure III.3). In contrast, horticulture and dairy producers in the evaluation are concentrated in a smaller number of departments than horticulture and dairy producers in the full population served by PBS. In particular, the department of Morazán and other departments with smaller numbers of participants (such as La Libertad and La Unión) are underrepresented in the evaluation sample for these two chains.

30%

48%

7%

4%

3%

4% 1%

4%

All Participants

Dairy

Horticulture/Fruit

Handicrafts

Fisheries

Apiary

Tourism

Forestry

Coffee

30%

34%

36%

Participants in the

Evaluation


23

Figure III.3. Proportion of PBS Participants Residing in Each Department, by Value Chain

Source: FOMILENIO, Office of Monitoring and Evaluation, October 2010.

Note: Sample sizes are 14,410 (all participants) and 1,736 (participants in the 2011 evaluation sample).

14

32

2

15

5

21

11

1

42

15

4

11

27

0

0

5

10

15

20

25

30

35

40

45

Cabañas Chalatenango Cuscatlán Morazán San Miguel La Unión Other

Departments

Dairy

All Participants

All Participants in the Evaluation

13

34

9

14

4

16

10

21

29

6

0

18

26

0

0

5

10

15

20

25

30

35

40

Cabañas Chalatenango Cuscatlán Morazán San Salvador Santa Ana Other

Departments

Horticulture

All Participants

All Participants in the Evaluation

17

49

7

24

2

16

43

2

34

5

0

10

20

30

40

50

60

Cabañas Chalatenango Cuscatlán Morazán San Miguel

Handicrafts

All Participants

Participants in the Evaluation

Percentages

Percentages

Percentages


24

1. Demographic Characteristics: Producers

In Table III.1, we summarize the characteristics of all producers in the evaluation. In this analysis, treatment group producers are pooled with control group producers. This facilitates a simple comparison of producers across the three value chains, as well as a summary of producers in all chains (see last column). As illustrated, the majority of producers in the dairy and horticulture chains are married men with a basic level of education. This differs from producers in the handicrafts chain, who are largely female (but have similar marital status and levels of education). On average, producers in the handicraft, horticulture, and dairy chain are 40, 45, and 50 years old, respectively. Almost no producers in the horticulture chain reported belonging to a productive group, and only 13 percent of producers in the dairy chain reported membership with a productive group. In contrast, almost half of producers in the handicrafts chain reported being members of an organized group of artisans. Also notable in Table III.1, similar portions of producers in dairy and horticulture chains reported employing workers, but dairy producers generated the most full-time equivalent jobs of the three value chains.

Table III.1. Baseline Information on Producers, by Value Chain (Averages in U.S. Dollars Unless

Otherwise Indicated)

Characteristic/Measure Dairy Horticulture Handicrafts

Female (%) 16 23 84

Age (in years) 49.8 44.5 40.0

Married or Cohabitating (%) 86 78 67

Educational Level (%)

None 19 27 22

Basic 66 65 67

More than basic 15 8 11

Member of a Productive Group (%) 13 1 44

Annual Investment and Input Costs for Productive

Activitiesa

6,940 919 570

Net Annual Productive Incomeb

1,670 119 401

Annual Income from Salaries 506 479 239

Net Annual Business Income 3,044 286 90

Annual Income from Remittances and Other Sources 1,837 749 253

Total Net Annual Incomec

7,054 1,638 985

Employed Workers Last Year (%) 50 44 6

Average Number of Full-time Productive Jobs Offered by

Producer Last Yearc

1.25 0.32 0.03

Producers 518 593

625

Randomized Groups/Municipalities 28 31 19

Source: Baseline Productive Development Survey (PDS) interviews conducted from September 2009 to June

2010.

a

This is the sum of respondents’ short- and medium-term productive investments and costs, including the cost of

machinery, feed, materials, packaging, transportation, and marketing.

b

This is the sum of respondents’ sales after subtracting the costs of all investments and inputs.

c

This is the sum of respondents’ productive income, wages, business income, and additional income (including

remittances), after subtracting agricultural and business expenses.

c

There is some concern that full-time productive jobs offered by the producer were overestimated at baseline in the

dairy and horticulture chains because some producers may have misinterpreted the question regarding the number

of individuals they employed. This battery of questions was enhanced at follow-up, in which producers were asked

to detail all months, weeks, and days worked by each individual they employed. This likely precluded any

overestimation of labor costs at follow-up.


25

Using PBS data compiled by FOMILENIO, we can compare a limited number of demographic characteristics of individual producers in our study sample to all PBS participants for whom baseline data are available.25 Regarding participants’ sex and age, the PBS participants in the evaluation are similar to the larger population of PBS participants. In the handicrafts chain, the full population of PBS participants had a similar average age as artisans in the study (39 years in late 2009, versus 40 years among the study population) and a similar gender balance (18 percent male versus 16 percent males in the study population). In the horticulture chain, the full population of PBS participants had a similar average age as horticulture producers in the study (44 years in early 2010, versus 45 years in the study population) and an identical gender balance (77 percent male in both populations). In the dairy chain, the full population of PBS participants had a lower average age as horticulture producers in the study (47 years in early 2010 versus 50 years in the study population) and a similar gender balance (86 percent male versus 84 percent male in the study population).

Next, we describe key baseline economic outcomes of producers in the evaluation. With an average net annual productive income of around $1,670 (Table III.1), producers in the dairy chain report much higher productive income than producers in the other two chains. (Producers in the handicraft chain had a net annual productive income of around $400, and producers in the horticulture chain made slightly over $100 last year from agricultural activities, on average.) This discrepancy is more pronounced for total net annual income, which is over $7,000 for dairy producers, but only around $1,650 and $1,000 for horticulture and handicraft producers, respectively.

We compared these baseline income estimates with baseline data compiled by Chemonics for a large portion of PBS participants during Phase I implementation in 2009 and 2010, and found that our estimates of net and gross income were similar to, but generally lower than, those of Chemonics.26 In the dairy chain, our estimate of baseline gross productive income of around $8,600 was substantially lower than the average gross productive income of all Phase I participants at baseline (around $12,000 after excluding one large outlier).27 In the handicrafts chain, our estimate of baseline net productive income of around $400 was similar to Phase I participants’ average net income of $550 at baseline. In the horticulture chain, our estimate of baseline net agricultural income of $120 was less than Phase I participants’ average net income of nearly $300 at baseline (after excluding three large outliers), but within $200.

Due to these baseline differences in income between the study sample and all Phase I PBS participants at baseline, we cannot extrapolate this study’s impact estimates to the full population of PBS participants in the dairy, horticulture, and handicrafts chains.28 Given the heterogeneity of

25 The sample size for these data is 14,410, the same sample used to analyze the geographic dispersion of PBS

participants in and outside of the study sample.

26 The number of producers included in Chemonics’s baseline databases were 629, 1,279, and 624 for the dairy, horticulture, and handicraft chains, respectively.

27 At $10,500 per year, the treatment group’s gross productive income was closer to $12,000. However, the control group’s gross productive income was only $6,650, on average.

28 These differences may be the result of real differences in income between the study sample and all Phase I participants, or differences in data collection tools used by evaluators and implementers. Because a systematic comparison of data collection tools was not conducted, we cannot determine the true cause of these differences.


26

producers and activities in the eight value chains included in PBS assistance, there is even less basis to extrapolate this study’s impact findings to the entire universe of PBS participants in all value chains, including tourism, forestry, dairy, and other sectors.

2. Demographic Characteristics: Households

In Table III.2, we summarize the characteristics of all households in the evaluation. These are all the households corresponding to the producers described in Table III.1. Because some interviewed producers reported belonging to the same households, the number of households in each chain is lower than the number of producers in each chain. To facilitate comparison across chains, treatment group households are pooled with control group producers. As shown, households in all value chains have an average of five members. However, households in the dairy chain have significantly higher total net annual income than households in the other two chains (nearly $9,400 versus less than $3,100 in the other two chains). This higher net income is primarily due to higher productive income, higher net annual business income, and larger remittances among households in the dairy value chain.

Table III.2. Baseline Information on Households, by Value Chain (Averages in U.S. Dollars Unless


Characteristic/Measure Dairy Horticulture Handicrafts

Average Household Size (people) 4.6 4.8 4.9

Female-Headed Household (%) 8 12 34

Net Annual Productive Incomea 2,013 151 478

Annual Income from Salaries 1,387 1,171 1,925

Net Annual Business Income 3,858 579 285

Annual Income from Remittances and Other Sources 2,136 1,187 395

Total Net Annual Incomeb 9,377 3,099 3,080

Annual Consumptionc 7,530 3,238 2,322

Households in Extreme or Relative Poverty (%) 14 53 57

Households 505 559 573

Randomized Groups/Municipalities 28 31 19

Source: Baseline Productive Development Survey (PDS) interviews conducted from September 2009 to

June 2010.

a

This is the sum of productive income generated by all household members, after subtracting the costs of

all investments and inputs.

b

This is the sum of all household members’ productive income, wages, business income, and additional

income (including remittances), after subtracting agricultural and business expenses.

c

Includes all household expenses on food, household items, utilities, health care, transportation, and

education, among others.

B. Baseline Equivalence

We verified that the treatment and control groups were equivalent at baseline using statistical tests (see Tables D.1 and D.2). In general, we find the groups are very similar, but some key treatment-control differences are present in the dairy and handicrafts value chains. For the dairy value chain, treatment and control group producers were equivalent in all demographic characteristics. However, we found some statistical differences in baseline measures of investment


27

and employment. At baseline, the treatment group employed a higher number of people and spent more on productive inputs than the control group.29 Also in the dairy chain, households in the treatment group had fewer members and lower poverty levels than households in the control group. The impact analysis controls for these initial differences. For the handicrafts value chain, we found no statistically significant differences on baseline income, investment, or employment. However, the treatment group was statistically more likely to be female than the control group (a difference of 8 percent) and treatment group artisans were more likely to report being a member of an artisan group. The analysis controls for these initial differences through use of its regression specification.

C. Participation in the PBS Activity

Phase I Assistance. In this section, we use Chemonics monitoring data to characterize PBS implementation during both phases of the PBS Activity. According to these records, 196 producers in the treatment group in the dairy chain (or 66 percent of the treatment group) received at least one donation during Phase I. As illustrated in Figure III.4, a large number of donations were administered to dairy producers in June and July 2010. These donations, which averaged about 15 items per person, largely consisted of materials (51 percent of donations), equipment (28 percent of donations), and inputs (19 percent of donations). In addition, 181 producers in the treatment group in the dairy chain (or 61 percent of the treatment group) received at least one training session or technical assistance visit during Phase I.

According to Chemonics’s records, 84 producers in the treatment group in the horticulture chain (or 26 percent of the treatment group) received at least one donation during Phase I. As shown in Figure III.5, a moderate number of donations were granted to horticulture producers in June and July 2010. These donations, which averaged about six items per person, largely consisted of materials such as micro-tunnels and irrigation tubes (61 percent of donations) and inputs such as seeds and fertilizer (34 percent of donations). In addition, 64 producers in the treatment group in the horticulture chain (or 20 percent of the treatment group) received at least one training session or technical assistance visit during Phase I.

For the handicrafts value chain, administrative records indicate that 217 treatment producers (64 percent of the treatment group) received at least one training session or technical assistance visit during Phase I (Figure III.6). Unfortunately, administrative data on donations were not available for the handicrafts value chain.

Phase II Assistance. In addition, 182 dairy producers in the treatment group (or 62 percent of the treatment group) and 90 producers in the control group (or 30 percent of the control group) received at least one training session during 2011 (see Figure III.5). However, in accordance with the experimental protocol, the control group was largely excluded from assistance until May 2011, after the study’s follow-up period had ended. Training sessions for treatment group producers averaged about 6 sessions per person, and largely consisted of training on quality control, sanitation, and nutrition and disease management. Training sessions for control group producers averaged about

29 In terms of magnitude, the treatment-control difference in investments and inputs costs is large at 0.31 effect

sizes, and the difference in full-time employment is large at 0.32 effect sizes. Effect sizes are calculated as the difference in means divided by the pooled standard deviation.


28

4.5 sessions per person, and largely consisted of the same subjects as training administered to treatment group producers.

According to Chemonics’s records, 79 producers in the treatment group in the horticulture chain (or 24 percent of the treatment group) and 50 producers in the control group (or 15 percent of the control group) received at least one training session during 2011. However, in accordance with the experimental protocol, the control group was largely excluded from assistance until May 2011, after the study’s follow-up period had ended. Training sessions for treatment and control group producers averaged about 3 sessions per person, and largely consisted of training on cultivating rainy season crops, fertilization techniques, and pest control.

A simple analysis of crop cycles suggests that most assistance to horticulture producers in Phase II would be unlikely to generate measurable impacts during the follow-up period (see Table III.3). As cycles for several high-value crops featured in training—including tomatoes grown in a micro-tunnel setting—range from 175 to 250 days, some crops planted in February 2011 (at the start of Phase II assistance) could not have been sold by late April 2011. Assistance with shorter-cycle crops such as cucumbers could have resulted in increased production and sales during 2011. However, such production would only be possible with a consistent source of irrigation during the dry season. This stands in contrast to Phase I assistance to horticulture producers, in which assistance could have affected production and sales of a wide variety of crops throughout 2010.

Table III.3. Crop Cycle Durations and Production/Sales Potential

Crop

Cycle Duration

in Days

Potential for Assistance in

Phase I to Affect

Production/Sales in 2010

Potential for Assistance in

Phase II to Affect

Production/Sales in 2011

Squash 105 High Low

(dependent on irrigation)

Cucumber 90 High Moderate


Tomato (open air) 130 High Low


Tomato (tunnel) 175 Moderate


Low


Chili (protected) 250 Moderate


None

Source: Chemonics administrative data.

In contrast to the horticulture chain, Phase II assistance to dairy producers beginning in January or February 2011 could feasibly affect production and economic outcomes within one to two months. As such, there is potential in the dairy chain to measure the impact of PBS assistance in both Phase I and Phase II.

As illustrated in Figures III.4 and III.5, most training sessions administered to dairy and horticulture producers in the treatment group occurred after the end of the study period. There is a sharp increase in training sessions in May and June 2011, shortly after the end of the study’s follow-up period. In the horticulture chain, this increase in services is likely related to the higher proportion of treatment and control group beneficiaries who could benefit from assistance during the rainy


29

season, given constraints on irrigation water during the dry season. Because Phase II assistance in the dairy and horticulture chains did not scale up until the end of the study period, most potential gains in productivity, sales, and income generated Phase II assistance to both value chains were not measured in the study’s follow-up survey. In the horticulture chain, the general absence of assistance in Phase II and the length of key crop cycles would suggest that any impacts detected during the evaluation period would likely be related to assistance received in Phase I.

Participation Rates for the Evaluation. For this evaluation, participation in the handicrafts chain is defined as receiving at least one training session or technical assistance visit during Phase I implementation, which spanned from November 2009 to September 2010. In contrast, participation in the dairy chain is defined as receiving at least one training session, one technical assistance visit, or one donation related to PBS assistance before May 1, 2011, the final day of the follow-up period. Lastly, participation in the horticulture chain is defined as receiving at least one training session, one technical assistance visit, or one donation related to PBS assistance before April 1, 2011, one month before the end of the follow-up period.

The end-dates used to calculate participation rates vary by value chain due to the nature of each chain’s productive activities. Due to the length of crop cycles, horticulture producers could not feasibly generate improved agricultural outcomes within one month of the start of assistance. As such, a farmer who began assistance in April 2011 is not appropriately classified as a program participant. However, dairy producers could feasibly generate improved milk production and sales within one month of the start of assistance, particularly if this assistance involves donations and training in new technologies. For this reason, any dairy producer who received assistance during the entire study period, including the last month of the period, were classified as program beneficiaries.

Using these designated end-dates in each chain, a small number of control group respondents in the analysis sample (1 producer in the horticulture chain, 17 producers in the dairy chain, and no producers in the handicrafts chain) received at least one PBS training session during the follow-up period. Of the 17 individuals in the dairy chain’s control group (or 6 percent of the evaluation sample) who received at least one training session before May 1, 2011, 15 individuals received only one training session, which was an introduction to the training program. Given that substantive assistance to control group producers largely occurred after the follow-up reporting period, we can conclude that contamination was not a major threat to the evaluation’s impact estimates in any of the three value chains. As such, the comparison of treatment and control groups at follow-up remains a valid estimation of the impact of the offer of PBS assistance on producers’ employment and income.

30

Figure III.4. Number of Randomized Producers Who Received Assistance, by Month and Treatment Group: Dairy Chain, 2010-2011

Source: CHEMONICS administrative records, December 2011.

Note: Sample size includes all 295 producers randomized to treatment and all 300 producers randomized to control.

0

50

100

150

200

April

May

June

July

August

Septem

ber

October

Novem

ber

Decem

ber

January

February

March

April

May

June

July

August

Septem

ber

October

Novem

ber

2010 2011

Treatment: Training

Treatment: Technical Assistance

Treatment: Donations

Control: Training or Technical Assistance

Phase I

Phase II

Evaluation Period

31

Figure III.5. Number of Randomized Producers Who Received Assistance, by Month and Treatment Group: Horticulture Chain, 2010-2011


Note: Sample size includes all 324 producers randomized to treatment and all 323 producers randomized to control.

0

50

100

150

200

April

May

June

July

August

Septem

ber

October

Novem

ber

Decem

ber

January

February

March

April

May

June

July

August

Septem

ber

October

Novem

ber

2010 2011

Treatment: Training


Treatment: Donations


Phase I Phase II

Evaluation Period

32

Figure III.6. Number of Randomized Producers Who Received Assistance, by Month and Treatment Group: Handicrafts Chain, 2009-2010 (Phase I)


Note: Sample size includes all 337 artisans randomized to treatment and all 337 artisans randomized to control.

-25

25

75

125

175

225

Novem

ber

Decem

ber

January

February

March

April

May

June

July

August

Septem

ber

October

2009 2010

Treatment: Training



Evaluation Period


33

Table III.4 provides Phase I, Phase II, and overall participation rates for producers in the study’s analysis sample.30 Phase I participation is defined as receiving at least one form of assistance during all of Phase I. Phase II participation is defined as receiving at least one form of assistance from January 1, 2011 to May 1, 2011 for the dairy chain, and from January 1, 2011 and April 1, 2011 for the horticulture chain. Lastly, overall participation is defined as receiving either Phase I or Phase II assistance, or receiving assistance in both phases. The shorter time period measured in Phase II, as well as the dearth of assistance provided during this period, are reflected in smaller Phase II participation rates vis-à-vis Phase I participation rates.

As illustrated in Table III.4, the dairy chain has the highest overall participation rate among all three chains (nearly 80 percent), followed by the handicrafts chain (65 percent participation) and the horticulture chain (33 percent participation). In particular, much higher participation rates would have been preferred in the horticulture chain, as the evaluation’s intent-to-treat approach assumed that a large portion of treatment group producers would be offered assistance. Using an instrumental variable approach, however, we can estimate the effect of PBS assistance on those who participated in PBS assistance (treatment-on-the-treated).

Table III.4. Treatment Group Participation in PBS Activities During the Study Period, by Value Chain

and Implementation Phase


Phase I

Number 178 92 198

Percentage 68 31 65

Phase II (Services from January to April/May 2011)

Number 95 7 NA

Percentage 37 2 NA

Overall (Phases I and II)

Number 205 97 NA

Percentage 79 33 NA

Treatment Producers in the Evaluation 260 297 303


Note: Participation is defined as receiving at least one PBS training session, technical assistance

visit, or donation, according to Chemonics’s administrative records.

30 These rates are slightly different from participation rates discussed above, which included all producers in the

study’s treatment and control groups, regardless of whether they completed baseline and follow-up surveys.

34

IV. IMPACTS OF PBS ASSISTANCE

In this chapter, we discuss the impact of PBS after approximately one year of services for each value chain. This period includes both Phase I and Phase II services for the dairy and horticulture value chains, and only Phase I services for the handicrafts value chain.31 In addition, we assess the potential differential impacts of these two implementation phases on the outcomes of producers in the dairy and horticulture value chains.32 To discern these impacts, we analyze investments, employment, production and income by agricultural season, as Phase I services were largely confined to the rainy season and Phase II services were largely confined to the dry season.

As described before, for each of the three value chains in the evaluation, we produce two types of impact estimates: 1) intent-to-treat (ITT) effects that capture the impact of the offer of PBS, regardless of whether the producers in the treatment group accepted this offer, and 2) treatment-on-the-treated (TOT) effects, which can be interpreted as the effect of PBS on producers who accepted PBS assistance. Our main focus will be ITT effects, but we will briefly discuss TOT effects at the end of each section.

A. Impacts of PBS on Productive Practices

In analyzing the impact of PBS on producers’ income, we start with a discussion of these individuals’ productive practices. If treatment group producers were significantly more likely than control group producers to adopt new technologies and business practices, we would expect to find treatment-control differences in production, sales, and income.

For all three chains, we find that the offer of PBS had no effect on producers’ overall technology adoption or diversification.33 Treatment and control groups had similar adoption rates of new technologies, production techniques, and environment-friendly products. However, treatment group producers in the dairy chain were more likely to employ specific technologies and practices, including acidity tests, thermometers or density meters, and improved manufacturing processes (impacts of between 5 and 9 percentage points; not shown). Also related to specific technologies, treatment group farmers in the horticulture chain were much more likely than control group farmers to use improved seeds (impact of 23 percentage points; not shown). (See Tables D.3 and D.5 for detailed impact estimates regarding specific practices in the dairy and horticulture chains, respectively.) In addition, artisans in the treatment group were 12 percentage points more likely than control group artisans to report trying new production techniques. However, this difference was not

31 The first follow-up survey for the horticulture chain was fielded in May 2011. This allows us to measure the

impact of five months of PBS provided under Phase I and the first six months of PBS Phase II. The first follow-up survey for the dairy chain was fielded in June of 2011. This allows us to measure the impact of six months of services provided under Phase I and the first six months of Phase II services.

32 An analysis of Phase I versus Phase II impacts is not necessary for the handicrafts chain, as this follow-up period covers only Phase I services.

33 Examining specific technologies in detail for the horticulture chain, treatment group farmers were only slightly more likely to report using irrigation systems (29 versus 26 percent in control) and seed nurseries (34 versus 32 percent in control). However, larger treatment-control differences were detected in the use of improved seeds (82 versus 59 percent in control), crop rotation (37 versus 30 percent in control), and phased planting (33 versus 28 percent in control). Statistically significant treatment-control impacts were measured only in the use of improved seeds.

IV. Impacts of PBS Assistance Mathematica Policy Research

35

statistically significant due to smaller sample sizes associated with surveying only group leaders and independent artisans for these outcomes. Across all chains, we found no significant effects of the offer of PBS on producers’ attempts to try new crops or new products (Table IV.1).

For the dairy value chain, we find some positive effects of the offer of PBS on business practices that are not present in the other two chains. The treatment group of dairy producers was about 5 percentage points more likely to conduct quality control practices than the control group, and this difference is statistically significant. We also find that treatment group producers in the dairy chain were 23 percentage points more likely to report taking measures to reduce costs than control group producers. This difference is statistically significant at the 1 percent level. However, we found no differences in treatment and control groups’ development and use of business plans.

Regarding value chain integration, we find positive effects of the offer of PBS on the percent of dairy producers who reported looking for new clients in the commercial sector: treatment group producers were 7 percentage points more likely to report looking for new clients in the commercial sector than control group producers. In addition, treatment group producers in the horticulture chain were more likely to report selling their production to enterprises supported by PBS, but only by a small margin.

In summary, the offer of PBS had a positive effect on the use of specific technologies in the dairy and horticulture chains, but not on overall rates of technology adoption. In addition, the offer of PBS had a positive impact on dairy producers’ use of quality control and cost reduction measures, as well as their efforts to find new clients. However, the offer of PBS had no measured effect on artisans’ practices, and had only one effect in the horticulture chain related to selling production to a PBS-supported enterprise.

Context from Qualitative Interviews: Dairy Practices

In July 2012, Mathematica staff conducted qualitative interviews with FOMILENIO and Chemonics staff, as well as PBS participants in the dairy chain who appeared to benefit from assistance (according to survey data). One goal of these interviews was to learn more about these producers’ technology adoption and changes in production during the study period. Most interviewed participants reported receiving donated hayseeds in Phase I, in addition to technical assistance related to fodder cultivation and storage. As a result of these donations and assistance, most interviewed participants mentioned that PBS assistance helped them improve their supply of fodder through better pasture maintenance during the rainy season and better fodder storage during the dry season (through use of small silos). Largely due to this sustained supply of fodder, interviewed farmers reported large cost savings during the study period, as well as increased production during the dry season.

Interestingly, farmers in the treatment group did not report adopting new seeds and techniques related to pasture maintenance in follow-up surveys. This may be due to the fact that a large portion of farmers were already using these seeds and techniques at baseline, according to Chemonics staff (and survey data). Rather than encouraging producers to adopt new technologies, stakeholders stated that PBS assistance largely helped farmers improve their use of these technologies. Because participants emphasized their cost savings related to fodder production during qualitative interviews, the large positive impact (of 23 percentage points) in producers’ reported efforts to cut costs revealed by this analysis likely reflects these improved practices. (See Appendix C for a summary of qualitative interviews with PBS participants).

36

Table IV.1. Impact of PBS on Participants’ Productive and Marketing Practices, by Value Chain (Percentages unless Otherwise Indicated)


Measure T C Impact p-value T C Impact p-value T C Impact p-value

Technology Adoption

Used New Information Technologies 4 2 2 0.19 8 8 0 0.97 5 2 3 0.48

Tried New Production Techniquesa

95 91 4 0.15 98 91 7 0.09 31 19 12 0.15

Created Environment-friendly Products 4 3 1 0.56 12 7 5 0.37 5 7 -2 0.79

Diversification

Tried New Crops or Products 4 7 -3 0.27 6 4 2 0.49 22 17 5 0.64

Business Practices

Conducted Quality Control 11 6 5 0.03 5 8 -3 0.43 11 15 -4 0.51

Took Measures to Cut Costs 61 38 23 <0.01 37 26 11 0.24 8 11 -3 0.55

Developed/Used a Business Plan 2 4 -2 0.15 4 3 1 0.64 10 5 5 0.18

Value Chain Integration

Looked for New Clients in the

Commercial Sector

16 9 7 0.04 13 8 5 0.33 5 4 1 0.84

Sold to Enterprises Supported Under

PBS

0 0 0 1.00 2 0 2 0.02 NA NA NA NA

Average Number of Buyers in Past Year

(enterprises and individuals)

3.4 2.3 1.1 0.12 3.5 2.8 0.6 0.38 5.1 3.7 1.4 0.52

Producers 260 258 297 296 121 151

Randomized Groups 14 14 15 16 9 10

Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from September 2009 to July 2011

Note: Estimates reflect treatment-control differences at follow-up, after controlling for baseline differences.

a

For the horticulture chain, new production techniques include the use of any of the following: irrigation systems, tunnels, nurseries, phased planting, and

improved seeds, among other practices. For the dairy chain, new production techniques include the use of any of the following: improved grass types or

forage plants, soil conservation techniques, sugarcane for fodder, nutritional supplements, measures to avoid disease, new techniques related to herd health,

acidity tests, and a thermometer or density meter. Because these technologies were widely used before PBS assistance in both dairy and horticulture chains,

there is no reason to believe that the control group’s high adoption rate is evidence of contamination between treatment and control groups.


37

B. Impacts of PBS on Producer-Level Income, Employment, and Production

In this section, we focus on producer-level impacts on investment and input costs, productive income, and employment generation. These are important outcomes, as they are the most direct mechanism through which PBS assistance was likely to affect the evaluation’s final outcome of net household income. We present ITT estimates in our main analysis, followed by a comparison of ITT and TOT estimates.

1. Dairy Value Chain

For the dairy value chain, we find that the offer of PBS had a positive effect on producers’ annual income related to milk and secondary dairy sales (Table IV.2). Although we found no effects on investments and input costs, we find that the average net income from dairy sales was around $1,850 higher in the treatment group than in the control group during the follow-up period, after controlling for demographic characteristics and baseline income. This difference is statistically significant at the 1 percent level. When we examine income from all sources, we also find that the offer of PBS had a significant positive effect: on average, the treatment group reported over $2,400 more in annual total net income than the control group during the follow-up period (statistically significant at the 5 percent level). This difference appears to be primarily driven by treatment producers’ higher dairy income during this time period. (See Table D.4 for standard deviations of dairy producers’ economic and employment outcomes.)

Also in the dairy value chain, we find that the offer of PBS had some positive effect on employment. Producers in the treatment group were 13 percentage points more likely to employ workers than the control group, after controlling for baseline employment. The number of full-time jobs offered by the treatment group was also higher than the control group: a difference of 0.18 full-time jobs, or approximately 45 additional days of work. However, this difference is not statistically significant, largely due to the study’s low level of statistical precision.

When we analyze intermediate outcomes in the dairy chain, we also find a positive effect of the offer of PBS on producers’ sales. Treatment group producers sold more bottles of milk on a weekly basis, sold a greater portion of their production to the final consumer, and had higher net incomes from secondary dairy products than control group producers; these differences were statistically significant.34 Comparing these impacts with productive practices discussed above, treatment producers’ higher sales, higher net income, and higher likelihood of selling to final consumers appear directly linked to their higher propensity to adopt new practices, take measures to cut costs, conduct quality control measures, and find new clients.

34 We found no difference between treatment and control (see impacts on practices above) regarding sales to PBS-

supported entities. As such, we do not hypothesize that higher prices are due to selling directly to these businesses.


38

Table IV.2. Impact of PBS on Dairy Producers’ Investment, Income, Employment, and Production (in

U.S. Dollars Unless Otherwise Indicated)

Source: Baseline and follow-up Productive Development Survey-Dairy (PDS-D) interviews conducted

from May 2010 to July 2011.


differences. Income components do not sum perfectly to total net income, given a procedure

by which total net income was not calculated if more than one income component was

missing.

a

This is the sum of respondents’ short- and medium-term dairy investments and costs, including the cost of

machinery, cattle feed, vitamins, labor, packaging, transportation, and marketing.

b

Net dairy income is total individual income from milk and dairy sales after subtracting all investments and

inputs.

c

Number of full-time equivalent jobs is calculated as the number of jobs that require 250 days of work a

year.

To determine the robustness of the impact of PBS on net dairy income—particularly considering a small number of key treatment-control differences at baseline—we conducted several sensitivity tests. First, we found that this impact estimate was statistically significant regardless of the control variables included in regression specifications. In addition, we found comparable impacts for the related measure of gross productive income. However, we found that impacts on net dairy income were substantially lower—although still statistically significant—when seven individuals with very high baseline incomes were excluded from the study sample (impact of around $1,170 versus $1,850 for the full study sample). In addition, regressions that predicted impacts in the growth rate

Measure Treatment Control Impact p-value

Annual Investment and Income (US$)a

Dairy Investments and Input Costsa

6,368 5,208 1,160 0.16

Net Dairy Incomeb

3,608 1,759 1,849 0.01

Income from Salaries 484 572 -89 0.48

Net Business Income 1,967 1,239 728 0.29

Income from Remittances and Other Sources 374 286 88 0.65

Total Net Income 6,293 3,868 2,425 0.04

Annual Employment of Workers

Percentage of Producers who Employed Workers 36 23 13 0.01

Average Number of Full-time Equivalent Jobs Offered by

Producer Related to Dairy Productionc

0.58 0.40 0.18 0.16

Annual Production and Sales

Average Number of Cattle (includes producing and non-

producing cattle) 20 17 2 0.19

Sold milk on a weekly basis 85 74 11 <0.01

Average Number of Bottles Sold Weekly 478 358 119 0.02

Price per Bottle (in US$) Obtained 0.32 0.32 0.00 0.86

Percentage of Production Sold to Final Consumer 40 30 10 0.01

Sold Secondary Dairy Products 26 23 3 0.56

Net Income from Secondary Dairy Products 678 343 335 0.03

Producers 260 258

Randomized Groups 14 14


39

of producers’ gross dairy income found that although treatment producers’ gross income grew at a higher rate from baseline to follow-up than that of the control group, the difference in this growth rate was not statistically significant (difference of 4 percentage points; p-value of 0.88). Further exploratory analyses revealed that producers in the highest income tercile at baseline were largely responsible for the program’s positive impact on net dairy income. (See Appendix B for a full discussion and tables.)

Next, we examine the impact of PBS on dairy producers in Phase I and Phase II of the implementation period. For the dairy chain, we can estimate the effect of six months of PBS assistance under Phase I (April to September 2010) versus no services by comparing treatment and control outcomes for the 2010 rainy season (May to October 2010). We can also estimate the effect of four months of PBS assistance under Phase II (January to April 2011)—following six months of PBS under Phase I—versus no services by comparing treatment and control outcomes for the 2010-2011 dry season (November 2010 to April 2011).

For both seasons, we find positive and significant impacts on the number of bottles of milk sold and secondary dairy sales (Table IV.3). In addition, treatment group producers made between $850 and $1,000 more in net dairy income than control group producers during both phases, and these differences were statistically significant. Given these findings, there is evidence that PBS assistance began generating impacts in Phase I, and these gains were sustained in Phase II.

Context from Qualitative Interviews: Impact in the Dairy Chain

In July 2012, Mathematica staff spoke with FOMILENIO and Chemonics staff about PBS impact findings, particularly the concentration of positive impacts among producers who had relatively strong dairy production before assistance. One Chemonics representative reasoned that this concentration of increased production, sales, and income among established producers reflected a general tendency in the dairy sector, in which established producers have a higher likelihood of improving their production due to their access to investment capital, pre-existing market linkages, and higher degree of organization with fellow producers.

Qualitative interviews with ten PBS participants in the dairy chain yielded another interesting finding: The two milk producers with the highest baseline production and sales levels received the most substantive donations, particularly hay shredders and irrigation systems. Several of the ten interviewed producers noted that the largest producers generally received the most technical and in-kind assistance under PBS, and suggested that future interventions feature a more equitable distribution of donations and training. These qualitative findings strongly corroborate the impact evaluation’s findings: Because the strongest producers appeared to have a natural advantage over small producers due to their existing resources and networks—and they also received a larger proportion of assistance under PBS—these producers’ increases in production and sales were largely responsible for positive impacts in net income for the dairy chain. (See Appendix C for a summary of qualitative interviews with PBS participants in the dairy chain.)


40

Table IV.3. Impacts of PBS in Phase I and Phase II for Dairy Producers (In U.S. Dollars Unless




Notes: Sample size = 518 producers (260 in treatment, 258 in control). Estimates reflect treatment-

control differences at follow-up, after controlling for baseline differences.

a

This is the sum of respondents’ short- and medium-term dairy investments and costs, including the cost

of machinery, cattle feed, vitamins, labor, packaging, transportation, and marketing.

b

Net dairy income is total individual income from milk and dairy sales after subtracting the costs of all

investments and inputs.

c

Number of full-time equivalent jobs is calculated as the number of jobs that require 250 days of work a

year.

2. Horticulture Value Chain

For the horticulture value chain, the goal of PBS is to increase farmers’ income by helping them transition to the production of high value crops, particularly fruits and vegetables. With this goal in mind, we first present productive income from all crops, and then separate productive income into income from fruits and vegetable sales (favored by PBS) and income from other crop sales. First, we found no effects of the offer of PBS on investment and net income related to all crops during the follow-up period. However, we found a positive (and modest) impact of the intervention on investment in horticulture production, but no impact on net income related to fruit and vegetable sales (Table IV.4). When we examine income from other (non-horticulture) crop sales, we find significant and negative effects of the offer of PBS: The treatment group reported making $322 less than the control group in net income from other crop sales.

Rainy Season (Phase I) Dry Season (Phase II)

Measure Impact p-value Impact p-value

Sold Milk on a Weekly Basis (Percentage) 11 <0.01 8 0.04

Dairy Investments and Input Costsa

636 0.13 605 0.16

Net Dairy Incomeb

1,003 0.01 849 0.03

Average Number of Full-time Equivalent Jobs

Offered by Producer Related to Dairy Productionc

0.12 0.11 0.08 0.21

Average Number of Bottles Sold Weekly 135 0.01 111 0.04

Price per Bottle (in US$) Obtained 0.00 0.67 0.00 0.80

Net Income from Secondary Dairy Products 328 0.04 343 0.03


41

Table IV.4. Impact of PBS on Horticulture Producers’ Investment, Income, and Employment (in U.S.

Dollars Unless Otherwise Indicated)

Source: Baseline and follow-up Productive Development Survey-Horticulture (PDS-H) interviews conducted from

June 2010 to June 2011.

Notes: Estimates reflect treatment-control differences at follow-up, after controlling for baseline differences.

Income components do not sum perfectly to total net income, given a procedure by which total net

income was not calculated if more than one income component was missing.

a

This is the sum of respondents’ short- and medium-term agricultural investments and costs, including seeds,

fertilizer, labor, transportation, packaging, and marketing expenses.

b

Net income is gross individual income from crop sales after subtracting the costs of all investments and inputs.

c

Number of full-time equivalent jobs is calculated as the number of jobs that require 250 days of work a year.

When we examine employment for the horticulture chain, we find that the offer of PBS had no effect on employment related to all agricultural production. However, treatment group producers did offer 0.04 more full-time equivalent jobs related to horticulture production than control group producers, and this impact was statistically significant. However, at around 10 days of additional labor per year, this impact of PBS assistance is small in magnitude. (See Table D.6 for standard deviations of economic and employment outcomes.)



Investments and Input Costs for All Cropsa

739 609 130 0.32

Net Income from All Cropsb

-109 191 -300 0.09

Investments and Input Costs for Fruits/Vegetables 212 84 128 0.02

Net Income from Fruits/Vegetables

107 90 16 0.80

Investments and Input Costs for Other Crops 522 525 -3 0.98

Net Income from Other Crops -222 100 -322 0.04


Net Business Income 291 90 202 0.31


Total Net Income 1,508 1,614 -106 0.70


Percentage of Producers Who Employed Workers for

Agricultural Activities

54 52 2 0.52

Average Number of Full-time Equivalent Jobs Offered by

Producer Related to All Cropsc

0.22 0.14 0.08 0.26


Fruit/ Vegetable Production

11 8 3 0.38

Average Number of Full-time Jobs Offered by

Producer Related to Fruits/Vegetables

0.06 0.02 0.04 0.01


Other Crop Production

50 48 2 0.54


Producer Related to Other Crops

0.15 0.12 0.04 0.60

Producers 297 296



42

Turning to analyze agricultural production, we find that treatment group farmers were 11 percentage points more likely to cultivate at least one fruit or vegetable than control group farmers (37 versus 25 percent; statistically significant at the 5 percent level). In addition, the treatment group produced a significantly larger quantity of cucumbers than the control group and a significantly smaller quantity of corn during the implementation period (Table IV.5). Despite their higher production of cucumbers, farmers in the treatment group did not have significantly higher sales of cucumbers (or any other fruits and vegetables) than control group producers during the entire follow-up period.35 Although they did not experience higher fruit and vegetable sales than the control group, treatment group farmers had significantly lower corn, bean, and millet sales than control group farmers.

Summarizing these findings, we find evidence that the offer of PBS influenced treatment group farmers to increase fruit and vegetable production (especially cucumber). However, we do not find evidence that the offer of PBS translated into higher agricultural income for treatment group farmers. (See Table D.7 for additional impacts on producers’ agricultural production.)

We conducted exploratory analyses to confirm that the treatment group’s higher production of vegetables was associated with the receipt of PBS services. These analyses revealed evidence that producers who received a high number of donations and training sessions in Phase I experienced the largest increase in the number of fruits and vegetables cultivated. These producers also experienced the largest decrease in the amount of corn cultivated during the study period. This provides evidence that receipt of PBS services was associated with an increase in horticulture production and a decrease in basic grain production. (See Appendix B for a full discussion and tables.)

35 Examining cucumber production in more detail, farmers in the treatment group had significantly higher

cucumber sales than farmers in the control group during the dry season in early 2011, but not during the rainy season in late 2010 (not shown). This discrepancy appears related to treatment group producers’ significantly higher average levels of cucumber production in the dry season, but not in the rainy season.


43

Table IV.5. Impact of PBS on Horticulture Producers’ Agricultural Production (Averages Unless


Source: Baseline and follow-up Productive Development Survey-Horticulture (PDS-H) interviews conducted

from June 2010 to June 2011.


differences. Non-percentage estimates are unconditional, meaning that farmers who did not

produce specific crops were given values of zero for hectares under production, tons produced and

sales for those crops.

Next, we examine the impact of PBS on horticulture producers in Phase I and Phase II of implementation. For horticulture, we estimate the effect of five months of Phase I services (from April to August 2010) versus no services by comparing treatment and control outcomes for the 2010 rainy season (May to October 2010). We also estimate the effect of four months of Phase II services (from January to April 2011)—following six months of Phase I services—versus no services by comparing treatment and control outcomes for the 2010-2011 dry season (November 2010 to April 2011).


Annual Fruit and Vegetable Production

Farmers Who Produced Fruits or Vegetables (%) 37 25 11 0.02

Hectares Devoted to the Following Vegetables:

Squash 0.06 0.05 0.01 0.61

Cucumber 0.05 0.01 0.04 <0.01

Tomato 0.03 0.01 0.02 0.09

Tons Produced of the Following Vegetables:

Squash 0.08 0.13 -0.05 0.47

Cucumber 0.20 0.08 0.12 0.02

Tomato 0.16 0.08 0.08 0.32

Farmers Who Sold Fruits or Vegetables (%) 30 22 8 0.05

Sales of the Following Vegetables (US$):

Squash 37 49 -12 0.65

Cucumber 24 13 11 0.14

Tomato 89 69 20 0.65

Annual Production of Other Crops

Farmers Who Produced Other Crops (%) 79 78 1 0.75

Hectares Devoted to the Following Crops:

Corn 0.77 0.92 -0.15 0.04

Beans 0.38 0.44 -0.06 0.23

Millet 0.34 0.44 -0.10 0.11

Tons Produced of the Following Crops:

Corn 1.33 1.69 -0.36 0.05

Beans 0.13 0.21 -0.08 0.09

Millet 0.29 0.35 -0.06 0.52

Farmers Who Sold Other Crops (%) 51 54 -3 0.53

Sales of the Following Crops (US$):

Corn 136 283 -146 0.01

Beans 39 123 -84 <0.01

Millet 14 38 -24 0.01

Producers 297 296



44

For Phase I, we find that treatment group farmers were 13 percentage points more likely to produce at least one fruit or vegetable than control group famers, and treatment group farmers made larger investments in horticulture than control group farmers. For Phase II, however, we find no significant difference between treatment and control group farmers’ likelihood of producing fruits or vegetables. As illustrated in Table IV.6, we also see that treatment group producers made over $230 less from non-horticulture crop sales than control group producers during Phase I implementation (significant at the 5 percent level), but did not make significantly more from horticulture sales during the same period. This negative effect of PBS on non-horticulture sales is not present during Phase II. This suggests that if producers in fact lost income as a result of substituting high-value crops for low-value crops, the majority of these losses occurred during Phase I (the 2010 rainy season).

Table IV.6. Impacts of PBS in Phase I and Phase II for Horticulture Producers (in U.S. Dollars Unless


Source: Baseline and follow-up Productive Development Survey-Horticulture (PDS-H) interviews conducted

from June 2010 to June 2011.

Notes: Sample size = 593 producers (297 in treatment and 296 in control). Estimates reflect treatment-

control differences at follow-up, after controlling for baseline differences.

a

This is the sum of respondents’ short- and medium-term agricultural investments and costs, including seeds,


b

Net income is gross individual income from crop sales after subtracting the costs of all investments and inputs.

c


Rainy Season (Phase I) Dry Season (Phase II)

Measure Impact p-value Impact p-value

Percentage of Farmers Who Produced Fruits or

Vegetables 13 <0.01 5 0.38

Percentage of Farmers Who Sold Fruits or Vegetables 5 0.14 7 0.07

Percentage of Farmers Who Produced Other Crops 0 0.97 -2 0.44

Percentage of Farmers Who Sold Other Crops -3 0.51 -1 0.53

All Agricultural Investments and Input Costsa

36 0.78 89 0.07

Net Agricultural Incomeb

-232 0.09 -56 0.53

Horticulture Investments and Input Costs 42 0.01 77 0.06

Investments and Input Costs for Other Crops -16 0.90 9 0.68

Net Horticulture Income -3 0.96 22 0.41

Net Income from Other Crops -235 0.04 -83 0.32


Producer Related to Horticulturec

0.01 0.05 0.03 0.05


Producer Related to Other Crops 0.03 0.62 0.00 0.80


45

3. Handicrafts Value Chain

In the handicrafts value chain, we found no significant treatment-control differences in investment or productive income (Table IV.7). However there is some evidence that treatment group artisans invested slightly more in handicrafts than control group artisans in the intervention period, but this modest investment did not produce an economic return. Given the lack of statistical significance of these differences, this evidence is inconclusive. However, we find that the offer of PBS had a positive and significant effect on the number of full-time jobs offered by artisans. The treatment group offered 0.13 more full-time jobs than the control group, which translates to around 33 additional days of labor per year. This difference is statistically significant at the 1 percent level. (See Tables D.8, D.9, and D.10 for standard deviations of artisans’ economic and employment outcomes, PBS impacts among female- versus non-female-headed households, and additional information on group outcomes, respectively.)

Table IV.7. Impact of PBS on Handicraft Producers’ Investment, Income, Employment, and Labor (in

U.S. Dollars Unless Otherwise Indicated)


Annual Investment, Sales, and Income (US$)a

Sold Handicrafts Last Year (Percentage) 74 58 16 0.07

Handicraft Investments and Input Costsa

474 423 51 0.65

Net Handicraft Incomeb

679 654 24 0.92

Income from Salaries 340 340 0 1.00

Net Business Income 106 131 -25 0.28


Total Net Income 1,516 1,443 73 0.78


Percentage of Artisans Who Employ

Workers Outside the Workshop or Group

12 6 6 0.11

Average Number of Full-time Equivalent

Jobs Offered by Artisan Last Year Related

to Handicraftsc

0.15 0.02 0.13 <0.01

Context from Qualitative Interviews: Impact in the Horticulture Chain

In July 2012, Mathematica staff spoke with FOMILENIO and Chemonics staff about impact findings in the horticulture chain, particularly the apparent transition to vegetable production but no accompanying increase in income from vegetable sales. One Chemonics representative reasoned that these results likely reflected the gap in services between late 2010 and early 2011. Many PBS participants in the study sample received seeds, plants, and technical assistance with soil management during the 2010 rainy season, but did not receive any assistance for at least three months at the end of Phase I and before the start of Phase II. Partly as a result of this gap in assistance, many of these producers lost their crops or produced poor quality crops in late 2010 and early 2011. Perhaps for this reason, we measure increased investments in horticulture production—particularly in Phase I—but no apparent benefit of those investments in the form of increased net income from horticulture sales.


46

Source: Baseline and Follow-up Productive Development Survey-Handicrafts (PDS-H) interviews conducted

from October 2009 to November 2010.

Note: Estimates reflect treatment-control differences at follow-up, controlling for baseline differences.

Income components do not sum perfectly to total net income, given a procedure by which total net

income was not calculated if more than one income component was missing.

a

This is the sum of respondents’ short- and medium-term handicraft investments and costs, including raw

materials, labor, transportation, packaging, and marketing expenses.

b

Net handicrafts income is individual income from handicraft sales after subtracting costs of all investments and

inputs.

c


4. Comparison of Intent-to-Treat and Treatment-on-the-Treated Impact Estimates

In this section, we summarize the impacts of PBS at the producer level and compare and contrast intent-to-treat (ITT) and treatment-on-the-treated (TOT) estimates at this level. As mentioned, ITT estimates can be interpreted as the effect of the offer of PBS assistance, regardless of whether the services were accepted, and TOT estimates can be interpreted as the effect of PBS on those who accepted these services. Under both ITT and TOT approaches, we find that the offer of PBS assistance had a statistically significant and positive impact on employment generation among artisans. Among all artisans in the treatment group, PBS assistance resulted in 0.13 additional jobs (ITT, Figure IV.1), and among artisans who participated in the intervention, PBS assistance resulted in 0.19 additional jobs (TOT, Table IV.8). This difference is equivalent to nearly 50 days of full-time employment per year.

Labor Devoted to Handicrafts

Average Months Worked in Handicrafts by

Artisan During Last Year

7.1 5.7 1.4 0.12

Average Days Worked per Month During

the Month with Highest Sales

6.4 5.1 1.3 0.33

Average Days Worked per Month During

the Month with Lowest Sales

5.0 3.4 1.7 0.09

Producers 303 322

Randomized Municipalities 9 10

Context from Qualitative Interviews: Impact in the Handicrafts Chain

In July 2012, Mathematica staff spoke with FOMILENIO and Chemonics staff about impact findings in the handicrafts chain. One Chemonics representative reasoned that the impact evaluation’s interim results of a positive impact in paid labor—but no impact in sales and production—likely reflected the state of handicraft production among artisans in the study sample during Phase I. By late 2010, assisted artisans had begun to invest more in production, including contracting paid labor, but this production had not yet resulted in increased sales at the time of the survey. The representative suggested that the evaluation’s second follow-up surveys would likely capture positive impacts in sales, particularly during December 2010 and December 2011. In addition, stakeholders mentioned that Phase I assistance in the handicrafts chain had a strong emphasis on production, but a relatively weak emphasis on market access compared to Phase II assistance. The lack of positive impacts on sales and income in the evaluation may reflect this prioritization of production over sales during Phase I.


47

Figure IV.1. Annual Employment Generated by PBS Participants, by Value Chain (in Full-time Jobs)


2009 to July 2011.


differences. Employment in the horticulture chain reflects all employment related to agricultural



Table IV.8. Impacts of PBS at the Producer Level, by Value Chain (in U.S. Dollars Unless Otherwise

Indicated)


Impact p-value Impact p-value Impact p-value


Investments and Input Costs 1,160 0.16 130 0.32 52 0. 64


Generateda

0.18 0.16 0.08 0.26 0.13 <0.01

Net Productive Incomeb 1,849 0.01 -300 0.09 23 0.92


Investments and Input Costs 1,998 0.16 428 0.34 80 0. 65


Generateda

0.31 0.16 0.26 0.30 0.19 <0.01

Net Productive Incomeb 3,114 0.01 -983 0.10 36 0.92

Sample Size 518

593

625


2009 to July 2011.

Notes: Estimates reflect treatment-control differences at follow-up, after controlling for baseline differences.

Participation was 79, 33, and 65 percent for the dairy, horticulture, and handicrafts chains,

respectively.

a


b


As discussed above, we find evidence that the offer of PBS had a significant positive impact of around $1,850 on dairy producers’ productive income (ITT, Table IV.8 and Figure IV.2). At over $3,000, this impact is even larger among individuals who participated in PBS assistance (TOT,

0.58

0.22

0.15

0.40

0.14

0.02

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0


Treatment

Control No Significant

Impact

Impact = 0.13

p-value < 0.01

No Significant

Impact


48

Table IV.8). Across all three value chains, we detected no significant impacts of PBS on producers’ investments and costs using ITT and TOT approaches.

Figure IV.2. Net Annual Productive Income of PBS Participants, by Value Chain (in U.S. Dollars)


2009 to July 2011.


differences. Productive income in the horticulture chain reflects all agricultural income. Treatment

group participation rates were 79, 33, and 65 percent for the dairy, horticulture, and handicrafts

chains, respectively.

C. Impacts of PBS on Household Income, Consumption, and Poverty

In this section, we focus on income and consumption at the household level. This information comes from sections in the PDS in which we asked respondents about the income of each member of their household, in addition to monthly, bi-yearly, and annual household expenditures. Based on these responses, we created household-level income and consumption measures. If two respondents in the study sample belonged to the same household, their responses were averaged to create one set of income and consumption variables per household. We present ITT estimates at the household level, followed by a comparison of ITT and TOT estimates at this level.

Examining income measures, we find that the offer of PBS may have had a similar effect on households as producers, particularly for the dairy chain. As shown in Table IV.9, we find some positive and statistically significant effects on household income for the dairy value chain, but we find no effects for the horticulture and handicrafts value chains. In the dairy chain, treatment group households made on average around $2,000 more in net dairy income than control group households, and this difference is statistically significant at the 5 percent level. When we include income from other sources, treatment group households in the dairy value chain make on average around $2,600 more in net income than control group households (Figure IV.3). However, this difference is not statistically significant. In part, the lack of statistical significance of this treatment-control difference in total income at the household level reflects the low level of statistical precision in the study, primarily caused by the small number of randomized producer groups in the sample.

In summary, we cannot conclude with certainty that the offer of PBS improved the net income of households in the dairy chain. Similar to our producer-level analysis, we find no significant impacts of PBS on household income for the horticulture and handicraft chains. (See Table D.11 for standard deviations of households’ economic outcomes.)

3,608

-109

679

1,759

191

654

-500

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000


Treatment

Control

No Significant

Impact

No Significant

Impact

Impact = $1,849

p-value = 0.01

49

Table IV.9. Impact of PBS on Annual Household Income, Consumption, and Poverty, by Value Chain (in U.S. Dollars Unless Otherwise

Indicated)



Net Productive Income 4,034 2,018 2,017 0.02 -54 198 -252 0.21 646 657 -11 0.96

Total Net Incomea

9,231 6,622 2,608 0.12 2,869 2,927 -58 0.87 2,464 2,615 -151 0.71

Consumptionb

6,062 4,841 1,221 0.13 2,294 2,246 48 0.82 1,383 1,489 -106 0.55

Percentage of Households in

Extreme or Relative Povertyc

15 16 -1 0.73 49 55 -6 0.14 66 62 4 0.47

Households 255 250

282 277

285 286


Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from September 2009 to July 2011.


a

This is the sum of all household members’ productive income, wages, business income, and additional income (including remittances), after

subtracting agricultural and business expenses.

b

Includes all household expenses on food, household items, utilities, health care, transportation, and education, among others.

c

Extreme poverty is defined as gross household income of less than $0.93 per person per day. Relative poverty is defined as gross household income

of less than $1.86 per person per day (DIGESTYC 2009).


50

Figure IV.3. Net Annual Household Income of PBS Participants, by Value Chain (in U.S. Dollars)




differences. Household participation rates were 73, 33, and 67 percent for the dairy,


In addition, we find no significant effects of the offer of PBS on consumption, despite a sizable treatment-control difference of more than $1,200 in consumption in the dairy chain (Figure IV.4). This is likely related to the low level of statistical precision in the study, primarily caused by the small number of randomized producer groups in the study sample. In all three value chains, treatment and control groups also had similar poverty indicators (Table IV.9).

Figure IV.4. Annual Household Consumption of PBS Participants, by Value Chain (in U.S. Dollars)




differences. Household participation rates were 73, 33, and 67 percent for the dairy,


9,231

2,869 2,464

6,622

2,927 2,615

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000


Treatment

Control

6,062

2,294

1,383

4,841

2,246

1,489

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000


Treatment

Control

No Significant

Impact No Significant

Impact

No Significant

Impact

No Significant

Impact

Difference = $2,608

p-value = 0.12

Difference = $1,221

p-value = 0.13


51

In Table IV.10, we summarize the impacts of PBS at the household level and compare and contrast ITT and TOT estimates for households in the evaluation. As illustrated, we found no statistically significant impacts of PBS assistance on total net household income or consumption for any value chain (ITT and TOT).

Table IV.10. Impacts of PBS at the Household Level, by Value Chain (in U.S. Dollars Unless Otherwise

Indicated)


Impact p-value Impact p-value Impact p-value


Net Household Income (from all sources) 2,608 0.12 -58 0.87 -151 0.71

Household Consumption 1,221 0.13 48 0.82 -106 0.55


Net Household Income (from all sources) 4,294 0.11 -194 0.87 -227 0.71

Household Consumption 2,011 0.12 159 0.82 -156 0.54

Sample Size 505

559

573




differences. Participation rates were 73, 33, and 67 percent for the dairy, horticulture, and

handicrafts chains, respectively. Defined at the household level, these participation rates are

slightly different than rates used in Table IV.9.

D. Economic Rates of Return

To assess the anticipated economic impact of any intervention, MCC uses economic rates return (ERR) to determine whether the benefits of an intervention exceed its costs over a certain period of time. As defined by MCC, ERR calculations consider the benefits and costs to all members of society. This includes costs incurred by anyone impacted by the project, all costs paid by MCC, costs by other donors or local governments, and resource costs when they are incurred. The benefits include increments in income for the beneficiaries (above and beyond what they would have made without the program); cost savings; time savings; and indirect benefits (for example, fiscal savings or increased value of natural resources).

The ERR is calculated as the discount rate at which the net present value (NPV) of the total benefits of the program equals the NPV of the total costs over the determined period of time (normally 20 years). To construct ex-ante ERRs before projects are implemented, MCC uses information from a variety of data sources to estimate the potential benefits and costs of projects.

Regarding next steps related to ERR calculations of PBS assistance, we will discuss options to calculate an ex-post ERR for the PBS Activity with MCC staff. We should note that MCC’s existing ERR for the Production Development Project includes the project’s three main activities. In contrast, only one activity (PBS) is explicitly studied in this report. This will be one of the primary challenges we will face when calculating the actual ERRs. In addition, MCC staff completed separate ERR calculations for four value chains (dairy, horticulture, fruits, and forestry), but also calculated an overall ERR across the four value chains. We should note that our evaluation includes the dairy chain, but we do not separate horticulture and fruit production as in the current ERR calculations. More importantly, our evaluation covers only a small portion of the total number of PBS


52

participants as of November 2011, and our calculations should be adjusted to account for this discrepancy. In conversations with MCC, we will discuss how to address these issues.

We are currently considering two approaches for calculating the ex-post ERR. First, we can employ the same model that MCC used to calculate the ex-ante ERR and verify if the values for all the parameters initially assumed for the ex-ante calculation are supported by survey data. In addition, instead of using the estimated expenditure for costs, we could obtain actual cost data from FOMILENIO to incorporate more updated cost information into the calculations. The second approach would be to propose a different model for ERR calculation that incorporates additional benefits measured by the evaluation. We will discussing these options with MCC in order to determine the best process to calculate an ex-post ERR.

53

V. CONCLUSIONS

A. Key Findings

In brief, we believe there are two key findings regarding the impact of PBS on employment and income:

First, we find that the offer (and delivery) of PBS services had a statistically significant and substantively positive impact on employment generation among artisans. This is a promising finding, as it provides evidence that artisans who participated in training and technical assistance made a tangible investment in additional labor related to their production.

Second, there is strong evidence that the offer (and delivery) of PBS assistance to dairy producers significantly raised these individuals’ productive incomes. Our analysis of intermediate outcomes suggests that treatment group producers in the dairy chain sold a larger volume of milk, as well as more secondary dairy products than control group producers. The positive impact of PBS assistance on producers’ net income is likely related to both of these products.

To some extent, we can compare these findings with the activity’s ultimate goal of a 15 percent increase in annual income. In the dairy chain, treatment producers had an average net annual productive income of around $3,600 and a total net income of around $6,300 during the implementation period (ITT). This was over 45 percent higher than control group producers’ average productive income and total net income during the same period. These impacts were even higher among PBS participants, who had an average net productive income of over $4,700 during the follow-up period, compared to under $1,700 among control group producers (TOT). However, we found no significant impacts on income for the other value chains, either at the producer or household level. Given the absence of significant impacts in these two chains, we cannot conclude that one year of PBS assistance is associated with an average income increase of 15 percent among all three value chains in the evaluation.36

B. Discussion

These key findings suggest that interventions of this type, particularly in the dairy chain, can generate significant effects on income within a relatively short time period of one year (or even one agricultural season). This result coincides with expectations set by Chemonics and FOMILENIO regarding the timeframe in which benefits of assistance would accrue. However, the concentration of positive impacts in dairy value chain underscores the importance of sector-specific factors—including market conditions as well as technical assistance provided—in influencing participants’ productive and economic outcomes.

36 It should be noted that this income goal corresponds to a 15 percent increase in income during the full five-year

compact period. In contrast, the time period covered in this interim evaluation is one year. In addition, this income goal is a before-after difference, whereas this impact analysis is centered upon comparing treatment and control groups at follow-up.

V. Conclusion Mathematica Policy Research

54

Triangulating impact estimates in light of program implementation findings, perhaps the relatively high frequency of assistance provided to most dairy producers in the study—compared to the lower frequency of assistance provided to a minority of horticulture producers in the study—was instrumental in generating positive economic impacts in the dairy chain as opposed to the horticulture chain. The quality and content of such assistance, and its relevance to producers’ needs, may have also influenced differential impacts in the dairy and horticulture chains.

In the handicrafts chain, our finding of positive employment generation but no significant impact on productive or household income is somewhat puzzling. According to stakeholder interviews and administrative data, a majority of artisans in the treatment group received regular assistance during Phase I. As such, we cannot hypothesize that the amount and frequency of assistance was the primary obstacle to improved outcomes. Because there is some evidence that artisans in the treatment group adopted new practices and invested more in handicraft production—including contracting over one month of additional labor per year, on average—we can conjecture that PBS assistance did increase and enhance handicraft production. However, it appears that these improvements did not lead to increased sales and income. This suggests that one or more structural obstacles to marketing and selling handicrafts may have inhibited positive impacts on artisans’ productive income. Examples of potential obstacles could include market access difficulties, limited demand during non-peak months, or the inability of assisted artisan groups to ensure the quality of their goods to secure large orders. Interestingly, Phase II assistance in the handicrafts chain featured a much stronger emphasis on market access than Phase I. Two- and three-year impact findings for the horticulture chain may reveal the potential effect of this market emphasis on artisans’ sales and income.

C. Next Steps

In September 2011, data collectors completed the second follow-up survey for the handicrafts value chain. A final round for the handicrafts survey is scheduled for August 2012. Scheduled for submission in January 2013, the final report of the PBS impact evaluation will present three-year impact findings for the handicrafts chain. In addition, Mathematica will complete a performance analysis of PBS assistance in the dairy, horticulture, and handicrafts value chains for submission in September 2012.

55

REFERENCES

―Baseline Comparison of Dairy, Horticulture, and Handicraft Producers in the Production Business Services Activity.‖ Submitted to MCC by Randall Blair and Larissa Campuzano on February 18, 2011.

Centro Nacional de Tecnología Agropecuaria y Forestal. PowerPoint presentation on losses related to tropical storms, 2010.

Chemonics. October 2011 Report: Productive Development Project.

Chemonics. Quarterly Report: Production and Business Services. September 15–December 14, 2011.

―Characterization of Beneficiaries of the PDP Project: Pilot Projects and General Implementation.‖ FOMILENIO, Office of Monitoring and Evaluation, October 2010.

―Compact Between MCC and El Salvador,‖ section 2(b) of schedule 2 to annex I, November 29, 2006.

―Final Impact Evaluation Design for the Production and Business Services Activity of the Productive Development Project. Submitted to MCC by Randall Blair and Larissa Campuzano on February 3, 2011.

―Final Report: Qualitative Data Collection with Beneficiaries, 2010.‖ ADEPRO. Submitted to FOMILENIO on January 25, 2011.

FOMILENIO. Electronic Bulletin #11: ―FOMILENIO Implements Emergency Plan, October 2011.

―Indicator Tracking Table (ITT),‖ El Salvador, Productive Development Project, May 2011.

Operations Manual, Productive Development Project, FOMILENIO, December 2010.

SNC-Lavalin. PDP Operations Manual, ACT-140, Report 8, July 2008.

APPENDIX A

SELECTION CRITERIA FOR PBS ASSISTANCE

Appendix A Mathematica Policy Research

A.2

Phase I: Selection Criteria for Production and Business Services

Selection Criteria for the Horticulture Value Chain

The individual must comply with each of the following requirements:

Experience or interest in producing and selling vegetables or fruits

Interest in forming alliances with other groups of producers in the Northern Zone

Interest in coordinating the sale of their products with those of other groups

Desire to participate in project activities, including sharing lessons learned with other producers

Willing to offer a counterpart contribution

Have access to land, preferably their own land, for productive activities

Selection Criteria for the Dairy Value Chain


Be an active cattle farmer in the Northern Zone

Willing to assist and participate in training sessions, field days and demonstrations

Possess a desire to increase the size of their herd

Willing to use and offer information about positive technical and financial changes in their production

Willing to make investments with their own funds

Willing to implement proposed technological changes

Selection Criteria for the Handicrafts Value Chain


Experience or interest in producing and selling handicrafts

Work within associated groups with at least 10 members or small businesses with at least 5 employees

Receive a score of at least 2 or more in all the diagnostic questions, and at least a total score of 34

Service providers will give priority to groups with many members, groups with good market potential, and groups with a strong entrepreneurial attitude


A.3

Phase II: Selection Criteria for Production and Business Services

Selection Criteria for the Dairy Value Chain


Dairy production and livestock are the individual’s main profession

The productive unit is located in the Northern Zone

The individual has not participated and/or received grants in a different value chain in a previous implementation phase

The individual is willing to actively attend training sessions and adopt new technologies featured in training

The individual agrees to regularly provide information on levels of production and performance

The individual has a minimum of 10 animals capable of reproduction

The individual is not currently being assisted under another productive chain

Selection Criteria for the Horticulture Value Chain


Experience or interest in producing and marketing fresh vegetables or fruit

Interest in forming alliances with other groups of producers in the Northern Zone

Interest in marketing and selling their products with those of other groups

Willingness to participate in project activities, including sharing lessons learned with other producers

Gender participation in commercial activities of the group

Willingness to provide a counterpart contribution

Has access to productive agricultural land

The size of the production area for this component is not a determining factor, provided that it allows the farmer to produce crops with a market focus and generate sustainable income. The minimum area available is subject to the type of crop to be established, provided that the producer is available to cultivate their own land or use rented land for cultivation.

Selection Criteria for the Handicrafts Value Chain

The individuals and groups must comply with each of the following requirements:

Be willing to work with textiles, clay, wood, or recycled materials

Have at least the basic tools for making the products they manufacture


A.4

Be organized in groups of between 2-8 members

At least 2 of the members must be able to read, write and perform basic arithmetic (add, subtract, multiply and divide)

Willingness to work with one of the project’s ―anchor‖ associations

Possess an entrepreneurial spirit and a willingness to improve their job skills and get involved in all phases of the business (production, marketing their products, and managing activities as appropriate)

Already produce handicrafts that meet current market needs (or incorporate suggestions and proposals of the anchor associations)

Have products with an acceptable level of quality to meet market demand or are otherwise easily modified to meet market demand (to be evaluated by the program specialist and anchor association)

Have a list of prices that permit the marketing of products in target markets defined by the anchor association

Have a sustainable raw material supply

Have a disposition to work and coordinate with a diverse group of local and external organizations (municipalities, town councils, NGOs, etc.)

Willingness to be trained in human rights and economic issues related to the success of the program

APPENDIX B

CONTEXTUAL INFORMATION, SENSITIVITY TESTS, AND

EXPLORATORY ANALYSES

Appendix B Mathematica Policy Research

B.2

Additional Analyses for the Dairy Chain

1. Information on Milk Prices

To provide context for impact estimates discussed above, Figure B.1 illustrates consumer prices of milk during the baseline and follow-up period. As illustrated, milk prices increased an average of $0.05 per bottle (or around 10 percent) during the follow-up period compared to the baseline period. At an event in March 2012, stakeholders mentioned that the Vaso de Leche program instituted by the Salvadoran government had some role in raising consumer demand for milk. Implemented by the Ministry of Agriculture and the Ministry of Education in March 2011, the program provides approximately 248,000 students in 74 municipalities in El Salvador with free milk. However, because the program was initiated near the end of the interim evaluation period, it likely had a small effect on producers’ reported income.

Figure B.1. Consumer Prices for Milk in El Salvador, 2009-2011 (Averages in U.S. Dollars per Bottle)

Source: Ministry of Agriculture, El Salvador, 2012.

Both treatment and control group farmers likely saw greater returns to milk production and sales during 2010 and early 2011, related to the generalized price increase during this period. Overall, the effect of this price increase would be the same across treatment and control groups, and would thus not constitute a driving factor in impact estimates. However, this generalized increase in the price of milk is likely a factor in the overall increase in gross productive income for both groups during the study period. According to raw baseline and follow-up data, the control group’s average gross dairy income increased over $300 from $6,645 at baseline to $6,968 at follow-up (an increase

$0.40

$0.45

$0.50

$0.55

$0.60

MY JN JL AG SP OC NV DC JN FB MR AP MY JN JL AG SP OC NV DC JN FB MR AP

2009 2010 2011

Evaluation Period


B.3

of 5 percent), and the treatment group’s average net dairy income increased nearly $1,400 from $10,525 at baseline to $11,912 at follow-up (an increase of 13 percent).37

2. Sensitivity Tests of Impacts on Net Productive Income in the Dairy Chain

We conducted several sensitivity tests to examine the robustness of the key finding that PBS assistance generated higher net incomes of around $1,850 during the course of one calendar year, on average. These tests were important given the statistically significant baseline treatment-control difference in net productive income, a primary outcome of interest. As shown in Table B.1, these sensitivity tests revealed that the impact of PBS on productive income is robust to various transformations and alternate specifications.

First, we calculated impact estimates for net productive income using a series of regression specifications. Regardless of the combination of baseline, geographic, and demographic controls in the regression, impact estimates were statistically significant and ranged from $1,533 to $1,883. Next, we examined the impact of PBS on gross productive income, a highly related measure to net productive income. This impact was large in magnitude and statistically significant (impact of $3,254 with a p-value of 0.01). In addition, the impact of PBS on net productive income remained statistically significant after extreme outliers (individuals in the top percentile of baseline net household income) were excluded from the study sample. However, after these outliers were excluded, the impact estimate dropped substantially from around $1,850 to $1,169. In addition, an analysis using the natural log of gross productive income showed that the treatment group experienced higher growth in gross income on the order of 4 percentage points during the study period, after controlling for baseline characteristics. However, this higher growth rate was not statistically significant.

37 Due to concerns that labor costs may have been overestimated at baseline, we do not present raw differences in

baseline and follow-up means for net productive income. If labor costs were overestimated at baseline and more accurately measured at follow-up, the change in net income from baseline to follow-up would be upwardly biased. Because we believe that baseline labor costs were overestimated by a similar magnitude across treatment and control groups, regression analysis still provides a largely unbiased estimate of the impact of PBS assistance on net productive income.


B.4

Table B.1. Description of Sensitivity Tests, Dairy Chain

Sensitivity Test Rationale Findings

Exclude different combinations

of covariates—including

demographic characteristics,

geographic controls, and

baseline values—in regressions

predicting the impact of PBS on

net productive income

Determine the extent to which

covariates alter the magnitude and

statistical significance of the impact.

Impacts ranged from $1,533

to $1,883, and were all

statistically significant at the 5

percent level.

Examine impacts on gross

productive income

Examine a measure closely related to

net productive income to confirm

that impacts on sales are statistically

significant, before subtracting

production costs.

Impact was $3,254 and

statistically significant at the 1

percent level.

Estimate impacts on the natural

log of gross productive income

Determine whether PBS assistance

affected income growth differentially

between treatment and control

groups.

Controlling for other factors,

the treatment group

experienced higher growth in

gross income on the order of 4

percentage points. This

treatment-control difference

was not statistically significant.

Exclude extreme outliers from

the analysis (individuals in the

top percentile of baseline net

household income)a

Determine whether a few relatively

high-resource individuals have a

substantial effect on impact

estimates.

Impact was statistically

significant at the 5 percent

level, but is smaller in

magnitude at $1,169.

a

This test reduced the sample size to 511 producers, and thus its significance level is not directly comparable

with those of the tests involving gross productive income and the log of net productive income (which used the

full sample of 518 producers).

The results of sensitivity tests related to excluding the top percentile in baseline productive income and the natural log of income suggest that a subgroup of higher-income participants could have a disproportionate effect on impact estimates. To better understand this phenomenon, we estimated impacts according to producers’ initial net productive income. As illustrated, producers in the highest income tercile at baseline experienced the largest impacts on net productive income (Table B.2). These results suggest that relatively wealthy producers in the treatment group benefited most from PBS assistance, whereas other dairy producers in the sample experienced mixed results.

Table B.2. Estimated Impact of PBS on Net Productive Income, by Baseline Net Income (in U.S. Dollars)

Tercile Impact p-value Sample Size Randomized

Groups

First (Net income of less than

-$868 at baseline) -395 0.73 173 28

Second (Net income of between

-$868 and $1,042 at baseline) 93 0.92 173 27

Third (Net income of more than

$1,042 at baseline) 1,795 0.17 173 27



Note: Impacts are follow-up treatment-control differences, controlling for baseline differences.


B.5

3. Additional Exploratory Analyses of Impact Estimates

In addition to the sensitivity tests discussed above, we conducted additional analyses to explore the relationship between program assistance and improved outcomes. First, we defined several subgroups of producers who would hypothetically benefit the most from PBS assistance, including producers who received several donations and participated in multiple training sessions. Next, we defined subgroups of producers who would hypothetically not be affected by PBS assistance, particularly control group producers and treatment group producers who did not participate in PBS assistance during either phase. Then we calculated the average change in net productive income from baseline to follow-up for each subgroup, and arranged the groups in order from the largest improvement to the smallest improvement (Table B.3).

As expected, treatment producers that received large numbers of donations and training sessions experienced the largest increase in net income during the study period, and control group producers (as well as non-participating treatment group producers) experienced the smallest increase in net income during the same period. Due to potential selection effects inherent in these groups, these averages should not be interpreted as impact estimates. However, they suggest a correlation between the receipt of PBS assistance and improved outcomes.

Table B.3. Change in Net Productive Income from Baseline to Follow-Up, by Subgroup (Averages

in U.S. Dollars)

Measure

Change from

Baseline to

Follow-up Producers

Treatment Producers Who Received at Least 5 Training Sessions or

Technical Assistance Visits (Phase I)

3,426 58

Treatment Producers Who Received at Least 15 Donated Items

(Phase I)

3,318 103

Treatment Producers who Adopted at Least Three Practices 1,661 127

Treatment Producers who Participated in Either Phase 1,651 187

All Treatment Producers 1,483 260

Treatment Producers who Did Not Participate in Either Phase 1,049 73

All Control Producers 570 258



In addition, we conducted an additional analysis to determine whether large impacts in net productive income were associated with membership in a dairy cooperative or enterprise at baseline. We found that cooperative membership among treatment group producers was correlated with positive impacts (with an interaction term of $3,841), but this analysis was inconclusive due to the interaction coefficient’s lack of statistical significance. Similarly, we conducted an analysis to determine whether large impacts in net productive income were associated with specific regions or departments. We found that treatment group producers in the department of La Unión were more likely to experience positive impacts (with an interaction term of $2,385), but this analysis was also inconclusive due to the interaction coefficient’s lack of statistical significance. In addition, we examined cooperative/group membership among the 25 dairy producers who reported the largest increase in net income from baseline to follow-up. Interestingly, these 25 producers were distributed in 13 cooperatives/groups, with one to two producers in most groups. However, one group (located in La Unión) had 7 members who were among these 25 individuals.


B.6

Additional Analyses for the Horticulture Chain

1. Information on Crop Prices

To provide context for impact estimates, Figure B.2 illustrates consumer prices of common vegetables and grains produced by farmers in the horticulture value chain. As illustrated, tomato prices fluctuated greatly from 2009 to 2011, and peaked in October and November 2010 during the study’s follow-up period. Similarly, the price of beans rose sharply in late 2010, and remained high throughout the follow-up period in early 2011. Given their higher average production and sale of beans, control group farmers were better positioned than treatment group farmers to benefit from these higher market prices of beans in late 2010 and 2011. Because there is not enough evidence to conclude that treatment farmers produced and sold a larger portion of tomatoes than control farmers, it does not appear that either group was better positioned to generate profits from tomato sales in late 2010, at the height of the crop’s market price from early 2009 to early 2011.


B.7

Figure B.2. Consumer Prices for Vegetables and Basic Grains in El Salvador, 2009-2011 (Averages in

U.S. Dollars per Pound)

Source: Ministry of Agriculture, El Salvador, 2012.

Note: Unfortunately, market prices for squash (pipián and ayote) were unavailable for the desired time

period.

$0.00

$0.20

$0.40

$0.60

$0.80

$1.00

$1.20

$1.40


2009 2010 2011

Cucumber

Tomato

$0.00

$0.20

$0.40

$0.60

$0.80

$1.00

$1.20

$1.40


2009 2010 2011

Corn

Beans

Millet

Evaluation Period

Evaluation Period


B.8

2. Additional Exploratory Analyses of Impact Estimates in the Horticulture Chain

In addition to the analyses above, we conducted additional analyses to explore the relationship between program assistance and behavior change in the horticulture chain. First, we defined several subgroups of producers who could hypothetically change their cropping patterns as a result of PBS assistance, including producers who received several donations and participated in several training sessions. Next, we defined subgroups of producers who would hypothetically be unaffected by PBS assistance, particularly treatment group producers who did not participate in PBS assistance during either phase. Then we calculated the average change in the number of fruits and vegetables cultivated and sold from baseline to follow-up for each subgroup, and arranged the groups in order from the largest change to the smallest change.

As expected, subgroups that received large numbers of donations and training sessions experienced the largest increase in the number of fruits and vegetables cultivated and sold during the study period, and treatment group producers who did not participate in either phase—as well as control group producers—experienced the smallest change in fruits and vegetables produced and sold (Table B.4). Treatment group producers who received multiple donations and training sessions also experienced the largest average decrease in the amount of corn produced during the study period, on average (not shown). These analyses provide some evidence that receipt of PBS services was associated with an increase in horticulture production and a decrease in basic grain production. Due to potential selection effects inherent in these groups, these averages should not be interpreted as impact estimates. However, they suggest a correlation between the receipt of PBS assistance and increased horticulture production.

Table B.4. Change in Number of Fruits and Vegetables Produced and Sold from Baseline to Follow-

Up, by Subgroup (Averages in U.S. Dollars)

Measure

Change in Number of

Fruits/Vegetables

Cultivated from Baseline

to Follow-Up

Change in Number

of Fruits/Vegetables

Sold from Baseline

to Follow-Up Producers

Treatment Producers Who Received

at Least 15 Donated Items (Phase I) 0.7 0.5 40

Treatment Producers Who Received

at Least 5 Training Sessions or

Technical Assistance Visits (Phase I)

0.4 0.4 45

Treatment Producers who

Participated in Either Phase 0.3 0.2 96

All Treatment Producers 0.2 0.1 297

Treatment Producers who Did Not

Participate in Either Phase 0.1 0.0 201

All Control Producers -0.1 -0.2 296

Source: Baseline and follow-up Productive Development Survey-Horticulture (PDS-H) interviews

conducted from June 2010 to June 2011.

APPENDIX C

QUALITATIVE FINDINGS FOR THE DAIRY CHAIN

Appendix C Mathematica Policy Research

C.2

During July 2012, Mathematica staff conducted ten interviews with PBS participants from the impact evaluation study sample. Our goal was to interview beneficiaries who experienced a large increase in net income, beneficiaries who experienced a modest increase in net income, and beneficiaries who experienced no increase in net income during the study period (according to data from our impact analysis) in order to compare and contrast their experiences, activities, and outcomes. About a third of the sample experienced a large income increase, a third had a small but positive income increase, and a third experienced no increase in income (or actually a decrease in income in two cases).

Some key findings from these interviews are summarized below:

In-kind donations. Regarding in-kind donations, all interviewed participants acknowledged receiving multiple donations in Phase I, as well as a few donations in Phase II. Participants felt that donations were distributed unequally among participants in general, and more donations were distributed to participants who had higher production before PBS assistance. Interestingly, the two participants in our sample with the highest initial production received the largest donations, including irrigation systems and hay shredders. Other participants consistently mentioned these two items as potential donations that could have improved the overall impact of program assistance.

Technical assistance. Regarding technical assistance, all participants acknowledged receiving technical assistance during both phases. However, two participants reported that they stopped attending field schools during Phase II. Three interviewed participants reported that Phase I technical assistance was comprised of both theory and practice, but that Phase II assistance included only theoretical training. In general, these three participants praised technical assistance in Phase I (provided by TechnoServe) and expressed dissatisfaction with technical assistance in Phase II (provided by Zamorano). One participant stated that there was too much focus on demonstration plots in Phase II. This stood in contrast with Phase I assistance, during which field-staff visited all participants’ fields on a rotating basis. However, two-thirds of interviewed participants were satisfied with technical assistance in both phases.

Agricultural practices. Regarding agricultural practices, most interviewed participants mentioned that they had improved their supply of fodder through better pasture maintenance during the rainy season and better fodder storage during the dry season. Most participants received donated hayseeds in Phase I, in addition to technical assistance regarding fodder cultivation and storage. As a result of these donations and assistance, interviewed farmers reported large cost savings during the study period, as well as increased production linked to a more sustained supply of fodder. The other practice commonly mentioned in interviews was improved herd maintenance. Through training, participants learned how to diagnose common health problems, prevent mastitis, and properly inject cattle with antibiotics. Participants noted that these practices decreased their need for veterinarian services in recent months. Participants also mentioned that they improved cattle hygiene practices by applying knowledge learned through technical assistance.

Impacts of PBS assistance. Regarding the effects of PBS assistance on production and costs, most participants with much improved economic outcomes (according to survey data) stated that the combination of in-kind donations—particularly hayseeds, vitamins and parasite medication—and technical assistance helped them reduce their costs and increase production. Some participants who experienced moderate improvements

Appendix C Mathematica Policy Research

C.3

acknowledged a small increase in production, and a couple of producers (with no registered improvements) stated that they had lower milk production as a result of recent cattle sales. As a result of substantial donations and assistance, the two largest producers in our sample acknowledged large increases in production.

Collaboration with FOMILENIO-supported enterprises. None of the participants in the sample mentioned selling their production to Lácteos Zona Norte. One producer mentioned that the minimum quantity required by the enterprise was 2,000 bottles a day, and only two of the interviewed participants had that relatively high level of production.

Lessons learned. When asked about weakness of PBS assistance, most participants answered that more in-kind donations in Phase II were needed. A couple of participants mentioned that more hands-on practice was needed in Phase II. One participant mentioned that small loans at favorable conditions were very important, but were not included in PBS assistance.

APPENDIX D

ADDITIONAL BASELINE AND IMPACT TABLES

D.2

Table D.1. Producers’ Baseline Demographic Information, by Value Chain (Averages in U.S. Dollars Unless Otherwise Indicated)


Characteristic T C Diff

p-

value T C Diff p-value T C Diff

p-

value

Female (%) 15 18 -4 0.38 25 22 3 0.57 88 80 8 0.37

(35) (39) (43) (41) (32) (40)

Average Age (in years) 50 49 1 0.65 45 44 2 0.40 39 40 -1 0.56

(14) (14) (16) (15) (13) (14)

Married or Cohabitating (%) 86 86 0 0.88 78 78 0 0.98 68 67 1 0.93

(35) (35) (42) (42) (47) (47)

Education (%)

None 17 21 -4 0.30 26 28 -2 0.50 22 21 1 0.82

(37) (40) (44) (45) (42) (41)

Basic 66 67 -1 0.91 65 65 0 0.91 69 66 3 0.60

(47) (47) (48) (48) (46) (48)

More than basic 17 13 5 0.34 9 7 2 0.50 9 13 -4 0.39

(38) (33) (28) (25) (28) (34)

Member of a Productive Group (%) 10 16 -6 0.38 1 0 1 0.24 62 27 35 0.02

(31) (37) (10) (6) (49) (45)

Annual Investment and Input 8,417 5,457 2,961 0.03 930 908 22 0.88 638 506 133 0.74

Costs for Productive Activitiesa

(9,629) (9,264) (1,256) (1,569) (2,743) (1,332)

Net Annual Productive Incomeb

2,149 1,189 960 0.44 69 169 -100 0.74 414 389 25 0.89

(14,571) (11,452) (2,945) (2,812) (1,347) (1,017)

Annual Income from Salaries 531 480 51 0.82 468 490 -22 0.88 189 287 -98 0.34

(2,367) (2,475) (1,410) (1,674) (751) (901)

Net Annual Business Income 2,797 3,295 -498 0.77 295 278 16 0.93 121 62 59 0.45

(14,462) (22,209) (2,576) (1,637) (572) (1,115)

Annual Income from Remittances 3,105 532 2,572 0.07 755 744 11 0.98 280 227 54 0.30

and Other Sources (18,260) (9,191) (3,406) (5,825) (666) (633)

Total Net Annual Incomec

8,488 5,615 2,874 0.13 1,577 1,700 -123 0.87 1,007 964 43 0.88

(25,500) (24,866) (5,570) (7,018) (1,696) (1,784)

Average Number of Full-time 1.56 0.95 0.61 0.02 0.31 0.33 -0.01 0.83 0.02 0.03 -0.01 0.67

Productive Jobs Offered by

Producer Last Year

(1.99) (1.85) (0.61) (0.84) (0.19) (0.33)

Producers 260 258 297 296 303 322


Source: Baseline Productive Development Survey (PDS) interviews conducted from September 2009 to June 2010.

Note: Standard deviations are provided in parentheses. Income components do not sum perfectly to total net income, given a procedure by which total net income

was not calculated if more than one income component was missing.

a

This is the sum of respondents’ short- and medium-term productive investments and costs, including the cost of machinery, feed, materials, packaging,

transportation, and marketing.

b


c

This is the sum of respondents’ productive income, wages, business income, and additional income (including remittances), after subtracting agricultural and

business expenses.

D.3

Table D.2. Baseline Annual Household Income, Consumption, and Poverty, by Value Chain (Averages in U.S. Dollars Unless Otherwise Indicated)

Source: Baseline Productive Development Survey-Dairy (PDS-D) interviews conducted from May to June 2010.

Note: Standard deviations are provided in parentheses. Income components do not sum perfectly to total net income, given a procedure by which total

net income was not calculated if more than one income component was missing.

a

This is the sum of all household members’ productive income, wages, business income, and additional income (including remittances), after subtracting

agricultural and business expenses.

b


c

Extreme poverty is defined as gross household income of less than $0.93 per person per day. Relative poverty is defined as gross household income of less than

$1.86 per person per day (DIGESTYC 2009).


Measure T C Diff

p-

value T C Diff

p-

value T C Diff

p-

value

Household Size (people) 4 5 -1 0.01 5 5 0 0.82 5 5 0 0.84

(2) (2) (2) (2) (2) (2)

Net Productive Income 2,474 1,537 937 0.50

74 228 -154 0.63 486 470 16 0.88

(13,577) (9,989) (2,774) (3,016) (1,436) (1,095)

Income from Salaries 1,409 1,365 44 0.90 979 1,365 -386 0.24 1,703 2,147 -444 0.36

(3,545) (3,773) (1,880) (4,477) (4,885) (6,659)

Net Business Income 3,249 4,478 -1,229 0.50 803 350 453 0.24 413 157 256 0.38

(15,045) (24,475) (5,142) (1,767) (4,654) (1,468)

Income from Remittances 3,379 820 2,560 0.09 1,308 1,068 240 0.63 478 314 164 0.94

and Other Sources (18,691) (9,640) (4,501) (6,073) (1,280) (2,420)

Total Net Incomea

10,344 8,378 1,966 0.33 3,146 3,050 96 0.92 3,060 3,100 -41 0.94

(26,651) (27,030) (8,082) (9,072) (6,800) (7,220)

Consumptionb

8,316 6,724 1,592 0.09 3,490 2,981 509 0.12 2,494 2,149 345 0.12

(7,962) (6,058) (3,089) (2,305) (3,053) (2,182)

Percentage of Households in Extreme 9 18 -9 <0.01 51 55 -4 0.58 59 54 5 0.19

or Relative Povertyc

(29) (39) (50) (50) (49) (50)

Households 255 250

282 277

287 286


Appendix D Mathematica Policy Research

D.4

Table D.3. Additional Impacts of PBS on Dairy Producers’ Practices and Technology Adoption

(Percentages), 2011

Measure Treatment Control Difference p-value

Tried improved grass types or forage plants 54 51 3 0.53

Practiced soil conservation 45 42 3 0.58

Used urea or sugarcane for fodder 35 34 1 0.77

Used nutritional supplements 80 73 7 0.06

Took measures to avoid diseases and deficiencies 67 63 4 0.47

Tried new techniques related to herd health and

reproduction

23 17 6 0.07

Produced a herd registry or inventory 41 41 0 0.98

Tried producing new dairy products 4 7 -3 0.27

Tried making environment-friendly products 4 3 1 0.56

Tried new cooling and packing techniques 2 1 1 0.40

Conducted quality control activities 11 6 5 0.03

Conducted acidity tests with milk 13 4 9 <0.01

Completed a registry of dairy practices 3 3 0 0.98

Used a thermometer or density meter 7 2 5 0.02

Used best manufacturing practices 9 3 6 0.01

Took measures to cut costs 61 38 23 <0.01

Conducted an inventory of materials or products 13 9 4 0.23

Used a formal accounting system 4 2 2 0.07

Produced a business plan 2 4 -2 0.15

Used the internet and other technologies to buy

inputs

3 1 2 0.08

Used the internet and other technologies to sell

products

1 1 0 0.77

Shared experiences with other farmers through

exchanges and events

42 35 7 0.05

Participated in fairs or other sales events 7 3 4 0.10

Looked for new clients in the commercial sector 16 9 7 0.04

Producers 260 258




Note: Estimates reflect treatment-control differences at follow-up, after controlling for baseline

differences.


D.5

Table D.4. Impact of PBS on Dairy Producers’ Investment, Income, and Employment (Averages Unless




Dairy Investments and Input Cost a

6,368 5,208 1,160 0.16

(12,792) (9,914)

Net Dairy Incomeb

3,608 1,759 1,849 0.01

(12,524) (5,000)


(2,028) (1,574)

Net Business Income 1,967 1,239 728 0.29

(16,368) (3,337)


(2,060) (2,757)

Total Net Income 6,293 3,868 2,425 0.04

(20,234) (5,742)


Percentage of Producers Who Employ Workers Outside 36 23 13 0.01

the Workshop or Group (50) (42)

Number of Full-time Equivalent Jobs Offered by 0.58 0.40 0.18 0.16

Producer Last Year Related to Dairy Productionc

(1.19) (0.97)

Producers 260 258


Source: Baseline and follow-up Productive Development Survey-Dairy (PDS-D) interviews conducted from

May 2010 to July 2011.


differences. Standard deviations are provided in parentheses. Income components do not sum

perfectly to total net income, given a procedure by which total net income was not calculated if

more than one income component was missing.

a

This is the sum of respondents’ short- and medium-term dairy investments and costs, including the cost of

machinery, cattle feed, vitamins, labor, packaging, transportation, and marketing.

b

Net dairy income is total individual income from milk and dairy sales after subtracting the costs of all


c



D.6

Table D.5. Additional Impacts of PBS on Horticulture Producers’ Practices and Technology Adoption

(Percentages), 2011

Measure Treatment Control Difference p-value

Used improved seeds 82 59 23 <0.01

Used seed nurseries 34 32 2 0.87

Used staggered planting techniques 33 28 5 0.57

Practiced crop rotation 37 30 7 0.41

Used seed nurseries 15 11 4 0.48

Used irrigation technologies 29 26 3 0.76

Practiced soil management techniques 18 18 0 0.94

Took virus control measures 36 30 6 0.53

Practiced Good Agricultural Practices (GAPs) 14 14 0 0.98

Used soil conservation methods 22 24 -2 0.78

Used protected horticulture or tunnel

technologies 4 4 0 0.99

Planted new crops, including high-value fruits and

vegetables 6 4 2 0.49

Used eco-friendly practices 12 7 5 0.37

Completed a schedule of expenses and income 10 7 3 0.58

Conducted quality control activities 5 8 -3 0.43

Took measures to cut costs 37 26 11 0.24

Used a formal accounting system 2 1 1 0.66

Completed a cultivation or business plan 4 3 1 0.64

Used information to determine market demand or

explore commercial opportunities 4 1 3 0.22

Used the internet and other technologies to find

better input prices -1 1 -2 0.21

Shared experiences with other farmers through

exchanges and events 26 18 8 0.44

Formed strategic alliances with producers or

groups 9 3 6 0.08

Participated in fairs or other sales events 1 1 0 0.93

Looked for new clients in the commercial sector 13 8 5 0.33

Producers 297 296






D.7

Table D.6. Impact of PBS on Horticulture Producers’ Investment, Income, and Employment (Averages

Unless Otherwise Indicated)




Standard deviations are provided in parentheses. Income components do not sum perfectly to total net

income, given a procedure by which total net income was not calculated if more than one income

component was missing.

a

This is the sum of respondents’ short- and medium-term horticulture investments and costs, including seeds,


b

Net income from fruits/vegetables or other crops is gross individual income from sales after subtracting the

costs of all investments and inputs.

c



Annual Income (US$)a

Investments and Input Costs for All Cropsa

739 609 130 0.32

(1,219) (1,430)

Net Income from All Cropsb

-109 191 -300 0.09

(1,456) (2,286)

Investments and Input Costs for Fruits/Vegetables 212 84 128 0.02

(814) (284)

Net Income from Fruits/Vegetables 107 90 16 0.80

(751) (567)

Investments and Input Costs for Other Crops 522 525 -3 0.98

(871) (1,395)

Net Income from Other Crops -222 100 -322 0.04

(1,220) (2,188)


(1,267) (1,282)

Net Business Income 291 90 202 0.31

(963) (3,713)


(1,337) (1,104)

Total Net Income 1,508 1,614 -106 0.70

(2,142) (4,371)


Percentage of Producers Who Employed Workers 54 52 2 0.52

Outside the Workshop or Group for All Crops (50) (50)

Number of Full-time Equivalent Jobs Offered by 0.22 0.14 0.08 0.26

Producer Last Year Related to All Cropsc

(0.50) (0.71)


Outside the Workshop or Group for Fruits/Vegetables (31) (28)

Number of Full-time Jobs Offered by Producer Last Year 0.06 0.02 0.04 0.01

Related to Fruits/Vegetables (0.28) (0.10)


Outside the Workshop or Group for Other Crops (50) (50)

Number of Full-time Jobs Offered by Producer Last Year 0.15 0.12 0.04 0.60

Related to Other Crops (0.41) (0.70)

Producers 297 296



D.8

Table D.7. Impact of PBS on Horticulture Producers’ Production and Sales (Averages Unless Otherwise

Indicated)

Treatment Control Difference p-value

Percentage Growing

Fruits and Vegetables

Squash (Pipián) 13 11 2 0.50

Cucumber 24 10 14 <0.01

Tomato 15 9 6 0.08

Squash (Ayote) 5 3 2 0.15

Other Crops

Corn 74 77 -3 0.29

Beans 53 54 -1 0.86

Millet 33 39 -6 0.12

Rice 3 2 1 0.60

Sesame 1 3 -2 0.18

Hectares Under Production


Squash (Pipián) 0.06 0.05 0.01 0.61

Cucumber 0.05 0.01 0.04 <0.01

Tomato 0.03 0.01 0.02 0.09

Squash (Ayote) 0.02 0.01 0.01 0.13

Other Crops

Corn 0.77 0.92 -0.15 0.04

Beans 0.38 0.44 -0.06 0.23

Millet 0.34 0.44 -0.10 0.11

Rice 0.01 0.01 0.00 0.95

Sesame 0.00 0.02 -0.02 0.19

Metric Tons Produced


Squash (Pipián) 0.08 0.13 -0.05 0.47

Cucumber 0.20 0.08 0.12 0.02

Tomato 0.16 0.08 0.08 0.32

Squash (Ayote) 0.01 0.01 0.00 0.33

Other Crops

Corn 1.33 1.69 -0.36 0.05

Beans 0.13 0.21 -0.08 0.09

Millet 0.29 0.35 -0.06 0.52

Rice 0.02 0.03 -0.01 0.31

Sesame 0.00 0.01 -0.01 0.13

Tons Produced per Hectarea


Squash (Pipián) 3.24 3.08 0.16 0.91

Cucumber 11.64 9.57 2.07 0.39

Tomato 4.68 5.61 -0.93 0.80

Squash (Ayote) 1.13 1.86 -0.73 0.40

Other Crops

Corn 1.95 1.96 -0.01 0.99

Beans 0.39 0.55 -0.16 0.11

Millet 1.22 0.89 0.33 0.73

Rice 1.82 1.59 0.23 0.01

Sesame 0.59 0.41 0.18 0.19

Percentage Selling


Squash (Pipián) 9 10 -1 0.67

Cucumber 16 7 9 <0.01

Tomato 9 5 4 0.12

Squash (Ayote) 2 1 1 0.24

Other Crops

Corn 42 47 -5 0.24

Beans 16 19 -3 0.42

Millet 12 14 -2 0.41

Rice 1 1 0 0.59

Sesame 1 3 -2 0.18

Value of Sales (US$)


Squash (Pipián) 37 49 -12 0.65

Cucumber 24 13 11 0.14

Tomato 89 69 20 0.65

Squash (Ayote) 5 12 -7 0.33


Table D.7 (continued)

D.9

Treatment Control Difference p-value

Other Crops

Corn 136 283 -146 0.01

Beans 39 123 -84 <0.01

Millet 14 38 -24 0.01

Rice 4 5 -1 0.78

Sesame 4 6 -3 0.35

Producers 297 296


Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from June 2010 to June

2011.

Note: Estimates reflect treatment-control differences at follow-up, after controlling for baseline differences. Non-

percentage estimates are unconditional, meaning that farmers who did not produce specific crops were given

values of zero for hectares under production, tons produced and sales for those crops.

a

This measure is conditional, meaning that producers who did not report producing the crop were given blank values.

As such, sample sizes are smaller than the full sample of producers.

Table D.8. Impact of PBS on Handicraft Producers’ Investment, Income, and Employment (Averages

Unless Otherwise Indicated)

Source: Baseline and Follow-up Productive Development Survey-Handicrafts (PDS-H) interviews conducted

from October 2009 to November 2010.


differences. Standard deviations are provided in parentheses. Income components do not sum

perfectly to total net income, given a procedure by which total net income was not calculated if

more than one income component was missing.

a

This is the sum of respondents’ short- and medium-term handicraft investments and costs, including raw

materials, labor, transportation, packaging, and marketing expenses.

b

Net handicraft income is individual income from handicraft sales after subtracting the costs of all investments

and inputs.

c




Handicraft Investments and Input Costsa

474 423 51 0.65

(1,222) (1,052)

Net Handicraft Incomeb

679 654 24 0.92

(1,159) (2,515)

Income from Salaries 340 340 0 1.00

(978) (1,075)

Net Business Income 106 131 -25 0.28

(441) (603)


(738) (731)

Total Net Income 1,516 1,443 73 0.78

(1,753) (2,837)


Percentage of Artisans Who Employ Workers 12 6 6 0.11

Outside the Workshop or Group (30) (20)

Average Number of Full-time Equivalent Jobs 0.15 0.02 0.13 <0.01

Offered by Artisan Last Year Related to

Handicraftsc

(0.75) (0.21)

Producers 303 322



D.10

Table D.9. Impact of PBS on Female- versus Non-Female-Headed Households in the Handicraft Value

Chain (in U.S. Dollars)

Measure

Impact for

Female-Headed

Household

Impact for Male-

Headed Household Difference p-value

Net Handicraft Incomea

-21 -8 -13 0.97

Total Net Income 172 -319 491 0.45

Consumption -41 -133 92 0.41

Households 199 374

Source: Baseline and Follow-up Productive Development Survey (PDS) interviews conducted from

October 2009 to November 2010.


differences.

a

Net handicraft income is all household income from handicraft sales after subtracting the costs of all


Table D.10. Additional Descriptive Statistics on Handicrafts Producer Groups’ Production and Sales

(Averages Unless Otherwise Indicated)

Measure Treatment Control Difference

Number of People in the Group 10.0 11.7 -1.7

Percentage of Groups in Which Members Produce Jointly 70 100 -30

Percentage of Groups Producing:

Ropework 81 47 34

Metalwork 0 11 -11

Dyed Textiles 4 1 3

Mats and Bedrolls 7 14 -7

Jewelry 2 3 -1

Other 6 24 -18

Number of Items Produced per Month:

Ropework 56 18 38

Metalwork NA 53 NA

Dyed Textiles 29 43 -14

Mats and Bedrolls 10 18 -7

Jewelry 33 27 6

Percentage of Sales Destined for Export 25 85 -60

Group’s Annual Gross Income (in US$) 2,572 1,647 925

Groups 12 14


Source: Baseline and Follow-up Productive Development Survey (PDS) interviews conducted from

October 2009 to November 2010.

D.1

1

Table D.11. Impact of PBS on Annual Household Income, Consumption, and Poverty, by Value Chain (in U.S. Dollars Unless Otherwise Indicated)



Net Productive 4,034 2,018 2,017 0.02 -54 198 -252 0.21 646 657 -11 0.96

Income (13,394) (5,195) (1,552) (2,337) (1,085) (2,615)

Total Net 9,231 6,622 2,608 0.12 2,869 2,927 -58 0.87 2,464 2,615 -151 0.71

Incomea

(22,708) (9,041) (3,203) (5,129) (3,516) (3,745)

Consumptionb

6,062 4,841 1,221 0.13 2,297 2,246 48 0.82

1,383 1,489 -106 0.55

(8,000) (5,079) (1,905) (1,511) (1,258) (1,082)

Percentage of Households in 15 16 -1 0.73 49 55 -6 0.14 66 62 4 0.47

Extreme or Relative Povertyc

(28) (37) (50) (50) (48) (49)

Households 255 250

279 280

287 286


Source: Baseline and follow-up Productive Development Survey (PDS) interviews conducted from September 2009 to July 2011.

Note: Estimates reflect treatment-control differences at follow-up, after controlling for baseline differences. Standard deviations are provided in parentheses.

a

This is the sum of all household members’ productive income, wages, business income, and additional income (including remittances), after subtracting agricultural

and business expenses.

b


c

Extreme poverty is defined as gross household income of less than $0.93 per person per day. Relative poverty is defined as gross household income of less than $1.86

per person per day (DIGESTYC 2009).

www.mathematica-mpr.com

Documents

Interim Report Final Draft: August 22, 2012search.oecd.org/countries/elsalvador/report-102012-evaluation-slv... · Interim Report Final Draft: August 22, 2012 Randall Blair Larissa