Annex 3: Inception Report

Ministry of Foreign Affairs (Danida)

Impact Evaluation of Aquaculture Interventions in Bangladesh

FINAL INCEPTION REPORT

Prepared jointly by:

Orbicon A/S Ringstedvej 20

DK-4000 Roskilde Denmark

LAMANS s.a. Management Services Management Consultants

56, 3rd Septemvriou str 104 33 Athens, Greece

Table of Contents 1.0 Introduction ..........................................................................................1

1.1 Evaluation Objectives...........................................................................1 1.2 Inception Period...................................................................................1

2.0 Preliminary Intervention Profile...........................................................2

2.1 Evolution of Interventions....................................................................3 2.1 Planned Intervention Results.................................................................4 2.2 Site and Target Group Selection ............................................................5 2.3 Programming Modalities .......................................................................6

3.0 Evaluation Approach, Issues, and Methodology.................................6

3.1 Phased Approach .................................................................................7 3.2 Theoretical Background ........................................................................7 3.3 Evaluation Issues..................................................................................7 3.4 Key Challenges.....................................................................................9 3.5 Methodology ...................................................................................... 12 3.6 Analysis.............................................................................................. 23

4.0 Operational Plan................................................................................. 25

4.1 Components....................................................................................... 25 4.2 Schedule............................................................................................. 26

List of Annexes Annex 1 - List of Persons Interviewed and Sites Visited – Inception Mission Annex 2 – Preliminary Intervention Profile Annex 3 – List of Evaluation Issues, Sub-Questions, and Indicators Annex 4 – List of Evaluation Issues Cross-Referenced with Collection Methods Annex 5 - Interview Guides Annex 6 – Focus Group Facilitation Guides Annex 7 – List of Available Data

List of Acronyms ADB Asian Development Bank ADC Additional Deputy Commissioner ADP Annual Development Plan AES Aquaculture Experiment Station ASPS Agricultural Sector Programme Support AU Administration Unit BBS Bangladesh Bureau of Statistics BFT Bistandsfaglig Tjeneste (Danida Department providing technical services

for development assistance) CBFM Community Based Fisheries Management CBO Community Based Organization CPA Chief Project Adviser CPM Central Project Management DAE Department of Agriculture Extension Danida Danish International Development Assistance DC Deputy Commissioner DD Difference in Difference (also known as Double Difference) DFID Department for International Development (United Kingdom) DG Director General DKK Danish Kroner DoF Department of Fisheries DTA Danida Technical Assistance ET Extension Trainers ETA Extension and Training Advisor FARS Freshwater Aquaculture Research Station FPSU Fisheries Programme Support Unit FFG Fish Farming Group FHTC Fish Hatchery and Training Centre (DoF, Raipur) FRI Fisheries Research Institute GNAEC Greater Noakhali Aquaculture Extension Project / Component GoB Government of Bangladesh Ha Hectare HH Household IFAD International Fund for Agricultural Development IFPRI International Food Policy Research Institute IGF Integrated Gher Farming IPF Integrated Pond Farming LFA Logical Framework Approach LGF Loan Guarantee Fund LMG Lake Management Groups MACH Management of Aquatic Ecosystem through Community Husbandry

MAEC Mymensingh Aquaculture Extension Project / Component MEU Monitoring and Evaluation Unit MoFL Ministry of Fisheries and Livestock MoL Ministry of Land MoU Memorandum of Understanding NGO Non Governmental Organization NRDS Noakhali Rural Development Society PBAEC Patuakhali and Barguna Aquaculture Extension Project / Component PCU Programme Coordination Unit PD Project Director PFG Pond Farming Group PMC Project Management Coordination (committee) PP Project Proforma PPSU Policy Planning Support Unit PRA Participatory Rapid Appraisal/Assessment PSC Project Steering Committee PSM Propensity Score Matching RDE Royal Danish Embassy RDF Rakhaing Development Foundation RFLDC Regional Fisheries and Livestock Development Component SDO Social Development Organization SPS Sector Programme Support SSDP Southern Socio-economic Development Programme TAC Technology Advisory Coordination TDT Technology Development and Transfer TDU Technology Development Unit TEU Training and Extension Unit Tk Taka (Bangladeshi unit of currency) ToT Trainer of Trainers ToR Terms of Reference TTU Technology Transfer Unit TWDC Thana Women’s Development Coordinator USD United States of America Dollars WPC Women’s Program Coordinator

Glossary Aman Crop during monsoon period Angula Fish fingerlings Aratdar Wholesaler Bazar Samity Market Organization Beparie Medium Trader Char Emergent land from the river sediment deposit/accretion

process areas Doba Ditch Faria Small trader Gher Enclosure for prawn cultivation by modifying rice fields by

building higher dikes around the field and excavating a canal several feet deep inside the periphery of the dikes to retain water during the dry season

Hali Unit of four Jalmohol Public water body that could be leased out by government for

revenue (also spelt Jolmohol) Kutcha Earth/Straw Maund Unit of weight 37.5 kg Mouza Sub-union administrative unit / geographical area Nola / Pona Fish fry Paiker Trader Pucca Stone/Concrete Thana Sub-district administrative unit now known as Upazila Touts and Batpars Middleman Union Sub-Upazila administrative unit Upazila Sub-district administrative unit, formerly known as Thana Impact Assessment Terminology Causality, the intervention logic captured in linkages in the results chain from activities through to outputs, outcomes, and impacts. The causal linkages in the chain together with sound logic underpin and supplement the attribution analysis. Examination of causality also answers questions of outputs did or did not lead to impacts.

Attribution, an assessment of the degree to which impacts can be linked back to the outputs delivered by, and “credited” to, the interventions. At the impact level, attribution is generally accepted to be at the level of “some contribution of outcomes in combination with many other important factors”.

Counterfactual, an assessment of what would have been the status of impacts if the intervention had not taken place.

The treatment group, households (HHs) or individuals who have been affected by the interventions.

The comparison group, HHs/individuals who have NOT been participating themselves, but who possess similar observable characteristics as those in the treatment group.

Propensity Score Matching, a technique to select members of the comparison group through estimation of a statistical model based on the probability of participation.

Pipeline Approach, a technique for comparison group selection where the comparison group will be composed of individuals who have been selected (eligible) to participate, but have not (yet) been involved nor have benefited from intervention activities. This assumes that such a pipeline exists, that there has been no change in selection criteria, and that applicants have not been ranked for participation.

Double Difference Measurement, the double difference measures the difference in the observed change between the treatment and comparison group, based on baseline and end-data. Thus the double difference eliminates external determinants of the outcome, in cases where these are the same for the treatment and comparison group during the intervention period. The double difference approach assumes common time effects across groups and no composition changes within each group.

Contamination, can arise from two sources – “own” and “external”. Contamination can come from the intervention itself due to spill over effects (own contamination). This type of contamination can occur if the comparison group is selected from a geographical area too close to the intervention area. Comparison groups from distant locations can also be contaminated through interventions by other agencies (external contamination).

Selection bias, is bias introduced from the way beneficiaries have been selected. When beneficiaries are not randomly selected, but some kind of selection process has taken place, then the comparison group should not be randomly selected either, but rather be drawn from a population with the same characteristics as the group of participants.

1

1.0 Introduction In December 2006, a consortium made up of Orbicon A/S (Denmark) and Lamans s.a. Management Services (Greece), together with a team of Bangladeshi subcontractors, was contracted by Danida to undertake an impact evaluation of the three main Danida funded aquaculture extension interventions in Bangladesh. The time period to be covered by the evaluation is 1989 – 2006. The three components1 are the Mymensingh Aquaculture Extension Project / Component (MAEC); The Greater Noakhali Aquaculture Extension Project / Component (GNAEC); and The Patuakhali and Barguna Aquaculture Extension Project / Component (PBAEC).

1.1 Evaluation Objectives The purpose of the Evaluation is to measure the impact of the Danida funded aquaculture extension interventions on the socio-economic position and physical well being of people living within the intervention areas. The central question to be answered by the Evaluation is “Has there been a change in people’s income, living standards and women’s empowerment amongst beneficiaries of the three components – and crucially, can any such changes be attributed to the interventions?” More specifically, the Evaluation will address the following questions: • What has been the impact on incomes? • What has been the impact on living standards in general? • What has been the impact on health? • What has been the impact on women’s empowerment? The Evaluation will also identify channels through which these impacts have occurred, and attempt to establish the determinants of change along with reasons for smaller than expected or unintended impacts. In terms of a forward-looking perspective, the Evaluation will draw lessons learned and identify key operational experiences that may be used for future interventions.

1.2 Inception Period In order to determine how to best address these questions, a series of inception activities were undertaken, culminating in this Inception Report. The activities included: • Extensive study and review of relevant and available documents and data;

1 The interventions were originally ‘projects’ and are still referred to as such by the Government of Bangladesh. Officially however, they became ‘components’ of the Agricultural Sector Programme Support in 2000.

2

• Assessment of the availability and quality of quantitative data sets for impact measurement;

• Identification and cataloguing of relevant documentation, made accessible to all team members via a password protected website;

• Detailed development of evaluation methodology and design; • Mapping of the causal chain of the interventions; • Preliminary identification of other relevant activities, which might have affected

the outcomes and impact of the Danida funded interventions; • Further development of Evaluation Questions derived from the Terms of

Reference (ToR); • Specification of indicators and corresponding components of data collection

instruments; • Development of draft guides for interviews and focus groups; • Two Inception Missions to Bangladesh by the Team Leader and one by the

Deputy Team Leader; • Meetings and consultation with various key stakeholders and other development

actors; • A team meeting in Copenhagen, linked to a team meeting in Dhaka, in order to

review available data, develop options for quantitative data collection and analysis, and draft Inception Report sections; and,

• Drafting of Inception Report. A full list of places visited and persons interviewed can be found in Annex 1.

2.0 Preliminary Intervention Profile In order to design an appropriate approach and methodology for the impact evaluation, an in-depth understanding of the Danida-funded aquaculture intervention is required. The following preliminary intervention profile provides a summary overview of the intervention elements and characteristics. This is supplemented by the preliminary intervention profile information in Annex 2. It should be noted that a profile of this type is not meant to be complete at the Inception Phase. It is intended to capture the essential elements of the intervention and is primarily based upon available documentation, and supplemented with a few interviews. At the Inception Phase the profile details the planned achievements of the intervention, together with the planned programming modalities. Details and data on actual achievements and actual programming modalities will be supplemented, reinforced, or changed in the main evaluation report, based upon data collected during the main evaluation phase of fieldwork. As such, there may be errors or inconsistencies in the preliminary version of the profile. These will be investigated by the Evaluation and addressed in the finalized intervention profile, which will be annexed to the main evaluation report.

3

2.1 Evolution of Interventions Danida has been supporting aquaculture extension in Bangladesh since 1976 when it started the Aquaculture Experiment Station (AES) in Mymensingh, later known as the Freshwater Aquaculture Research Station (FARS). This research was piloted through the implementation of the MAEC, which began in 1989 with the aim of increasing the production of fish protein and in order to raise the socio-economic status of residents in selected Upazilas. MAEC consisted of three phases of varied duration. Following the initial success of MAEC, similar interventions were designed for the Patuakhali and Barguna and Greater Noakhali districts, starting in 1997-98. As an additional feature, these interventions included a central prawn cultivation component and an expansion of activities to non-pond culture including paddy field rice-fish culture, juvenile prawn nursing, and cage culture. In addition, PBAEC included a component for community based fishery management (CBFM) in closed and semi-closed public water bodies (Jalmohols), thus expanding the target group to groups that did not own ponds, including landless HHs and poor fishermen. During the Inception Period it was decided that the CBFM component would be excluded from the Evaluation in order to focus strictly on aquaculture interventions. Table 2.1 summarizes key features of the three Danida funded aquaculture extension interventions. (Note: there are discrepancies in the documents regarding the start dates and actual disbursements for PBAEC. This table uses the dates from the draft PBAEC Component Completion Report, August 2007).

Table 2.1 Key Data of MAEC, PBAEC, and GNAEC

MAEC PBAEC GNAEC Phase I Phase II Consolidation

Phase Aquaculture

Component / Fisheries Component

Duration (Yrs.) 4 7 3.5 9 9

Start - Closing Date

July 1989 -June 1993

July 1993 – June 2000

July 2000 - Dec 2003

July 1998 - Sept 2006

July 1998 – September

2006

# of Districts 11 72 72 23 34 # of Upazilas 6 32 43 12 15 Beneficiaries (# of individuals / HHs

3,850 (individuals)

53,850 (individuals)

49,900 (individuals)

63,382 (HHs) 35,748 (HHs)

Total Danida disbursement (in 000s /DKK)

9,542 45,516 13,293 40,745 42,031

1) Mymensingh 2) Mymensingh, Kishoreganj, Tangail, Netrokona, Jamalpur, Sherpur, and Gazipur 3) Patuakhali and Barguna 4) Noakhali, Lakshimpur and Feni

4

2.1 Planned Intervention Results Given that Danida funded aquaculture extension interventions in Bangladesh have been implemented for nearly two decades, in multiple phases, and have covered 12 districts, benefiting more than 200,000 HHs and individuals, the objectives, scope, and focus of the interventions have evolved and changed with time and geography. Consequently, there is no documented “single” model or statement of objectives that synthesizes results across the interventions. In order to understand more fully the intent of these aquaculture extension interventions for the period 1989 to 2006, the Evaluation examined all existing results matrices for each phase. From these statements of expected results, it was possible to synthesize an “overall” logic model that captures the intent and scope of the Danida funded aquaculture extension interventions for the evaluation period. The logic model is presented in Figure 2.2. (The logic model does not include the CBFM component)

Figure 2.2: Danida Funded Aquaculture Interventions in Bangladesh - Logic Model

5

2.2 Site and Target Group Selection Target groups and entry points for the three interventions are presented in the table below.

Table 2.2: Entry Points and Target Groups

MAEC PBAEC GNAEC Entry Point

Ponds (size of 0.005 hectares [ha] to 2\0 ha) in Mymensingh District

Ponds and semi-closed public water bodies (jalmohals) in the districts of Patuakhali and Barguna

Ponds in the districts of Noakhali, Feni and Lakshmipur

Primary Target group/ Beneficiaries

- Pond owning HHs - Poor men and women with access to ponds (emphasis on landless and marginal farmers) - Extension trainers - Landless men and women who were

- All HHs with access to a fishpond (Special attention to landless people and marginal farmers, men and women, with access to fish cultivation in owned or leased ponds, 50% of the HHs2)

- Poor men and women with access to ponds - Poor rural pond operators

2 Definitions: poor = HHs with annual income less than Taka (Tk) 35,000, landless = less than 2000m2 and marginal = less than 8000m2

Activities

Outputs

Immediate

Outcomes

Intermediate

Outcomes

Impacts

Improved socio-economic position

(income, employment) and physical

well-being (health, living standards)

of poor men and women

Increased

aquaculture

production across

three project

areas

Increased

aquaculture

sector economic

activity* and

employment*

Increase in

semi-

intensive

polyculture

Enhanced support

network for aquaculture

(harvesting,

processing, net

making, etc)

Increased

hatchery

operations

(new and

existing)

Enhanced

aquaculture

extension

Fish

farming

groups

formed and

operational

Fish

farmers

trained in

advanced

techniques

Trained

harvesters,

net

makers,

etc*

Secondary

aquaculture

sector persons*

accessing credit

Fish

farmers

accessing

credit

Increased

commitment

to hatchery

construction

Trained

DoF**

staff* and

improved

systems

Trained

extension

workers*

Group

Formation

Training in

production

methods

Provision

of credit

Training of

aquaculture

support

network

Promotion of

hatchery

construction

Strengthening

DoF staff*

and

administration

Training in

aquaculture

extension

* Includes women and men

** Department of Fisheries

Activities

Outputs

Immediate

Outcomes

Intermediate

Outcomes

Impacts

Improved socio-economic position

(income, employment) and physical

well-being (health, living standards)

of poor men and women

Increased

aquaculture

production across

three project

areas

Increased

aquaculture

sector economic

activity* and

employment*

Increase in

semi-

intensive

polyculture

Enhanced support

network for aquaculture

(harvesting,

processing, net

making, etc)

Increased

hatchery

operations

(new and

existing)

Enhanced

aquaculture

extension

Fish

farming

groups

formed and

operational

Fish

farmers

trained in

advanced

techniques

Trained

harvesters,

net

makers,

etc*

Secondary

aquaculture

sector persons*

accessing credit

Fish

farmers

accessing

credit

Increased

commitment

to hatchery

construction

Trained

DoF**

staff* and

improved

systems

Trained

extension

workers*

Group

Formation

Training in

production

methods

Provision

of credit

Training of

aquaculture

support

network

Promotion of

hatchery

construction

Strengthening

DoF staff*

and

administration

Training in

aquaculture

extension

* Includes women and men

** Department of Fisheries

6

MAEC PBAEC GNAEC assisted to become service providers (fish traders, transporters, net makers etc.)

Secondary Target Group/ Beneficiaries

- Field workers (locally employed) - DoF and Non-Governmental Organization (NGO) staff

- Male and female participants in the aquaculture production chain: Nursery operators, hatchery operators, fry traders, vendors, feed producers, harvesting groups, pump operators, block supervisors, entrepreneurs - DoF staff - NGOs - Bank staff - Extension trainers

- Part-time pond operators - Nursery operators - Fry traders - Pond harvesting teams - Fish transporters - Fish traders/ market operators - Entrepreneurs - DOF staff - NGOs / Community Based Organizations (CBOs) - Extension trainers

2.3 Programming Modalities The programming modalities used by the MAEC, PBAEC and GNAEC interventions can be summarized as follows:

• Group formation among pond operators; • Development of sustainable integrated aquaculture practices and production

through training, technical advice, and awareness raising; • Development of a fisheries management model; • Preparation of training and information material; • Establishment of credit delivery services and disbursement of credit; • Women’s empowerment / Gender equality programming including gender

training, awareness raising, etc.; • Impact and operations monitoring; • Research, baseline studies, socio-economic studies, and resource surveys; • Promotion of hatchery construction and some construction of hatcheries; • Assisting target groups to get access to closed jalmohals; and, • Strengthening of DOF administration.

3.0 Evaluation Approach, Issues, and

7

Methodology

3.1 Phased Approach Based upon the evaluation objectives and the scope of work described in the ToR, the consortium has designed a three-phase approach to the assignment. The steps include: • An Inception Phase (Phase I); • A Fieldwork Phase (Phase II); and, • Analysis and Reporting Phase (Phase III). This Inception Report includes the findings from Phase I, the Inception Phase. A detailed planning and timing for Phase II and Phase III can be found in Section 4 of this report.

3.2 Theoretical Background As requested in the ToR, this evaluation is designed using a theory-based approach to impact evaluations. This approach, which has been adopted and applied in recent impact evaluations, represents a particular attempt to combine rigorous analysis with an analysis of the context of intervention. The aim of using a theory-based approach for this evaluation is not only to provide answers to whether the Danida funded aquaculture interventions have achieved expected outcomes and impact, but also to assess how and why outcomes and impact have been achieved or not. A list of definitions and technical terms used in this methodology section can be found in the Glossary.

3.3 Evaluation Issues The ToR that the consortium responded to contained evaluation questions on 10 different topics that the Evaluation was to see as “examples of issues to be explored”. The 10 topics were to be used in the formation of the specific evaluation themes and questions that the evaluation is to address. The Evaluation reviewed these in light of the stated objectives of the evaluation, the Scope of Work provided in the ToR, and the overall approach as stated in the section above. The focus was reduced to a set of main evaluation issues set out in Table 3.1. For each main evaluation issue, a series of sub-questions were developed.

Table 3.1: List of Evaluation Issues and Sub-Questions

8

Main Evaluation Issues and Questions

Sub-Questions

1.0 Impacts For primary (credit/non) vs. secondary target groups vs. non-participants (within intervention area and outside)

1.1 What has been the impact on incomes and employment?

a) What have been the changes in income? b) What kinds of employment opportunities are available in the sector and spin-offs? c) How have income sources changed? d) Who benefits from the interventions? e) What have been the determinants of any changes?

1.2 What has been the impact on living standards?

a) Have there been any significant changes in material possessions, including housing? b) Have there been any significant changes in consumption? c) What have been the main determinants of any changes?

1.3 What has been the impact on health?

a) What have been the changes in nutritional status? b) In general health? c) Has the intervention contributed to any adverse health effects? (Exposure to chemicals, increase in communicable diseases etc.) d) What have been the main determinants of any changes?

1.4 How have interventions contributed to women’s empowerment?

a) What have been the changes to women’s knowledge and skills? (Compared to men?) b) How has decision-making within HHs changed? c) How has decision-making changed in relation to aquaculture production and marketing? d) How has control over credit changed? e) How has women’s mobility changed? f) What have been the main determinants of any changes?

1.5 How have interventions affected non-intervention geographical areas?

a) Is there evidence of positive or negative impact on the non-intervention Upazilas and district levels?

1.6 Do aquaculture related activities have the potential to continue after donor funding is completed?

a) How likely are the above impacts to be sustainable? (Noakhali and Patuakhali) b) What is the extent of sustainability following termination of Danida support in Mymensingh?

1.7 Was there a difference in the impact of the different species and technologies (or combinations thereof) promoted by the intervention?

a) What was the impact of combined freshwater prawn and fish cultivation and pure table fish cultivation, or other combinations of promoted technologies / species? Ponds versus rice fields? b) If there are differences to what are these attributable? (nature of intervention, characteristics of beneficiaries)

1.8 Intervention areas and Phases [will be dependent on data, may not be included]

a) Was there any noticeable difference in impacts between intervention areas and across phases? b) What was the likely cause/explanation of these different impacts?

2.0 Intervention Design and Delivery of Aquaculture management and technology

2.1 Training Delivery a) What were the strengths and weaknesses of training (differentiating between types of training)?

2.2 Aquaculture Management and Technology

c) Were the farming methodologies and protocols appropriate for the context and were they followed (especially for carp polyculture). What were the effects of not following protocols? d) Were the hatcheries and supplies of fry components well managed?

2.3 Women’s participation a) Was the intervention successful in accommodating and encouraging women’s participation as staff? b) As beneficiaries?

3.0 Corruption 3.1 What was the effect of any corruption on the intervention?

a) Has there been any corruption and what effect has this had on impact? b) Were guidelines on participation eligibility followed?

9

The Sub-Questions have been further expanded into specific indicators that will be used for measurement, as per Annex 3.

3.4 Key Challenges

3.4.1 Baseline and Ex-Post Data Baseline Data Baseline surveys were carried out within all three intervention areas. However, the data assessment carried out by the Evaluation during the inception period revealed certain challenges on baseline data availability. In relation to Mymensingh, the Evaluation has not been able to locate any baseline data files, despite many efforts and involvement from current and former DoF staff and Danida advisors. It appears that the baseline databases and related household lists (with names and addresses of HHs included in the study) were stored on a removable hard drive that was handed over to DoF upon completion of the interventions. The Royal Danish Embassy (RDE) in Dhaka never received a copy of the files on this hard drive. Unfortunately, it has not been possible to locate the hard drive within DoF. Instead, the Evaluation has obtained access to a data set from a study carried out by the International Food Policy Research Institute (IFPRI) in Mymensingh during 1996/1997. This data set includes relevant socio-economic and gender indicators, as well as a distinction between MAEC participants and non-participants. Since the IFPRI data set includes information on HHs involved in the Danida funded aquaculture extension interventions, as well as non-participating HHs that fulfil criteria for participation (pipeline), the data could serve as a baseline for participants and non-participants (comparison group). In both Noakhali and Patuakhali, a Baseline Pond Survey was carried out prior to implementation of the interventions. In Noakhali the Baseline Pond Survey was carried out in two halves, one in year 2000 for 6 Upazilas and one in 2002 for the remaining 8 Upazilas. The surveys include information on every pond owner within the two regions. The survey questionnaires used for the interventions were not identical however. In particular, the questionnaire used in Patuakhali included fewer socio-economic indicators than the one used in Noakhali. The Patuakhali baseline survey therefore constitutes a particular challenge since socio-economic indicators that are comparable to MAEC and GNAEC are unavailable, especially for non-participants. The target group of people with access to fish cultivation in leased ponds is not covered by the Baseline Pond Surveys in Noakhali or Patuakhali. It will therefore be

10

necessary to examine other surveys carried out by GNAEC and PBAEC (such as surveys that were carried out when fish farmers groups were formed and training activities were initiated). These may enable the establishment of a baseline for this target group. Ex-post Data With regards to Mymensingh, the Evaluation has tried to gain access to the list of HH names and addresses used by the 1996/97 IFPRI survey in order to carry out a re-survey by re-questioning the same HHs. A Dhaka based company, DATA, is apparently in possession of the list of HH names and addresses. The Evaluation was in contact with, and visited, the DATA office several times during the inception period in order to obtain access to the household lists. However, DATA refused to release the lists to the Evaluation without official permission from IFPRI, due to confidentiality agreements signed with the HHs. The Evaluation repeatedly contacted IFPRI about this situation and, in August 2007, a decision was made by IFRPI that the lists with HH names and addresses would not be provided to the Evaluation. Instead, the Evaluation may be assisted by a recent resurvey carried out by IFPRI (2007) of the same HHs from the 1996/97 survey. As this survey has just been completed, it is not fully clear how useful the data will be to this impact evaluation. IFPRI and the local survey firm (again DATA) have made verbal assurances that electronic versions of the data and HH codes will be provided to the Evaluation once it is cleaned and processed. If, and when, the data set becomes available it will be analysed for its utility and possible inclusion. Another relevant secondary ex-post data set for MAEC was collected by the company Winrock International when carrying out the “MAEC Impact Evaluation Study” (2004). However, despite repeated requests, the Evaluation has not been provided access to the database and the related list with names and addresses of surveyed HHs from this MAEC evaluation study. Concerning Noakhali and Patuakhali, a “Baseline Study for Agricultural Sector Programme Support Phase II, Regional Fisheries and Livestock Development Component, Technical Support Units” is currently being finalized in relation to initiation of the new Agricultural Sector Programme Support (ASPS) in Noakhali and Patuakhali3. According to the time schedule for this study, the final report should be due in late August 2007. In view of the above mentioned challenges on data collection and availability, specific proposals for the quantitative data analysis are outlined in section 3.5.6. At this point it is assumed that the quantitative data analysis will be carried out entirely by use of

3 The field survey team finished data collection by end of June 2007 as scheduled.

11

secondary data sources. Although the Evaluation is not yet in possession of the 2007 survey data sets referred to above, our preliminary assessment of the methodologies, draft questionnaires, and HH selection criteria applied for these surveys, give us with a high degree of confidence that the data sets can be used as ex-post data for our analysis. However, should this not be the case, we will need to review the proposed methodology for the quantitative data analysis for the particular region(s) where issues with baseline and ex-post data sets are identified.

3.4.2 Comparison Group in Intervention Areas The geographical coverage of Danida funded aquaculture interventions has been widespread, in particular in Noakhali and Patuakhali where it is difficult to find Unions that have not been covered by these interventions. This creates a challenge for the Evaluation in the establishment of the counterfactual through the use of non-participating HHs.

3.4.3 Length of Elapsed Time from Aquaculture Interventions The fact that the last intervention phase in Mymensingh was completed four years ago in 2003 creates a particular challenge for the Evaluation when analyzing impacts of the intervention period. People’s recollections of MAEC events and impacts will have faded and other interventions will have taken place since MAEC ended. (See section 3.4.5 Other Programming in Intervention Areas)

3.4.4 Change in Programming across Phases Evaluation issue 1.7 in table 3.1 requires some measurement of the different impacts of the individual phases of MAEC. This will be very difficult to do due to the change in focus and emphasis, and the lack of comparable data across the three phases. For a limited set of indicators it might be possible to track differences in phases between Upazilas if programming did not span multiple phases. However, this analysis would not be statistically significant. The ability to do the analysis will depend to a great extent on the availability of data, which is most problematic in Mymensingh. This analysis, therefore, will only be undertaken if the data is available.

3.4.5 Other Programming in Intervention Areas Between 1985 and 2005 the 10 major donors active in Bangladesh in the aquaculture sector spent or committed 317 million USD in grants or loans. These donors included Danida, World Bank, Asian Development Bank (ADB), the International Fund for Agricultural Development (IFAD), and the Department for International Development (DFID), with the World Bank being the single largest donor. Danida was the fourth largest donor for grants4. A particular challenge will be to distinguish

4 Study Report prepared by The World Fish Center on “Donor Funding and NGO Activity in Aquaculture in Bangladesh”

12

specific impact from Danida-funded aquaculture interventions from the impact generated by other development and credit programmes, which were implemented by the Government of Bangladesh (GoB), other donors, and/or NGOs, around the same time and within the same intervention areas. The above-mentioned external contamination from other activities implemented after 2003 in Mymensingh will add to the complexity of what impact should be attributed to MAEC intervention from 1989 to 2003.

3.5 Methodology The following are the proposed methods to be applied by the Evaluation to collect the data and information required for this impact evaluation: • Document Review; • Key Informant Interviews; • Stakeholder Workshops; • Focus Groups; • Individual Case Studies; • Quantitative Data Collation and Analysis; and, • Site Observations. These methods are elaborated in the following sections. A table cross-referencing these methods with the evaluation issues, sub-questions, and indicators can be found in Annex 4.

3.5.1 Document Review The Evaluation has reviewed over 70 relevant documents and 20 databases in order to complete the Inception Report. This document review will be continued throughout the field study period as new documents and databases become available. This ongoing review will likely include extracting relevant data and information at the national, regional, district, upazila, union, and individual levels.

3.5.2 Structured Key Informant Interviews Key informant interviews will be conducted to obtain qualitative findings on most of the evaluation issues. These interviews will take the form of extended one-to-one exchanges with individuals who are knowledgeable about aquaculture interventions. The interviews will provide in-depth information that will allow the Evaluation to assess various aspects of the interventions. Key informant interviews will also be essential for identifying and understanding unforeseen obstacles and lessons learned

13

that are usually difficult to measure using quantitative data and methodologies. It is expected that around 40 key informant interviews will be carried out. The core team members of the Evaluation will facilitate the key informant interviews with a particular view to gender sensitivity. The first interviews will serve as tests. After these, the interview guides will be adjusted, if necessary. The key stakeholder groups the Evaluation expects to interview, and the estimated number of interviews to be carried out with representatives from each of these groups, are summarised in Table 3.2.

Table 3.2: List of Groups to be Interviewed

Stakeholder Group Number National / Intervention Level 13-15 Ministry of Fisheries and Livestock 2 Department of Fisheries 2-3 Danish Embassy in Dhaka 2 Danida Bistandsfaglig Tjeneste (BFT) staff in Copenhagen

2-3

Aquaculture-Related Institutions 2 Donors / NGOs 3 District Level 18-21 Department of Fisheries 3 Current and Ex-staff – Local and International

3-6

Participating NGOs / Banks - Private Sector Actors in Aquaculture (Hatcheries, etc)

6

Other Development Actors 6 Local Level 6 Local Government Representatives 6

Interview guides for the stakeholder groups can be found in Annex 5.

3.5.3 Stakeholder Workshops There has been a suggestion of carrying a few additional national and regional workshops with key stakeholders, in addition to the national Final Validation Workshop already included in the proposal. The additional workshops will primarily be used for presentation and discussion of the evaluation, not for data collection. However, they are included in this section as they represent an addition to the

14

implementation plan. The following additional workshops are being planned: • National Stakeholder Workshop in Dhaka: The national workshop will include

presentations and discussions based on the Inception Report. We believe that the kind of methodology and approach devised for this impact evaluation may provide new, valuable learning and experience for the GoB and DoF. In this sense, an initial national stakeholder workshop could be a good opportunity to ensure a proper common understanding of the evaluation work to be carried out. In addition, it would facilitate the provision of relevant knowledge and experience to the Evaluation before planning and implementation of fieldwork within the three regions; and,

• Regional Stakeholder Workshops (one in each of the three Regions covered by the interventions). The Regional workshops will be held prior to initiation of fieldwork in each specific region. The workshops will provide local stakeholders with a clearer understanding of the purpose of the evaluation activities to be carried out within their local area. They will also provide valuable input to the Evaluation for detailed planning and implementation of the fieldwork activities within each specific region.

The proposed time schedule for workshop implementation is presented in Section 4.2.

3.5.4 Focus Groups

The focus group method is especially well adapted to collect qualitative information from a sample of stakeholders, such as small, intentionally diverse or homogenous groups, who will meet to discuss a number of specific topics with a group moderator. Focus groups are very useful in providing context and depth to complex evaluation issues, and their exploratory nature enables evaluators and reviewers to pursue interpretive questions on the "how and why" of an issue. Focus groups can also provide a time efficient and more representative means of drawing out experiences from a particular group when compared to key informant interviews. It is estimated that 45 focus group sessions of 1-2 hours with key stakeholder groups will be organized. (See Table 3.3) The high number of focus groups is needed to provide adequate coverage of the: • Large geographical areas and socio-economic contexts where programming took

place; • Diversity of primary and secondary beneficiary groups; • Differences in experience and outcomes for women versus men within the

primary and secondary beneficiary groups; and, • Requirement to compare beneficiary group experience with the experience of

15

non-participating groups. The core team members of the Evaluation will facilitate the focus group interviews with a particular view to gender sensitivity. Given the team composition, it is estimated that a minimum of 4 focus groups can be conducted each day. Generally, an equal number of focus groups will be conducted in each intervention area (i.e., 3 focus groups equals one per intervention area). However, to compensate for the lack of income related quantitative data for PBAEC (see Section 3.5.6), additional focus groups will be implemented there with beneficiary groups.

Table 3.3: List of Focus Groups

Stakeholder Group Number District Level 6 Current and Ex- Staff (Local) 3 Participating NGOs / CBOs / Banks 3 Local Level 36 – 40 Participating Farmers with Small Ponds 7 – 8 Participating Farmers with Medium to Large Ponds

3 – 6

Participating Women Working in Aquaculture

7 – 8

People Working in Aquaculture Related Activities (Nursers, Pump Operators, Harvesters, Net Makers, etc)

3 Male 3 Female

Non-Participating Farmers With Small Ponds

6

Non-Participating Farmers With Medium to Large Ponds

3

Non-Participating Women Working in Aquaculture

3

Facilitation guides for focus groups can be found in Annex 6. Focus groups will be undertaken following the quantitative data analysis and the initial stakeholder workshops, and an attempt will be made to target them in two unions per intervention area. If possible, one union showing good results from the quantitative data and another showing relatively poor results will be selected. To the extent possible, participants will be selected from the available databases and, also if possible they will be randomly selected. In Mymensingh where lists of participants are not available, unions will be targeted based upon the “Impact Evaluation Study” carried out by Winrock International in 2004. Household names will be drawn from lists from participating NGOs and extension workers. Focus group participants that

16

did not take part in the Danida funded aquaculture extension interventions will also be drawn from two different local areas – one from an intervention area and one from a non-intervention area. The names of these HHs will be gathered from multiple sources including extension workers, NGOs, hatchery owners, etc.

3.5.5 Individual Case Studies In addition to conducting Focus Group discussions and Key Informant Interviews, a ‘case study’ approach will be used to collect information from individual women and men. In keeping with other approaches in qualitative research, the case study aims to understand a person’s involvement in depth, and in its particular context, recognizing its complexity. It also has a holistic focus, aiming to preserve and understand the wholeness and unity of the case. The case study method gives a unitary character to the data being studied by interrelating a variety of facts to a single case. It also provides an opportunity for the intensive analysis of many specific details that are often overlooked with other methods. For the impact evaluation, individual cases will be selected to investigate personal situations that demonstrate valuable aspects of impact (or lack thereof). It is not the intention of such a study to generalize, but rather to understand this case in its complexity and its entirety, as well as in its context. Important and interesting cases will be identified from within the participants of focus group discussions. Individual women or men will also be selected through observation, key informants, and the information given by local people. It is expected that there will be a total of 6 case studies (3 women, 3 men), ideally distributed from three different interventions and located in different districts. However, the cases may also be used to supplement the lack of quantitative data in a particular intervention area.

3.5.6 Quantitative Data The impact evaluation will rely exclusively on secondary data for the quantitative analysis. However, as described in Section 3.4 (Key Challenges), there is a wide variation of available data across the three interventions. Consequently, the methodology proposed for data analysis differs from one intervention area to another, based on data availability. A number of options were considered for each intervention. The following are the selected options, given the data situation in early August 2007. A list of available data sets for all three intervention areas can be found in Annex 7.

MAEC Data availability For Mymensingh, aggregated baseline and impact data is available but electronic databases and related HH names and addresses are not. The lack of proper baseline studies in the initial phase and a general lack of suitable data for impact evaluation in the Mymensingh area has been a major concern of the Evaluation (see also section

17

3.4 on Key Challenges). The IFPRI data set with 1996/97 data includes relevant socio-economic and gender indicators, as well as a distinction between MAEC participants and non-participants. The data includes information on HHs already included in the programme as well as non-participating HHs that fulfil criteria for participation (pipeline). Given that HHs are followed over time, data for pipeliner’s in early 1996 could serve as baseline for participants and non-participants (comparison group) in late 1997. The data and proposed methodology is described more thoroughly below. The IFPRI data set contains information gathered by household surveys from 10 villages in rural areas of Bangladesh. Data were collected as part of an impact evaluation of new agricultural technologies that were disseminated in rural areas of the country. Data were collected in three different sites: (1) Manikganj District (Saturia Thana), (2) Jessore District (Jessore Sadar Thana) and (3) Mymensingh and Kishoreganj District (Gaffargaon, Pakundia and Kishoreganj Sadar Thanas, respectively). The analysis for the evaluation will be limited to the latter sites - Mymensingh and Kishoreganj District, where MAEC was implemented. In each site, a census of HHs was conducted in villages where MAEC had been implemented and in comparable villages where the MAEC was operating, but where the project had not yet started. Three types of HHs were selected from the census for the survey: (1) MAEC-participating HHs, (2) MAEC “likely participants” in villages where the project had not yet started, and (3) a sampling of all other HHs in both types of villages to represent the general population. In each site, approximately 330 HHs were approached. In villages where the MAEC project had started, 110 MAEC “treated” HHs and 55 non-participant HHs were chosen. In villages where MAEC had not yet started, 110 “likely participants” (control HHs) and 55 “non-likely participants” were selected. “Treatment” and “control” HHs were chosen along very similar observable dimensions. The fact that “control” HHs wished to adopt the project when offered to them, assures some similarity between treatment and control HHs in otherwise unobservable dimensions. As a follow up to the census, a four-round survey was conducted by IFPRI in 1996-1997. This survey collected detailed information on HHs and their individual members regarding demographic characteristics, agricultural and aquaculture production, other income-earning activities, expenditure patterns, time allocation, nutrient intake and nutrition, micronutrients, and morbidity status. Family background data were collected individually from males and females from the second survey round. Both men and women were asked about family background, marriage history, premarital assets, and inheritance. In addition, women were asked about transfers at marriage and about indicators of their mobility and empowerment. The surveys, therefore, contain sufficient information to provide outcome indicators in

18

accordance with the main objectives of MAEC. Data and methodology A combination of the four-round IFPRI 1996-97 data and the Household Census Data 1996 (described in IFPRI (1998) and IFPRI (2003)) provides the Evaluation with the opportunity to fulfil the requirements for applying a combination of a difference-in-difference (DD) estimator and selected matching estimators5. This combination of estimators has the potential to significantly improve the quality of the evaluation results by providing robust estimates of the impact of the treatment. The surveys will help to identify which of the Census pipeliners received support from MAEC enabling the use of the DD estimator (in combination with selected matching techniques).6 In this way, it should be possible to control for any initial heterogeneity that is crucial to the credibility of DD estimates. Combining DD with matching selected estimators can assist in addressing the concern in the literature on matching, of eliminating the additive time-invariant component of selection bias. However, there is concern about the short time horizon in the data, and issues of whether any impact of MAEC can be detected using this method in such a short evaluation period. Consequently, while the Evaluation proposes to rely on one main data source from IFPRI for the quantitative impact analysis, analysis and conclusions will be supplemented by household level census data provided by the Bangladesh Bureau of Statistics for 1995, 2000, and 2005. This will be combined with Demographic Health Surveys (DHS) data and should give an aggregate picture of development in a district over time. Controlling for the fact that other donor interventions have been going on in the Upazilas around the district, the Evaluation will try to compare the performance indicators of participating and non-participating Upazilas over time. This approach requires very detailed Upazila level information that is available in these surveys. However, given the aggregate nature of these data, the main focus of

5 In the analysis the Evaluation will rely on several different matching estimators. A principle of the method used is that there is an assumption that assignment to the treatment group is “un-confounded”. This means that there exists a set of observable characteristics (“X variables”), such that outcomes are independent of participation in the programme conditional on X. It is important to restrict the choice of these x variables to ones that are not influenced by the programme. If this is not the case the matching estimator will not correctly measure the effect of the programme, because it will not capture changes in the distribution of the X variables induced by the programme. Consequently X variables will be characteristics of the households prior to entering the programme. In addition, it is assumed the probability of assignment to the treatment group is bounded away from zero and one, which is an identification assumption. This assures that if households with certain set of characteristics choose to participate in the programme, then no observations exist on similar households who would not choose to accept treatment if offered to them. The implication of these assumptions is that systematic differences in outcomes between treated and control households, with the same values of observable characteristics (X variable) are attributable to the treatment. The specific choice of X variables has not been identified yet, however the IFPRI data contains sufficient household level information to facilitate the analysis. The impact estimates from matching will be supplemented by regression-based estimates (relying both on the control function approach and the difference-in-differences estimator). This will enable assessment of the robustness of the quantitative results. 6 Note that a combination of approaches mentioned above will be applied in the analysis.

19

the Evaluation will be on the secondary data source provided by IFPRI. This analysis may be strengthened by a recent re-survey of some of the same HHs by IFPRI in 2007. Due to the recent completion of this IFPRI re-survey, it is not clear at this point how useful the data could be for the evaluation. Apparently, the new data set includes 380 HHs from the earlier surveys, 130 of whom are reported to be MAEC participants. When, and if the data set is provided to the team, it will be analysed for its utility and possible inclusion.

GNAEC Data availability Out of the three intervention areas Noakhali provides, by far, the best available secondary data for undertaking a rigorous impact evaluation. Robust and detailed HH level data exists for both participants and non-participants, including both baseline and impact measurements. Consequently, it should be possible to use Propensity Score Matching (PSM) to select comparison group members with similar characteristics to the treatment group and then perform a DD calculation. The Evaluation will analyze the “2006 Pro-poor study – baseline and impact” and the “2007 Baseline Study for ASPS Phase II, Noakhali”, both of which contain information on participants and non-participants. The Evaluation will then attempt to locate the names and addresses of these HHs in the “Baseline Pond Survey” and “2000 Fish farmers groups (FFG)” databases. Then, PSM will be used to select an appropriate comparison group based on the characteristics of the treatment group surveyed in the 2006 and 2007 studies. Given the ongoing measurement in Noakhali, preference will be given for HHs that were repeatedly measured over 3 or more surveys. Data and methodology The design of the impact evaluation for Noakhali will rely on baseline and follow-up survey data collected for a sample of pond owner participants and for a comparison-group identified through different matching techniques. Baseline and post-intervention information on socio-economic indicators such as pond owner age, sex, family composition, education, occupation, land ownership, HH income, HH assets and women’s role in the decision-making processes, will be collected from a sample of treatment and comparison group pond owners. In addition, specific pond characteristics, fish culture status, and aquaculture production yields have been documented. These data will be used to compare changes in outcomes before and after the GNAEC intervention, and between participants and non-participants using a DD estimator. Ideally, treatment and comparison HHs should be equivalent in all their observed and unobserved characteristics. The only difference between them would

20

be that GNAEC treated HHs benefit from the intervention, whereas comparison HHs would not. However, since random assignment to GNAEC treatment and non-participants did not take place, the Evaluation will use information collected for the evaluation and carry out selected matching techniques to ensure that selected non-participating pond owners are appropriate as a comparison group. Any comparison group members with unusual attributes relative to the treatment communities will be removed from the sample. As such, the methodology proposed is comparable to the one suggested for Mymensingh, a combination of matching and difference-in-difference techniques, although in Noakhali it is based on a baseline survey, as compared to the pipeline approach in Mymensingh. Moreover, the evaluation period (time horizon) in Noakhali is longer than in Mymensingh, which might improve the probability of identifying GNAEC impact (if present) when compared to Mymensingh.

PBAEC Data availability Patuakhali has extensive and available data, but the baseline data for participants, and especially non-participants, is generally weak in the area of income measurement and women’s empowerment. Other than income however, there is some baseline data on non-income socio-economic indicators and repeated measurements of participants over time. The data is well organized in Microsoft Access databases, copies of which have already been provided to the Evaluation. As in Noakhali, a Baseline Pond Survey was carried out prior to programme implementation. The surveys include information on every pond owner within the district. However, the Patuakhali pond survey lacks crucial information on several of the main socio-economic outcome indicators important for the evaluation process.7 The Patuakhali Baseline Pond Survey, therefore, constitutes a particular challenge for the evaluation since proxy variables, such as total HH assets, would be needed for the missing socio-economic indicators, if comparisons are to be made with the other districts. Another source of baseline information for participants only would be the Fish Farmers Group (FFG) databases for Integrated Pond Farming (IPF) and Integrated Gher Farming (IGF). These databases contain repeated measures of variables such as income, occupation, education, family size, total land, and pond size. However, since the databases only contain records for component participants, they cannot be used for DD calculations. Another available data source, the “Baseline Surveys for GNAEC, PBAEC, and

7 There is no, or limited, information on key indicators such as household consumption, household income and women’s empowerment.

21

SLDP-2” carried out by Winrock International during 2002-2003, also has a number of problems associated with the data. While the title of the study is “Baseline”, in fact, the study was undertaken well after PBAEC started. In addition, the sampling method was not randomly based and therefore design error reduces the accuracy of the findings. Consequently, the team proposes to base the quantitative data analysis on the pond baseline survey and the FFG databases, despite the fact that socio-economic indicators are limited and especially for comparative purposes.

Data and methodology

The design of the impact evaluation will rely on baseline and follow-up survey data collected for a sample of pond owner participants and a comparison-group, identified through different matching techniques. Baseline and post-intervention information on socio-economic indicators such as pond owner age, sex, family composition, education, occupation, and land ownership, will be collected from a sample of treated pond owners and non-participating pond owners. In addition, specific pond characteristics and fish culture status have been documented including aquaculture production yields. These data will be used to compare changes in outcomes before and after the PBAEC intervention between participants and non-participants using a DD estimator. Ideally, treatment and comparison HHs should be equivalent in all their observed and unobserved characteristics with the only difference between them being that PBAEC treated HHs benefit from the intervention whereas comparison HHs do not. Since random assignment to PBAEC treatment and non-participants did not take place, the Evaluation will use information collected for the evaluation, and carry out selected matching techniques to ensure that selected non-participating pond owners are appropriate as a comparison group. Any comparison group members with unusual attributes relative to the treatment communities will be removed from the sample. In addition, income changes will be calculated for IPF and IGF participants, although the changes in income will not be able to be compared to a group of non-participants. As such, the methodology proposed is the same as the one suggested for Noakhali, as it is based on a baseline survey undertaken prior to full-scale implementation. However, this survey will be missing the crucial income and asset related information needed to compare changes between participants and non-participants. All possible attempts will be made to bring in proxy indicators or income/asset related data to assess impact in this area, between the two groups. In addition, the focus group and case study methods will be used more extensively in the PBAEC area, in order to address the lack of income related data for comparison purposes.

3.5.7 Site Observations

22

Site observations will be carried out to provide more context for quantitative and qualitative data; to identify issues for discussion in focus groups, and to undertake a “spot check” of reported information. The following site visits will be undertaken on an opportunistic basis depending on the intervention area.

Randomly se le c t ed ponds Visits to ponds would include semi-structured discussions with household members and observations of pond aquaculture activities. Depending on the timing of the visits it is anticipated that observation could be made of liming, fertilization, stocking, feeding, general management and problem solving, distribution of labour, harvesting, and marketing.

Fish Markets Site visits to markets would include semi-structured discussions with fish vendors and observations of the storage of fish; fish varieties; sanitary conditions; prices; distribution of labour; and the role of middlemen.

Hatcheri e s Site visits to hatcheries would include an interview with the hatchery owner and observations of the hatchery activities and processes. Depending on the exact timing of the visits in relation to the cyclical phases of hatchery operations, it is anticipated that the site visits could include observation of the following: employment of men and women; water treatment; the induction of spawning; egg management; first feeding activities; the nursery phase; and the preparation for transportation.

Other aquacu lture / fi sherie s ac ti vi t y s i te s According to relevance and availability team members may also visit processing sites, observe the making or mending of nets, and the distribution and retail of aquaculture inputs.

3.5.8 Participatory Rural Appraisal (PRA) Although a reference was made to PRA in the proposal, the Evaluation has determined that this methodology is not required in this impact evaluation on account of the three following factors: • Availability of quantitative data – Given the amount of available quantitative data

there is less need for a methodology like PRA; • Intervention Modality – The PRA is less suitable for this Evaluation since the

interventions were mainly with groups and individuals rather than a geographical unit such as a village; and,

• Coverage and representativeness – Given the wide coverage and thousands of beneficiaries of the interventions, it would not be feasible to attempt a PRA process that would provide any kind of representativeness.

23

3.6 Analysis Once the data has been collected, the Evaluation will undertake a number of different areas of analysis. This complex process will be based on interaction and contributions from all team members in order to ensure that all key competencies within the team will be reflected in the analysis. While the various facets of analysis are broken down for the purposes of this Inception Report, it is understood that they are all interrelated and need to be considered in a holistic fashion.

3.6.1 Completion of the Intervention Profile Based on the data collected, the preliminary intervention profile will be finalized and will contain a detailed synopsis of all the interventions undertaken. It is anticipated that this will include financial information on resource use by each component over time; key policy decisions or changes; a map of the evolution of interventions over time; geographical focus of the interventions and changes over time; and key areas of results achieved versus planned results. The finalized intervention profile will also include an updated “logic model” of the causal chain established during the inception period, which will assist in the process of determining causality.

3.6.2 Major Outcomes and Impacts Achievement The evaluation will combine data from the variety of sources, collected by the mixed methods used, to paint a detailed picture of achievements, especially versus the evidence of the counterfactual. The analysis of the actual achievements at each stage of the process will determine the causality from completed activities through to the final impact.

3.6.3 Review and Analysis of External Factors By means of the analysis, the Evaluation will create a list of influencing factors, mainly focused on programs of other entities than Danida (donor, GoB, and NGOs) in the intervention areas. These factors will then be analyzed vis-à-vis the logic model and the intervention profile, along with data collected from the fieldwork phase. Where influences are detected these will be systematically mapped and their effects articulated. This will assist in the assessment of major achievements (outcomes and impact) and for attribution of these to the aquaculture interventions.

3.6.4 Sustainability Analysis The sustainability analysis will include several elements. There will be an analysis of the financial, technical, and institutional sustainability of interventions in order to produce a synthesis of the sustainability of results. The financial sustainability analysis will answer the question of whether the necessary financial resources are still available to sustain outcomes and impact. The technical analysis will consider the various technical inputs required to sustain achievements and whether those continue to be

24

available. Finally, the institutional capacity required to sustain the outcomes and impact will also be examined. Together, these three elements of analysis will be combined to produce findings and conclusions on the overall sustainability of aquaculture interventions (actual or potential).

3.6.5 Spill Over and Replication Analysis Spill over and replication analysis will consider the extent to which major intervention elements, components, and thrusts have permeated, either intentionally or in an unintended fashion, beyond the boundaries of programming (temporal, geographical, beneficiary group, etc). The analysis will produce an understanding of the “ripple effect” of the interventions.

3.6.6 Assessment of Attribution and the Counterfactual Using a combination of secondary data analysis and the focus group data, the question of attribution will be analyzed. This will focus on the determinants of the changes identified through the DD measurement, linking them back to either specific interventions or other external factors identified in Section 3.6.3. The DD analysis will also be used to determine what would have been the likely outcome or impact if the interventions had not taken place.

3.6.7 Synthesis of Evaluation Findings Upon completion of initial quantitative and qualitative analyses, following the principles and methods mentioned above, the full core Evaluation team will take part in a synthesis workshop in Bangladesh. This internal workshop is planned for 5 days and will allow the multi-disciplinary team to consider carefully and discuss all aspects related to the evaluation exercise and the analysis carried out. The team expects that this internal team working session will contribute to a further strengthening of the analysis carried out so far and build consensus within the team on key findings. The synthesis of the evaluation results will eventually lead to preparation of the Draft Evaluation Report.

3.6.8 Validation Workshop A validation workshop will be carried out following the submission of the draft evaluation report and is scheduled for April 2008 in Bangladesh. It will involve a presentation and a thorough discussion of all key findings to relevant stakeholders. The purpose of the validation workshop will be to examine and discuss draft evaluation findings, conclusions, and recommendations, and to provide the Evaluation with any direction and information needed to address factual errors and to

25

refine their findings prior to preparation of the Final Evaluation Report. The Team Leader, together with the Deputy Team Leader and the local experts will facilitate the workshop and give presentations. The comments and input provided at the workshop will be synthesized and used for preparation of the Final Evaluation Report.

4.0 Operational Plan

4.1 Components

4.1.2 Planning and Implementation Tasks (September 2007 – January 2008) • Document and Literature Review – Continue the review as new documents

become available; • Quantitative Data Acquisition – Acquisition of data sets from recent relevant

surveys carried out in the intervention areas; • Quantitative Data Assessment - Sorting, selection of records, cleaning if needed

and processing of the available data sets; • Pre-Departure Preparation and Logistics – Setting up meetings, arranging vehicles

and booking meeting rooms, in order to prepare for the workshops, focus groups, and other data collection activities;

• Initial Key Informant Interviews – Undertake a series of initial interviews at the national level (Bangladesh) and at Danida in Copenhagen prior to initiation of the fieldwork. These interviews will be used to provide further understanding of the context, overall intervention success, and to identify areas of focus for the other data collection methods;

• Workshops – Organise and facilitate an Initial National Stakeholder Workshop in Dhaka and three Regional Stakeholder Workshops;

• Questionnaires – Adjustment and refinement of the draft questionnaires developed for focus groups and individual interviews, based upon the workshops, initial interviews, and the preliminary results of the quantitative data analysis;

• Focus Groups and individual Interviews – Undertake focus groups and interviews in all three intervention areas; and,

• Site Visits – Undertake site visits to ponds, hatcheries, etc in conjunction with focus groups and key informant interviews.

4.1.3 Analysis Tasks (September 2007 – March 2008) • Preliminary Analysis – Undertake preliminary data analysis including collation and

summarization of data from all sources, including quantitative data, according to

26

the evaluation issues; • Team Meeting – Carry out a five-day team meeting in Dhaka in early March 2008

to go through the data and analysis in a multi-disciplinary fashion. The team meeting will develop detailed findings, conclusions, recommendations, and lessons learned according to the evaluation issues; and,

• Follow-up Analysis – Conduct further analysis as required following the team meeting.

4.1.4 Reporting Tasks (February 2008 – June 2008) • Debriefing Note and Presentation on Preliminary Findings – Prepare a debriefing

note containing preliminary findings following completion of the field mission. This will be presented at the RDE in Dhaka and to the Evaluation Department in Copenhagen at the end of January 2008;

• Draft Evaluation Report – Prepare the evaluation report detailing the intervention profile, evaluation activities, findings, conclusions, recommendations, and lessons learned for delivery by March 31, 2008;

• Evaluation Workshop (Bangladesh) – Prepare a PowerPoint presentation of the key elements of the draft report three weeks after circulation. A workshop will be organized in Dhaka with evaluation stakeholders and the report will be presented and discussed over a one-day session;

• Final Evaluation Report – Prepare a final draft of the report, taking into account comments received on the draft report and the feedback from the workshop, to be delivered by May 31, 2008;

• Evaluation Summary – Prepare a four-page summary of the evaluation to be submitted to Danida at the same time as the Final Evaluation Report;

• Team post-mortem report of the experience – Prepare a five-page team report capturing the crucial experiences and lessons learned from the team of the evaluation process, submitted to Danida after the Final Report Presentation in Copenhagen; and,

• Final Report Presentation (Copenhagen) – Prepare and present a summary of the final report in Copenhagen, June 2008.

4.2 Schedule The following is the planned schedule for the impact evaluation following the inception phase. Table 4.1 Overall Impact Evaluation Schedule

Component Timing Planning and Implementation Tasks September 2007 – January 2008 Document Review Ongoing Assessment and Cleaning of Secondary Data for Quantitative Data Analysis

Ongoing

Key Informant Interviews Copenhagen (Danida staff): October

27

Component Timing - November 2007

Dhaka: October – November 2007

Regions: November 2007 – January

2008 Initial National Stakeholder Workshop, Dhaka November 2007 Fieldwork Planning and Logistics October 2007 – January 2008 Fieldwork Implementation: Regional Stakeholder Workshops, Focus Groups, Interviews and Site Visits – approx. 1.5 weeks per District

November 2007 - January 2008

Follow-up Data Collection January - February 2008

Analysis Tasks October 2007 – March 2008 Preliminary Quantitative Data Analysis

October - December 2007 Preliminary Analysis of Primary Data

January – February 2008 Team Meeting (Dhaka)

March 2008 (5 days) Follow-up Analysis March 2008 Reporting Tasks March 2008 – June 2008 Debriefing Note January 2008 Draft Evaluation Report March 31 2008 Evaluation Workshop (Dhaka) April 2008 Final Evaluation Report May 31, 2008 Evaluation Summary May 31, 2008 Final Report Presentation (Copenhagen) June 2008 Team Post-Mortem Report of the Experience June 2008

The following is the planned schedule represented in a Gantt chart format. Table 5.2 Impact Evaluation Schedule by Month

29

Sep

tO

ct

No

vD

ec

Jan

Feb

Mar

Ap

r M

ay

Ju

n

1.0

Imp

lem

en

tati

on

1.1

Fie

ldw

ork

Logis

tics

1.2

Docum

ent R

evie

w

1.3

Quantita

tive D

ata

Assessm

ent and

Cle

anin

g

1.4

Sta

kehold

er

Work

shops

Dhaka (

Nov 2

5)

Mym

ensin

gh (

Nov 2

6)

Noakhali

(Nov 2

8)

Patu

akhali

(Jan 3

0)

1.5

Key Info

rmant In

terv

iew

s

Copenhagen (

Danid

a S

taff

)

Dhaka

1.6

Fie

ldw

ork

(K

ey Info

rmant In

terv

iew

s,

Focus G

roups, and S

ite V

isits

Mym

ensin

gh (

Jan 9

-17)

Noakhali

(Jan 2

0 -

28)

Patu

akhali

(Jan 3

0 -

Feb 8

)

2.0

An

aly

sis

2.1

Pre

limin

ary

quantita

tive d

ata

analy

sis

2.2

Pre

limin

ary

analy

sis

of

prim

ary

data

2.3

Team

Meeting (

Dhaka)

2.4

Follo

w-u

p a

naly

sis

3.0

Report

ing

3.1

Deb

riefi

ng

No

te

3.2

Dra

ft E

valu

ation R

eport

(M

arc

h 3

1)

3.3

Evalu

ation W

ork

shop (

Dhaka)

3.4

Fin

al E

valu

ation R

eport

(M

ay 3

1)

3.5

Evalu

ation S

um

mary

(M

ay 3

1)

3.6

Fin

al R

eport

Pre

senta

tion

(Copenhagen)

3.7

Team

post-

mort

em

report

of

the

experience