Progress of the SIAC program - Doug Gollin

Standing Panel on Impact AssessmentDoug Gollin, ISPC 10University of Copenhagen

SPIA

Doug Gollin

Chair

ISPC 10 - Copenhagen http://impact.cgiar.org

Karen Macours

Activity leader

Erwin Bulte

Activity leader

JV Meenakshi

MemberBob Herdt

Member

Secretariat staff (L to R): James

Stevenson, Lakshmi Krishnan, Ira Vater,

Tim Kelley

Impact Assessment Focal Points


Tahirou Abdoulaye IITA / HumidTropics Nancy Johnson IFPRI / A4NH

Aminou Arouna AfricaRice/GRiSP Ricardo Labarta CIAT / RTB / GRiSP

Marie-Charlotte Buisson

IWMI Lucy Lapar ILRI

Monica Fisher CIMMYT / Maize / Wheat

Ravinder Malik IWMI / WLE

Steve Franzel ICRAF / FTA Sam Mohanty IRRI / GRiSP

Guy Hareau CIP / RTB Aden Aw-Hassan ICARDA / Dryland Systems

Peter Hazell IFPRI / PIM Frank Place IFPRI / PIM

Elisabetta Gotor Bioversity / RTB Keith Child Livestock & Fish

Charlie Crissman WorldFish / AAS Jupiter Ndjeunga ICRISAT / Grain Legumes / Dryland Cereals


Donor demands study: Background

Objectives:1. Donor demand for and use of ex post impact assessments2. Donor priorities for ex post IA and quality standards:3. Baseline for SIAC.

2005 round: Main findings• Many factors influence donor decisions:

• political priorities• continuity in funding• perceptions of scientific quality

• epIAs important for confidence in the CG System and defending budgets

• Desire for: greater clarity and transparency in epIA studies; broader coverage in research domains; greater focus on deep impacts and magnitude / distribution of benefits

• Some skepticism about accuracy of some past studies of rates of return


Donor demands study: Results


Donor demands study: Results


SIAC Program: 4 Objectives

Objective 1: $1.5 million (approx 4-year budget)

Experiment with new methods for estimating

adoption

Objective 2: $4.3 million

Institutionalize the collection of adoption data


Impact assessment of under-evaluated areas of

CGIAR research


Build community of practice on impact

assessment


SIAC Program – Objective 4

Build community of practice on impact assessment


Objective 4: Capacity-building

Competitive awards to collaborative programs aimed at

capacity-building• CIP and CIFOR with Virginia Tech• ICRISAT with University of Illinois

SPIA-organized technical workshops

Minneapolis, July 2014• impact assessment focal point workshop• workshop on methodology for poverty impacts

Science Forum 2015• impact assessment focal point workshop

Smart new website with resources


http://impact.cgiar.org


Impact assessment of under-evaluated areas of CGIAR research


Objective 3: Nutrition and health impacts“A range of methodologies may be brought to bear on the proper

identification of causal effects. ... We are seeking a portfolio of

complementary and mixed methodologies to show plausible health

and nutrition impacts from research-contributed agricultural

development initiatives...” 2013 call doc.

2013 call: 56 concept notes 5 projects (two-stage external

review)

Columbia University Irrigated horticulture in Senegal

MIT NERICA in Sierra Leone

Virginia Tech High iron beans in Rwanda

CIMMYT Maize-legume intercropping in Ethiopia and

Malawi

ILRI Dairy hubs in Tanzania


Objective 3: Nutrition and health impactsInception workshop, Wageningen, July 2014

20 participants: scientists from the 5 projects + A4NH, LSHTM, WUR, SPIA, ISPC

Various impact channelsDirect e.g. food, income gains for

producersIndirecte.g. food prices, women’s

empowerment

Heterogeneous treatment effectsDistributional issues; who gains and who

loses?Context-specific causal mechanisms and

impacts


Objective 3: Randomized Control Trials call“…proposals should put forward a research plan that promises to add to our knowledge regarding the potential impacts of CGIAR technologies in particular agroecological environments and social settings in specific years. This includes a better understanding and evidence base regarding the potential lack of impacts of certain technologies.”

April 2014 call: 21 Expressions of interest 8 full

proposals

Next steps:

Full proposals have been submitted; now being externally

reviewed

Decision by mid-October to fund 3 studies

Projects to start Dec 2014 / Jan 2015; Inception workshop Jan

2015



Proponents Technology / Program Country

CIMMYT / Harvard Hermetically sealed bags + Aflatoxin Kenya

Yale / ICIMOD Intercropping adoption and social networks Nepal

ICRISAT / UEA Multi-purpose pigeonpea Malawi

ILRI Infection and treatment method Tanzania

UC Berkeley / Tufts / IRRI

Drought resistance and water saving in rice Bangladesh

Tufts / Kilimo Trust Direct contracting and improved bean adoption

Uganda

Purdue / IITA Improved storage bags + credit Tanzania

J-PAL / World Bank / UC Berkeley

Plot-specific soil analysis + personalized input recommendations

Mexico

Objective 3: Long-term / large-scale impact“… studies that seek to measure the impacts of widely-adopted CGIAR research related innovations... ‘research successes’ that, due to their already widespread diffusion, lend empirical support to the global (or regional) public goods argument for CGIAR research.”

Rigorously assessing long-term and/or large-scale impacts is a big challenge; however, donor demand for these assessments remains strong

September 2014: SPIA issued a two-stage call for proposals

Topic / geography / methods left deliberately open

Next steps:Review Expressions of Interest, invite full proposals, review proposals

SPIA to commission 3 or 4 studies to run throughout 2015/16


Objective 3: Under-evaluated areas


Motivation:Majority of CGIAR ex post IAs have focused on germplasm improvement Serious gaps in the other areas of CG research (livestock, irrigation management, agroforestry, policy research, biodiversity, NRM)

Making a start on overcoming this deficiency, starting with irrigation/water management

First step with consultant (Doug Merrey): Commissioned critical review of the IA work to-date on irrigation and water management research, broadly conceived: e.g., including much of IWMI’s research, IRRI’s research on wetting and drying, IFPRI’s water policy research


Institutionalize the collection of adoption data


Objective 2: Adoption data


Impact assessment has been hampered by a lack of core data on adoption of CGIAR-generated technologies and policies

Bill and Melinda Gates Foundation have been the stimulus for addressing this deficiency in recent years (2010 – 2013):• Diffusion and Impact of Improved Varieties in Africa (DIIVA project)• Tracking Improved Varieties in South Asia (TRIVSA project)

Despite these efforts, gaps remain in global picture for crop germplasm

Large areas of research (NRM, livestock, policy research) have no equivalent datasets

Objective 2: Policy research influence


Develop a mechanism to track outcomes of CGIAR policy-oriented research

Compile best available information on policy influence events plausibly linked to research outputs

Phase 1: Underway with consultant (Mitch Renkow)• Draw on information collected in the PMS exercises (2006 through 2010)• Reported “outcomes” (statements & evidence) proposed by Centers• At the time, these were then vetted and scored by the Science Council• Extract from the PMS database those outcomes deemed to be

(a) POR-related(b) external reviewers gave score above a given level

• These form first entries into the database

Objective 2: Policy research influence


Phase 2: Under consideration(a) Review annual reports and other relevant documents published since 2011, from Centers and CRPs(b) Solicit ideas directly from the Centers and CRPs for more recent significant outcome cases

Mechanism and specific steps still to be defined for going from large body of more recent outcome claims, to a smaller set of reviewed and verified influence events

Steps will involve series of interactions among Centers, CRPs, SPIA and consultant

SPIA co-organizing a PIM-hosted workshop at IFPRI in October 2014

Objective 2: NRM & livestock outcomes


• Consultant (Nuri Niyazi) reviewed Performance Measurement System submissions (2006-2010) and Annual Reports (2003-2012) for all 15 CGIAR centers (February – May 2014)• Long-list of approx 200 country x research outcome claims drawn up• NRM conceived very broadly thus far

Next steps:• SPIA to decide on strategy for selecting 40-50 country x research outcomes• SPIA to circulate this sample for comment CGIAR colleagues• Call for EoIs (late September 2014) for institutions, external to CGIAR, to propose process for assessing current status of adoption of these technologies / innovations / management practices• Open to mix of methods – surveys, expert opinion interviews, remote sensing, to be used throughout 2015 and 2016

Objective 2: Improved germplasm adoption


• Regional priority for 1st phase: South Asia, South-East Asia, and East Asia• Initial planning workshop with Centers and NARS completed in early 2014

CIMMYT: 42 crop-country combination (CCCs) estimatesIRRI: 21CIP: 41CIAT: 10ICRISAT: 15MSU/NARS 6 legume crops

• Oversight for all CCCs provided by MSU through sub-grant • Joint workshops with NARS to start data collection• SPIA to participate directly where possible• Centers will submit release and adoption databases in July-October 2015 (delay of about 6-9 months from the original timeframe)

Objective 2: Partnership with LSMS-ISA


• World Bank Living Standards Measurement Study – Integrated Surveys of Agriculture (LSMS-ISA)

• 8 countries in SSA – all important to CGIAR

• Average of 5,000 HHs / country, nationally representative

• Panel – visited every 2 years SPIA role:• Surveys lack modules / questions on

agricultural technologies (varieties, NRM practices)

• SPIA’s comparative advantage to work to improve this for benefit of CGIAR as a whole

Objective 2: Partnership with LSMS-ISA


Frederic Kosmowski ILRI, Addis Ababa

• Scoping Ethiopian priorities across CGIAR centers• Later, Niger / Burkina / Mali surveys

John IlukorIITA, Malawi• Cassava DNA fingerprinting in Malawi• Later, Uganda / Tanzania / Nigeria surveys

Two SPIA Research Associates hosted by CGIAR and working with LSMS-ISA

Objective 2: LSMS-ISA


• Unique identification of technologies – e.g., individual varieties

• Experiments to test farmer ability to provide accurate survey responses using different data solicitation methods

• Integrate new protocols / innovative methods into these large-scale surveys.

Objective 2: LSMS-ISA


• Example: instead of asking farmers whether or not they are retaining crop residues on their fields, ask them to look at photos to turn a (dubious) binary variable into a continuous variable.


Experiment with new methods for estimating adoption


Objective 1:New methods for collecting adoption data

A number of DNA fingerprinting experiments in

Africa

Remote sensing and cell phone apps for

estimating adoption of crop management technologies

– funded two pilots (IRRI and CIMMYT respectively)

Outsourcing adoption surveys to private sector

contractors (2015)

Writing up what we find into good practice guidelines

(2015)


Objective 1:New methods for NRM adoption monitoring

July 2013: Call for concept notes

Seven received, two funded• IRRI: Hyper-spectral signature analysis: a proof of

concept for tracking adoption of crop management

practices• CIMMYT: Mobile phone apps for tracking adoption of

NRM technologies in Indian Agriculture

Status:• Both studies mid-way through implementation• Final reports expected April 2015


Objective 1: DNA fingerprinting methods

BMGF independently working on

this as routine M&E:• Maize and Wheat in Ethiopia• Rice in Asia

3 SIAC-funded crop-country

pilots so far:• Cassava in Ghana• Beans in Zambia• Maize in Uganda (later…)


Preliminary results

suggest that farmers are

frequently unable to

identify their own

varieties correctly.

Expert views also seem

to be flawed, in

comparison to DNA

results.

Objective 1: DNA fingerprinting methods


Objective 1: Cassava in Ghana

500 HHs in three major cassava producing regions - fieldwork jointly supported by SIAC and the RTB CRP

Conducted in partnership with MSU, IITA, Crop Research Institute (CRI)-Ghana, and Agriculture Innovation Consulting (AIC) Ghana

DNA analysis of field samples submitted by Cornell to IITA in July 2014

DNA Convening workshop (August 2014) in Seattle, MSU, IITA and NaCRRI (Uganda) presenting project update and preliminary results

Further analysis of field survey data on-going feed into descriptive report with results of different methods tested for varietal identification


Objective 1: Beans in Zambia

Benchmark is bean seed samples collected from farmers for fingerprinting

Collaboration with CIAT and ZARI (Zambia) – PABRA bean adoption study

Seed samples + data from 4 methods from 402 HHs under PABRA study

Seeds germinated in June 2014, DNA extraction in July 2014, samples shipped to LGC Genomics (K-Biosciences)

Next 3-5 months: 1) establish the library of SNPs (experts at CIAT)2) analyze farmer samples using a short list of SNPs3) results to be compared with the ‘library’ to give a varietal identification for each data point in the list of 1050 seed samples collected


Objective 1: DNA fingerprinting


• Initial results confirm prior suspicions that for at least some crops, farmers assign the same name to different varieties; assign different names to the same variety.

•Results also seem to confirm previous results suggesting that expert opinion may be flawed.

•If results hold consistently, it suggests that we will need dramatically to re-think our methods for collecting adoption and diffusion data.

•May also mean that existing estimates are flawed – but this may differ substantially across contexts.

•Potentially a major issue for SPIA and partners engaged in impact assessment.

THANK YOU

http://impact.cgiar.org


http://impact.cgiar.org/