Independent Evaluation Arrangement Quality of science in CRP evaluations ISPC11: CIFOR, Indonesia Rachel Bedouin (via Skype)

March 2015

CRP Evaluations Current Status Final Evaluation

Report

Responses

CRP Mgmt Consortium

PIM Finalizing report April 2015 April 2015 May 2015

AAS Finalizing report April 2015 April 2015 May 2015

Maize Finalizing report April 2015 April 2015 May 2015

Wheat Finalizing report April 2015 April 2015 May 2015

Synthesis of first 5 CRP evaluations July 2015

CCAFS Inquiry phase Nov 2015 Dec 2015 Jan 2016

RTB Inquiry phase Dec 2015 Jan 2016 Feb 2016

WLE Inquiry phase Dec 2015 Jan 2016 Feb 2016

GRiSP Inquiry phase Oct 2015 Nov 2015 Dec 2015

Livestock and Fish Inquiry phase Dec 2015 Jan 2016 Feb 2016

CRP Commissioned Evaluations (IEA QA support) CRP Mgmt QA Review (IEA)

A4NH Inquiry phase July 2015 Oct 2015 Feb 2016

Dryland Systems Startup phase Oct 2015 Dec 2015 Feb 2016

Dryland Cereals Team selection Dec 2015 Feb 2016 Feb 2016

GrainLegumes Team selection Dec 2015 Feb 2016 Feb 2016

Humidtropics Team selection Nov 2015 Dec 2015 Feb 2016

Synthesis of all 15 CRP evaluations April 2016

Evaluation Criteria

Relevance

Efficiency Quality of Science

Effectiveness Impact and

sustainability

Gender Partnerships and

Capacity Development

Special attention to:

Programmatic Aspects within evaluation criteria

• Research and development challenges

• Coherence of design and portfolio alignment

• Quality of Science

• Comparative Advantage

• Progress against commitments / likelihood of outcomes

• Evidence of impacts from past research

• Partnerships for effectiveness

• Theory of Change

Quality of Science - Approach

Harmonized approach to assessing QoS although tailored to the type of research and Evaluation Team’s own touch

Quality of scientists • H-indexes (combines volume and citation; commonly used but limitations) • Other considerations: allocation of competences, appropriate skill mix

Environment • Infrastructure; research data management; research design • QA processes, learning and knowledge management; incentives to researchers for good QoS

Outputs • Bibliometric analysis (quantity, venues [grey, peer-reviewed], citations – use suitable

comparators) • Qualitative peer analysis (best publications by cluster/FP) • Use and downloading of Web-based outputs • Qualitative assessment of non-publication outputs

– Genetic materials (distribution, traits, breeding process) – Data – Databases, training modules, tools (modelling etc.) etc.

QoS: WHEAT CRP Evaluation (1of2)

General assessment of high quality and eminence of program and project leaders, principle investigators and also, other research staff

Scientists: High quality and eminence of senior researchers; for younger staff mentoring needed.

Environment: Internally, perceptions of WHEAT researchers are quite favourable regarding mechanisms and processes to ensure quality of science; most satisfied with “infrastructure needed for high quality science”, least satisfied with “personal incentives for high quality research” and “encouragement for learning from failure”.

Outputs: Publication record is very good /success in publishing in appropriate and high impact journals that ensure visibility of work within the relevant discipline

• Publication volume varied markedly across FPs due to staffing levels, financial resources, and the role of collaborators in driving project topics, activities and orientation (upstream vs downstream).

• High quality of germplasm and advanced lines in terms of key traits and novel trait combinations (e.g. resistance to UG99)

• Scope to improve efficiency of breeding pipeline through breeding approaches and program design

• SI team: call for Enhanced Lateral Learning, Investment in Systematic Review and Meta-Analysis and Long term trails.

• Data management needs improvement.

QoS: WHEAT CRP Evaluation (2of2)

QoS: MAIZE CRP Evaluation Scientists

• Large proportion of new recruits - need for mentoring

• H-index range shows a reasonable proportion of senior scientists who have good visibility, but comparisons are difficult

Environment: Improvement needed on personal incentives for high quality

Outputs • Good publication record, success in publishing in appropriate & high impact

journals but higher portion published in journals with no IF (compared to WHEAT).

• Accessible data sets from Sustainable Intensification research & important IPG

• Valuable genetic materials at different stages of advancement

• Need for a better stage-plan for breeding and delivery

• Overall need for better processes and protocols, including data management, given large growth in staff and bilateral projects

QofS: PIM Evaluation (1of2)

Builds on (i) Stripe Review of Social Sciences in the CGIAR and (ii) ISPC commentaries on PIM proposals

Scientists • Variable in terms of productivity (skewed distribution with

23/137 PI producing more than 5P/year but >50% PI did not produce peer-reviewed publications;

• Variability IFPRI and non-IFPRI scientists;

Environment • Good QA mechanism, in particular at IFPRI

Topics/Design/Methods • High quality of scientific thinking, State-of-the Art knowledge and

novelty in research approaches;

• Less successful in supporting large research programs involving many disciplines and longer-term data collection (ST perspective of W1/2; research overwhelmingly dominated by economic perspectives) → Rec in line with the Stripe Review

Outputs • Large volume of publications during the first 3 years of PIM;

• Well-placed in some of the best journals;

• Excellent examples of PIM-supported work with large number of citations – but too few.

• Too many non peer-reviewed publications;

QofS: PIM Evaluation (2of2)

QofS: AAS Evaluation (1of2) Scientists • Research staff are enthusiastic about the program • AAS has too few PhD researchers and they are spread too thinly over

hubs and themes; majority of PhDs are at HQ and several are not sufficiently engaged in research

Environment • Survey shows satisfaction of how science quality is managed – high

satisfaction with “encouragement for learning from failure”

• Research design : heavy emphasis on community engagement reduces focus on aquatic agriculture paradox and issues where there is IPG potential

• Approach and methods not sufficiently based on accumulated knowledge on farming systems and PAR

QofS: AAS Evaluation (2of2) Outputs

• Good publishing record on legacy research with shift to publishing on new

areas, e.g. policies and institutions

• Roll-out processes completed

• Good gender strategy and establishment of Gender Transformative Approach

in two hubs

• Development of methodological guidance and protocols started recently