SCIENCE EVALUATION AS A PREREQUISITE FOR PROMOTING EXCELLENCE IN RESEARCH

NOVEMBER 6, 2012CHISINAU, MOLDOVA

EVALUATION OF RESEARCH PERFORMANCE:

A System’s Approach (Klaus Schuch, ZSI)

Contents

21/04/23EVAL-INNO

2

General aspectsDifferent levels of Science and Technology Policy

Evaluation Project Evaluation Programme/Instrument Evaluation Institution Evaluation Policy Evaluation

Situation in Southeast EuropeThe EVAL-INNO project

Public Interest

21/04/23EVAL-INNO

3

Democratic values: transparency and fairnessS&T policy values:

S&T policy has to be in a position to show how and why R&D investments are worthwhile

a developed culture of evaluation is an integral part of a strategically oriented S&T policy

Evaluations help to reduce the risk of future interventions to combat market and system failure

Evaluations contribute external knowledge and opinions to policy-making and policy-delivery and also provide the necessary documentation of how policies are implemented

But evaluations do not replace policy action!

Regulatory Framework – Example Austria

21/04/23EVAL-INNO

4

• Public Procurement Law• Opt for “economically most advantageous offer” and not always

“best price”• Include also knowledgeable experts from outside the country

• In Austria a soft law is in place that demands evaluations for each R&D programme

• Guidelines to support economic-technical research and technology development (2007 )

• Recommendations of the Austrian Council for Research and Technology Development on the Evaluation and Monitoring of RTDI Programmes from 12 April 2005

• Evaluation Standards in Research and Technology Policy issued by the Austrian Platform for Research and Technology Policy Evaluation (FTEVAL)

21/04/23EVAL-INNO 5

Utilisation of Evaluations

21/04/23EVAL-INNO

6

• Create a demand for evaluation results by• Ensuring the support of top-level decision-makers• Raising realistic expectations• Integrating those involved in and affected from

• Create an appropriate environment by• Carrying out evaluations systematically• Planning evaluations within the policy cycle• Ensuring access and collecting all relevant data – and only that!• Reserving sufficient funding for a sound evaluation• Ensuring the methodological quality of evaluations• Follow-up on the implementation of results• Designing appropriate communication strategies etc.

EVALUATIONS ARE NOT CARRIED OUT FOR THEIR OWN SAKE!

Functions of Evaluations

EVAL-INNO

Legitimisation:

Information:

Learning:

Steering:

Mediation:

justifying the use of public

funds

to the public how funds are being used and to what effect

for all stakeholders to do things better

for establishing policy objectives, planning of measures etc.balancing interest and dialogue

21/04/23

7

Evaluation Pyramid

21/04/23EVAL-INNO

8

Project evaluations

Programme evaluations

Institution evaluations

Portfolio evaluations

System evaluations

Policyevaluations

Instrument evaluations

In a developed NIS evaluation occurs on different levels!

Project Evaluation

21/04/23EVAL-INNO

9

The basis of any evaluation system is a sound project evaluation.• A project is an individual activity with its own fixed time plan and

its own budget. Projects produce concrete results (output, outcomes).

• Ex-ante evaluations project proposals are of regular component of justifying funding decisions. They are either based on peer reviews or jury decisions (with external experts).

• They assess the relevance of the proposal, its likelihood of reaching the objectives (of the project and contribution to the programme, if any), the quality of its content and methodology (research design), the quality of its management, its efficiency in times of expected input-output ratio, its economic viability (if economic R&D), it potential impact etc.

• Larger projects should also be subject to interim or monitoring and terminal or ex-post evaluation.

Austrian Science FundAustrian Science Fund Applied Research (FFG)Applied Research (FFG)

21/04/23EVAL-INNO

Only competition based – all fields of science compete (no quota)

Only peer review based Only international peers

invited Proposals mostly in English Effect on the Austrian

participation in European RTDI

Open and targeted (e.g. thematic calls) – often programme based

Economic content requires discretion

Mix of peer reviews and jury reviews involving external experts

Proposals often in German language

10

Project Evaluation in Austria

Programme/Instrument Evaluation (1)

21/04/23EVAL-INNO

11

A programme is a combination of measures where the underlying intentions refer to each other and which aim at achieving one or several specific defined objectives. A programme usually exists for a fixed time period and has its own allocated and often centrally administered budget.

• An instrument describes a specific form of an intervention. Monetary instruments (funding, financing) are to be differentiated from non-monetary instruments (consultancy, awareness-raising measures, regulations, standards). Programmes can use one instrument or a combination of instruments (mix of instruments).

Programme/Instrument Evaluation (2)

21/04/23EVAL-INNO

12

Ex-ante concept evaluations review the mission, assumptions, fundamental hypotheses and framework conditions of any possible interventions.

• Design evaluations deal with the organisational structure of an intervention (timing, budgeting, conflict resolution), control whether the measures used are suited to meet the objectives and if it was possible to reach them. They are typical elements of programme evaluations at all stages (ex-ante, interim, ex-post). The trend goes from a mere “before and after” comparison (black-box evaluation) to process evaluation (typical for formative interim evaluations).

• Impact analyses (mostly ex-post) assess the extent to which the objectives of an intervention have been achieved and aim to identify all relevant directly or indirectly, intentional or unintentional caused effects in a summative way. They look at impacts, outcomes and outputs, scientific effects, social effects and economic effects.

Institution Evaluation

21/04/23EVAL-INNO

13

Institutions are mostly of permanent nature. Focus is on universities with their combination of research and teaching, research institutions, intermediaries, and funding institutions or agencies.

• Institution evaluations are often peer based. They should help to avoid lock-in phenomena and blind spots.

• When assessing the research performance the focus should be on internal, external and negotiated factors concerning

• knowledge generation• knowledge utilisation• knowledge diffusion

• The RECORD Benchmarking Manual provides a useful methodology for transition and catching-up countries.

• Example: Benchmarking of Nanotechnology Research Institutes in EECA

Policy Evaluation

21/04/23EVAL-INNO

14

Policy constitutes a set of activities (funding instruments, procedures, regulations etc.) which may be very different in type but share a common motivation or objective. In contrast to projects and programmes, policy is generally not limited by time nor budget.

• Policy evaluations are often carried out in form of international peer reviews (e.g. DAC peer reviews of OECD in the field of development assistance). In S&T policy initiated through CREST (now ERAC).

• They critically examine the existing S&T policy mix by using a “critical friends” approach, provide policy recommendations and engage in a dialogue with policy makers and S&T experts from the host country to increase mutual understanding and learning.

• A S&T Policy Mix Peer Review is rather a (mutual) learning than a rigid assessment exercise!

S&T Policy Mix Peer Review Moldova

21/04/23EVAL-INNO

15

Carried out 2012 within the INCO-NET EECA project by involving S&T policy experts from Austria, Belarus, Estonia, Germany and Greece.

Hosted by the Academy of Science of Moldova• Focus:

• Public research and innovation system• Private sector RTDI• Funding of RTDI• Human Ressources• Internationalisation and FP7 association• Regional dimension

Situation in SEE

21/04/23EVAL-INNO

16

Number of R&D programmes, applied instruments etc. grows, but …

Capacities to evaluate RTDI programmes, instruments, policies and organisations are underdeveloped

lack of knowledge on professional tendering procedures (including public procurement laws) to obtain the best evaluation results

Potential of evaluations for strategic intelligence building is not exploited (no policy cycle; no systematic approach)

• Difficulties to access comparable RTDI evaluation information and good practices

Difficulties to access international databases with important output data (e.g. patents, publications)

• Lack of completed good practice RTDI programmes, institutions and policy evaluations

Lack of systematic exchange with evaluators in EU and globally.

EVAL-INNO Objectives

21/04/23EVAL-INNO

17

Key objective is to strengthen regional as well as national evaluation capacities in order to improve the framework conditions for innovations (policies, programmes, projects, institutions).

In particular, the project aims to promote the role of RTDI evaluation for a reflexive

learning innovation system to develop the needed capacities for comprehensive RTDI

evaluations to provide procedural and methodological know-how and

tool-kits both on the side of evaluators and on the side of awarding authorities.

EVAL-INNO Core Activities

21/04/23EVAL-INNO

18

1. mapping evaluation experts and RTDI programmes in SEE2. training evaluators and policy delivery systems3. strengthening the use of external evaluation through right-

and meaningful public procurement4. establishing a virtual regional RTDI evaluation platform5. developing standards6. running RTDI programme evaluations and benchmarking

exercises7. linking to international networks

EVAL-INNO Training Invitation

21/04/23EVAL-INNO

19

1. 4 Training Courses2. Separate training courses for evaluators and policy delivery

systems including programme owners Bulgaria: March 2013 Hungary: April 2013 Montenegro: June 2013 Serbia: October 2013

EVALUATORSEVALUATORS POLICY DELIVERY AND PROGRAMME OWNERSPOLICY DELIVERY AND PROGRAMME OWNERS

21/04/23EVAL-INNO

Hermeneutics and theory of evaluation

Rules and ethics Real evaluation cases from

different policy spheres Concrete evaluation

methods (qualitative & quantitative)

Exercises

Hermeneutics and theory of evaluation

Rules and ethics Real evaluation cases from

different policy spheres Public Procurement ToR construction Development of Monitoring

Systems

20

Eval-INNO Training Invitation

Training Course for Programme Owners

21/04/23EVAL-INNO

21

Monday Tuesday Wednesday Thursday

Introduction to the course and overview of RTDI evaluations

RTDI System Evaluation. A case study

Evaluation Platforms. The Austrian example

Constructing the Terms of References

History of RTDI evaluation, definitions, types, levels, timing of evaluations

A Structural Funds Operational Programme Evaluation. A case study

Evaluating Economic Impacts

Constructing the Terms of References

       

Design of evaluation-logic charts

Programme Evaluation. A case study

Evaluation of social impact of research

The EVAL-INNO Platform and Databases of Evaluators

Rules and ethics for evaluators and commissioning institutes

Research Institute Evaluation. A case study

Overview of evaluation basic tools and methodologies

Monitoring of evaluations from the perspective of the awarding authorities.

       

Competence of evaluators and awarding authorities

University Evaluation. A case study

RTDI Public Procurement Legislation in ERDF and IPA countries

real-case based group exercises

Utilisation of evaluation results - Usefulness of evaluation

Ministry/Research Agency/ Awarding authority Evaluation. A case study

RTDI Public Procurement Legislation in ERDF and IPA countries

real-case based group exercises

Training Course for Evaluators

21/04/23EVAL-INNO

22

Monday Tuesday Wednesday Thursday Friday

Introduction to the course and overview of RTDI evaluations

RTDI System Evaluation. A case study

Evaluation Platforms. The Austrian example

Bibliometrics and patent analysis

expert panels, focus group, participatory evaluation approaches

History of RTDI evaluation, definitions, types, levels, timing of evaluations

A Structural Funds Operational Programme Evaluation. A case study

Evaluating Economic Impacts

RTDI Indicators Foresight and technology assessment

         

Design of evaluation-logic charts

Programme Evaluation. A case study

Evaluation of social impact of research

Questionnaires, Interviews, and field/case studies

Network Analysis

Rules and ethics for evaluators and commissioning institutes

Research Institute Evaluation. A case study

Overview of evaluation basic tools and methodologies

Benchmarking Analysis Additionality. Control Groups

         

Competence of evaluators and awarding authorities

University Evaluation. A case study

real-case based group exercises

real-case based group exercises

real-case based group exercises

Utilisation of evaluation results - Usefulness of evaluation

Ministry/Research Agency/ Awarding authority Evaluation. A case study

real-case based group exercises

real-case based group exercises

real-case based group exercises

Contact

Klaus Schuch

ZSI – Centre for Social Innovation Austria

http://www.eval-inno.eu

23

21/04/23EVAL-INNO