E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR

‘Using evaluation research as a means for policy

analysis in a ‘new’ mission-oriented policy

context’

E. Amanatidou, UNIMAN / MIoIRI. Garefi, Atlantis Consulting S.A.P. Cunningham, UNIMAN / MIoIR

A. Gök, UNIMAN / MIoIR

2012 EU–SPRI Conference, Karlsruhe, 12-13 June 2012

Outline• The ‘new’ policy context• Special nature of grand challenges• Challenges for evaluation• Meeting the evaluation challenges• Usefulness of evaluations• Impact assessment• Behavioural additionality• The Structural Funds framework• Meeting the evaluation challenges - revisited

The ‘new’ policy context

• Mission – oriented • Orientation to dealing with ‘grand challenges’• Not new as policy orientation but some important

different features:– From national, military – industry lead projects in the

‘40s & ‘50s to more global, socially-driven endeavours oriented to tackle challenges impossible to solve by single entities or countries or rational planning approaches

Special nature of grand challenges

Need to break long-standing boundaries; need approaches that are:•Multi-disciplinary in science & technology•Addressing s&t as well as social innovations•Multi-level in governance•Cross departmental in policy•Multi-actor, multi-agency•Cross – sectoral•Applying longer-term horizons

Grand challenges Evaluation challenges

Challenges for evaluation (1/2)

Multi-disciplinarityMulti-disciplinarity

Multi-level gover’nceMulti-level gover’nce

Policy coordinationPolicy coordination

Multi-actor / agencyMulti-actor / agency

Long-term approachLong-term approach

S&t and social innov.S&t and social innov.

S&t fragmentation

Policy silos

Broader set of stkh’s

Impact identification

Multiple impact types

Multiple levels of ref.

Evaluation challenges Challenges’ groups

Challenges for evaluation (2/2)

S&t fragmentation

Policy silos

Broader set of stkh’s

Impact identification

Multiple impact types

Multiple levels of ref.

Need for policy learning at operational, policy and especially

system level

Wider set of impactsBeyond S&T&E

Beyond inputs/outputs,Behavioural change

Evaluation challenges

Meeting the evaluation challenges


system level



Usefulness of evaluations

Usefulness of evaluations

Impact assessmentImpact assessment

Behavioural additionalityBehavioural additionality

Usefulness of evaluations (1/2)

INNO Appraisal results showed that:

• Significant positive correlations with usefulness were identified for:

– Use of open tendering process when commissioning an evaluation

– Use of external evaluators

– Summative over formative evaluations

– Evaluations that examined goal attainment and effectiveness and policy/ strategy development

– Evaluations that employed: case study analysis; participant surveys; interviews; focus groups/workshops and meetings; peer review

– Evaluations that resulted in minor redesign or expansion/prolongation of the measure

– Evaluations not conducted primarily for auditors/financial authorities

– Evaluations whose reports were published in English

• Evaluations of measures for science-industry cooperation and creation of start-ups/spin-offs significantly more useful

Usefulness of evaluations (2/2)*

Broadly, an evaluation may be considered useful if it:

• delivers the Terms of Reference in a consistent manner

• provides actionable recommendations

• delivers value for money

• delivers some degree of policy learning.

• BUT evaluations not extensively used to mobilise the community (only 50% targeted beyond policy makers and programme managers)

• usefulness is highly subjective and context specific

* Based on the INNO-APPRAISAL study

Impact assessment*

• Impact assessment limited and simplistic in its approach • Assessment of economic impact most dominant• Assessment of new impact types rather uncommon• Demand for non-economic impacts and spill-over effects• What new impacts under the ‘new’ mission orientation?

(beyond intended, beyond visible, that can last, that span across and beyond levels of references, that refer to multiple stakeholders, and cross different policy arenas)

• New sets of criteria and indicators required• Known challenges in assessing social impacts (e.g. causal

links, evidence, attribution problems, timing)


Behavioural additionality (1/2)*

• Emphasis on learning, long-term horizon• Gaining importance (addressed in 50% of national evaluations)• Three types of use:

– behaviour-focused way– Integrated ways– Instrumental way

• BA evaluations broadly discussed and more often targeted towards the general public and towards users

learning and mobilisation potential• A need to demonstrate the conceptual link between the

behavioural change and (intended) innovation effect• The ‘black box’ remains


Behavioural additionality (2/2)*

• If an evaluation is considered as an administrative exercise that was imposed by a supra-national sponsor, the policy learning and behavioural additionality it creates is limited. However, if it becomes part of the policy learning experience its usefulness and behavioural additionality increase.

Context and wider framework within which evaluation is implemented crucial for both usefulness and BA


The Structural Funds frameworkIf and how the SF framework affects the quality and usefulness of

evaluations and thus the potential for BA*

SF evaluation requirements in relation to• Structures• Evaluation design• Evaluation execution• aim, type, and nature of evaluations• Evaluators’ selection• Publicity & Dissemination• Quality assurance

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

The Structural Funds frameworkIf and how the SF framework affects the quality and usefulness of

evaluations and thus the potential for BA*

Research hypotheses• SF requirements may lead to specific characteristics in

delivery & practice of evaluation• SF requirements may lead to higher quality evaluations• High quality SF evaluations may have greater impact• SF regulations demand high standards on structures and

processes that inevitable need some institutional learning and structure building


SF requirements may lead to specific characteristics in delivery

& practice of evaluation

Use of external evaluators Data Analysis and collection methods: not surprising

differences Evaluation topics: SF requirements do make a difference

in guiding the evaluation topics to cover across the different evaluation types (ex-ante, interim, ex-post)

Clearer differences across the evaluation types within each group than across the two groups (SF and non-SF group)

Hypothesis valid but also differences in features may relate more to evaluation type than SF or non-SF framew.


SF requirements may lead to higher quality evaluations X

• Compliance to quality standards is less in SF evaluations• Certain standards clearly ‘overlooked’: production of useful

recommendations; discussion of results with government and stakeholders

SF requirements do not necessarily lead to high quality appraisals

High quality SF evaluations may have greater impact X

• What clearly improves usefulness of recommendations is the evaluation design and application of quantitative methods

• Increased discussions with government and wider stakeholders are more caused by high quality non SF eval.

SF requirements for high quality do not necessarily lead to high impact in terms of usefulness and dissemination



SF regulations demand high standards on structures and

processes that inevitable need some institutional learning and

structure building

• SF evaluations more as ‘internal’ exercises limited discussions with government and stakeholders• SF regulations too focused on financial aspects and

correctness of implementation - lack of qualitative impact assessment or BA

limited quality inputs to new programmes and schemes

limited usefulness of recommendations

high quality SF evaluations ≠ high impacts in terms of usefulness and dissemination to national stakeholders

Typical application of procedures ≠ institutional and policy learning BUT significant capacity building

Usefulness: not extensive; relates more to operational issues

Usefulness: not extensive; relates more to operational issues

Impact assessment: Several issues pending;

new issues emerging

Impact assessment: Several issues pending;

new issues emerging

BA: Still a ‘black box’ BA: Still a ‘black box’

SF framework: typical application; not

opportunity for learning

SF framework: typical application; not

opportunity for learning


system level



Meeting the evaluation challenges revisited

Thank you for your attention!

E. Amanatidou, UNIMAN / MIoIR, [email protected] I. Garefi, Atlantis Consulting S.A., [email protected]

P. Cunningham, UNIMAN / MIoIR, [email protected] A. Gök, UNIMAN / MIoIR, [email protected]

Documents

E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR