Evaluation plans for 2014-2020 programming period in Poland

Experience and new arrangements

Ministry of Infrastructure and Development, Poland

Athens, 15th May, 2014

Joint Evaluation Network Meeting

Institutional context for evaluation planning

Closer coordination as a main challenge for 2014-2010 period

7 persons

20 persons

35 persons

43 persons

60 persons

2007-2013 period: evaluation plans established for NSRF and all operational programmes

Strategic evaluation plans for the whole 2007-2013 period

strategic goals for evaluation process organisational arrangements (thematic steering groups) main thematic areas main evaluation capacity building activities

Annual evaluation plans list of evaluation studies + description evaluation capacity building activities

Polish experience on evaluation planning 2007-2013

All plans examined by the National Evaluation Unit

Shortcomings: difficult to coordinate no strategic integration of an evaluation process with

implementation and intervention logic → annually planned evaluations mainly for current informational needs

no coherent evaluation agenda for the whole Cohesion Policy

Advantages:

high degree of flexibility

Polish experiences on evaluation planning 2007-2013

Process of evaluation planning in Poland to be regulated by the Guidelines on evaluation activities in the 2014-2020 period (draft)

In the field of the evaluation plans the Guidelines contains: general rules obligations for institutions, and role of the main

stakeholders structure and content of evaluation plans deadlines and logistic/organisational matters list of obligatory evaluation studies (e.g. ex-ante, mid-term,

ex-post, each priority evaluation, annual systematic review of evaluation studies, gender equality evaluation)

Evaluation planning 2014-2020 in Poland

(1) Evaluation process closely related to the intervention logic

evaluation plans (esp. scope and timing) reflect the intervention logic of the programme

intervention logic as a starting point for identification of the informational needs (what kind of and when evaluation evidence will be needed)

all planned evaluation activities should be justified in this context

(2) Planning process closely coordinated by NEU in order to ensure the coherence of the evaluation system

twofold aim of the planning: 1) effective evaluation of each programme 2) effective evaluation of the whole Partnership Agreement intervention

coherent evaluation agenda is needed → operational programme evaluations + evaluations in the PA level (meta-evaluations, systematic reviews, strategic and horizontal evaluations)

Evaluation planning – general principles

(3) Monitoring of evaluation plans implementation

whole evaluation process monitored by National Evaluation Unit

special emphasis on recommendation implementation process → Recommendations Implementation System (RIS)

annual reports on the progress of implementation of evaluation plans (esp. recommendations implementation)

Evaluation planning – general principles

Evaluation plans prepared for every operational programme by the Evaluation Units for the whole programming period

National Evaluation Unit preparing the Evaluation Plan for Partnership Agreement 2014-2020

Annual update as a tool for changes and supplementation

Dilemmas: Evaluation plan for the whole period contains all evaluation studies planned

for each programme vs. contains only strategic and most important studies Annual update of one plan vs. annual plans as a supplement for the

strategic plan Evaluation Units are obligated to ensure:

linkage between the planned evaluation studies and the intervention logic of the programme

high quality of evaluation studies identification of needed data resources needed for evaluation process

Evaluation planning in Poland – proposed rules and procedures

All evaluation plans and their annual updates accepted by National Evaluation Unit

NEU’s opinion based on the criteria of: adequacy, coherence, feasibility, and efficiency in order to ensure the coherence of the whole cohesion policy evaluation process

After the NEU decision, evaluation plans are accepted by Monitoring Committees

Evaluation planning in Poland – proposed rules and procedures

strategic planning vs. current evaluation needs → annual update or annual plans as a tool for flexibility

effective integration of evaluation process with the intervention

logic (and then implementation process) within evaluation plans → high expert and evaluation potential needed → cooperation with external experts + high quality trainings

effective coordination in order to establish coherent evaluation

agenda for the whole Cohesion Policy → National Evaluation Unit responsibility

identification and access to the needed data (in particular for counterfactual studies)

Main challenges for evaluation planning

National Evaluation Unit prepared detailed Guidelines on evaluation activities in the 2014-2020 period with section dedicated to evaluation planning.

After the discussion with the OP Evaluation Units, NEU decided to support

evaluation units, and to organise the evaluation planning process in closely coordinated manner.

NEU is going to launch the project: Process of preparing evaluation plans for Cohesion Policy 2014-2020.

In cooperation with experts on evaluation and sectoral matters

Main goals of the project: Development of the evaluation planning methodology (taking into

account intervention logic) Preparation of evaluation plans for Operational Programmes and

Partnership Agreement

Preparation to evaluation planning in Poland

Main stages of the project:

1. Pilot project – preparation of an evaluation plan for one chosen Polish region (Regional Operational Programme) intervention logic analysis → expert panel workshops with main Regional Operational Programme

stakeholders and partners preparation of the Regional Operational Programme

Evaluation Plan

2. Preparation of the Partnership Agreement Evaluation Plan presentation and discussion on a pilot project findings intervention logic analysis → expert panel workshops with main stakeholders and partners preparation of the Partnership Agreement Evaluation Plan

Preparation to evaluation planning in Poland

3. Preparation of evaluation plans for all Operational Programmeswith the support of the NEU (help desk) and external evaluation expertsNEU opinion

4. Monitoring Committee approval

5. Sending plans to European Commission

Main products: evaluation plans for OP’s and PAmethodology document on evaluation planning

Preparation to evaluation planning in Poland

EVALUATION PLAN OF KNOWLEDGE – EDUCATION – DEVELOPMENT OPERATIONAL PROGRAMME

– MAIN ASSUMPTIONS

A ROLE OF EVALUATION PLANA ROLE OF EVALUATION PLAN

1) Programme evaluation management

2) Predictable information needs fullfilment

3) Evaluation WWW - (What When Who)

4) Rising awarness about preparations to be done

5) Transparency

6) Stakeholder’s interest taken into account

MAIN PRINCIPLES MAIN PRINCIPLES OF THE OF THE EVALUATION PLANEVALUATION PLAN

1) 1 EP for 1 OP

2) Evaluation experts functionally independent

3) Evaluation capacity building

4) Social partners involvement

5) Long-term orientation = flexibility of provisions

FLEXIBILITY FLEXIBILITY OF THE EVALUATION OF THE EVALUATION PLANPLAN

1) EP should be a strategic oriented document.

2) List of evaluation studies is open and

indicative.

3) The list covers only evaluation studies that

are predictable.

4) Detailed information about the studies will be

updated yearly.

1) Definition – Goals – Challanges and Principles

for evaluation process

2) Data access and provisions

3) Process of coordination

4) Capacity building and evaluation culture

5) Use and dissemination of findings

6) Indicative list of planned evaluation studies

ELEMENTS OF THE EVALUATION PLANELEMENTS OF THE EVALUATION PLAN

EVALUATION STUDIES EVALUATION STUDIES CHARACTERISTICSCHARACTERISTICS

1) Subject /area

2) Purpouse and scope

3) Who

4) Evaluation approach planned

5) Tentative term – usually approximately

6) Budget – broadly estimated

7) Other information if necessary

TEMPLATE - EXAMPLE TEMPLATE - EXAMPLE

1) Obligatory studies

2) Metaevalution, syntheses

3) Diagnoses, needs and barriers identification

4) Horizontal principles

5) Specific target groups oriented

6) Comparability studies

7) Capturing the effects

8) Ad hoc studies

TYPES OF EVALUATION STUDIESTYPES OF EVALUATION STUDIES

1) Research obligatory or necessary

2) Proposals from ex-ante evaluation

3) Proposals made by the IB’s

4) 2004-2006 and 2007-2013 experience

5) Policy makers needs – if possible to identify

MAIN MAIN SOURCES OF IDEASSOURCES OF IDEAS

Thank you for your attention!

Tomasz.Kot@mir.gov.plTomasz.Kot@mir.gov.pl

Michał.Sułkowski@mir.gov.plMichał.Sułkowski@mir.gov.pl

Ministry of Infrastructure & DevelopmentMinistry of Infrastructure & Development

POLANDPOLAND