Evaluation of the system for implementing

recommendations by the institutions involved in the implementation of the HRD SOP 2004 - 2006

and Equal Community Initiative Programme

Report concerning meta-evaluation study

Prepared for: Department of the European Social Fund Management at the Ministry of Regional Development Authors: Ośrodek Ewaluacji Sp. z o.o. Agnieszka Borek (team leader) Magdalena Szostakowska Jolita Karczewska Tomasz Kochan Maria Makowska Magdalena Mentrak Alina Stanaszek

Warszawa, November 2007 r.

This report from a meta-evaluation study of the system for implementing rec-

ommendations by the institutions involved in the implementation of the SOP

HRD 2004-2006 and EQUAL CIP has been prepared by a team of evaluators

from Ośrodek Ewaluacji Sp. z o.o. contracted by the Ministry of Regional De-

velopment.

Published by: Ministry of Regional Development ul. Wspólna 2/4 00-926 Warsaw tel. (+48 22) 461 30 00 ISBN: 978-83-7610-001-2 Department for the European Social Fund Management ul. Wspólna 2/4, 00-926 Warsaw tel. (+48 22) 501 50 01, fax (+48 22) 501 50 31 E-mail: zarzadzanieEFS@mrr.gov.pl oraz pokl@mrr.gov.pl Internet: www.efs.gov.pl Info line EFS 0 801 EFS 801 0 801 337 801 charged as a local call

Publication co-financed by the European Union from the European Social Fund

3

Glossary

ESF

ESG

IB

IB II

IA

MA

MC

MRD

EQUAL CIP

Human Capital

OP (HC OP)

ROP

ToR

DOC

SOP HRD

European Social Fund, one of the structural funds, established to support

the Community Social Policy

Evaluation Steering Group

Intermediate Body

2nd degree Intermediate Body

Implementing Authority

Managing Authority. In the case of EQUAL CI the Managing Authority

is the Department for ESF Management at the Ministry of Regional De-

velopment

Monitoring Committee

Ministry of Regional Development

EQUAL Community Initiative Programme

One of the Operational Programmes prepared for 2007-2013

Regional Operational Programme

Terms of Reference

Description of the Object of the Contract

SOP Human Resource Development

5

Table of contents Glossary 03

Abstract 07

Summary of results 09

Introduction 12

Methodology 13

Results 15

Relevance of contracted studies 15

Evaluation planning 15

Contractor selection procedures 19

Evaluation management 24

Implementation time 27

Systemic solution 29

Evaluation potential at regional level 29

Closing brief 30

Usefulness of recommendations 31

Institutional usefulness of evaluation 31

Dissemination of evaluation results 32

Factors influencing recommendation usefulness 36

Factors influencing recommendation implementation 37

Possible use of recommendations 41

Closing brief 41

Effectiveness of recommendation implementation 42

Recommendation implementation 42

Closing brief 45

Conclusions and recommendations 47

7

Abstract The study prepared by Ośrodek Ewaluacji as commissioned by the Department for ESF Man-

agement at the Ministry of Regional Development indicates that there are difficulties in the

implementation of recommendations stemming from both systemic problems and mental bar-

riers.

Thus far, no effective method/system has been developed to implement recommendations

(first steps in the right direction have already been taken by the MRD and the Polish Agency

for Enterprise Development, PARP). There is no effective system to engage a wide group of

stakeholders in the process of evaluation planning and performance, which reduces the rele-

vance and usefulness of evaluative-study results for the institutions involved in the implemen-

tation of the Programmes under evaluation. Stakeholders are not involved in evaluation since it is not perceived as a tool which supports

the management of the Programmes. Evaluation is still too little known, which is why it is

often treated as a control tool. Best practices in the implementation of recommendations are

not promoted, which does not help in overcoming mental barriers in using evaluation results.

The implementation of recommendations is also influenced by the quality of recommenda-

tions, which is satisfactory, contrary to widespread opinion of the staff of the institutions in-

volved in the implementation of the Programmes. Most recommendations include, apart from

a description of the desired state, quite specific guidelines on what to do to achieve a given

goal. A weakness of the recommendations is that it is often not stated to whom they are ad-

dressed.

Some recommendations fail to take into consideration the regulatory environment of the

structural funds, although it should not be seen clearly as a drawback since one role recom-

mendations should play is showing directions legislative changes should take.

The following actions are recommended in order to improve the implementation of recom-

mendations:

1. incorporating evaluation in the decision-making process by getting members of the

managerial staff (department heads and division directors) involved in Evaluation

Steering Groups and by education concerning evaluation;

2. intensifying activities which promote evaluation as a tool supporting the implementa-

tion of the Programmes;

8

3. creating strong evaluation units in institutions implementing the Programmes, manned

with highly qualified staff so that they can be partners for evaluators and educators in

evaluation for other staff members of such institutions;

4. involving a wide group of stakeholders in the process of evaluation planning and per-

formance as well as capitalising on of evaluation results;

5. conducting a debate concerning the advisability of implementing recommendations,

planning and monitoring of such implementation, and evaluating the results of the

changes made;;

6. coordination by the Managing Authority of all evaluation-related Programme activities

at the central and regional levels.

9

Summary of results

1. The relevance and usefulness of evaluation are reduced due to the shortage of systemic

solutions to specify the scope of evaluation performed by institutions at various levels

of the implementation of the Programmes and to allow for a wider use of data from

project monitoring.

2. The relevance of evaluation performed as part of the SOP HRD and EQUAL CIP is

limited, in most studies, to the needs of the Managing Authority. Given the needs of

lower-level institutions, conclusions and recommendations which are a result of the

study are of little use as they pertain to a higher level of programme management. The

main reason for this is the fact that representatives of stakeholders involved in the im-

plementation of the Programmes (Intermediate Bodies and Implementing Authorities)

are not sufficiently involved in the planning of evaluation.

3. Reducing the involvement in the planning process to the narrow group of Managing

Authority staff results in the staff’s limited ownership of goals and concepts behind

the study only just (if any). As a result, negative conclusions from reports are some-

times challenged whilst recommendations are not implemented as the interested par-

ties fail to give them the green light.

4. The evaluation carried out at the level of the MA does not fully reflect the applicant’s

perspective, relations with the final beneficiaries or the operation of committees which

assess and select applications for implementation; all this reduces the usefulness of the

study at the lowest implementation level.

5. The usefulness of evaluation results is down with a low, as found by the recipients,

novelty value of the conclusions and recommendations coming from evaluation re-

ports. We have identified several reasons for this. First, the work contributed into the

preparation of the request for the evaluation study makes the staff at the contracting

institution experts in a given field. Frequently, the researcher working on material pre-

pared for him/her must go more deeply into only some of the conclusions. Another

factor hampering the inventiveness of the researchers is very detailed terms of order.

Then the study is limited to the issues identified by the contracting parties. The third

factor limiting the usefulness of evaluation is the passive attitude of Evaluation Steer-

ing Groups. Not very enlightening comments contributed by representatives of such

groups stem primarily from the fact that evaluation is not treated as a priority, as it is

10

not perceived to be a management tool. This is exacerbated by heavy workload: peo-

ple would always first focus their efforts on what they find most important (for the

reasons described above it is not evaluation).

6. A vital reason for the low effectiveness of recommendation implementation is the fact

that evaluation is seen as a tool to control lower level institutions. This is first and

foremost due to the low level of involvement of administrative staff in the evaluation

study at the evaluation planning stage. As a result, there is a lack of understanding be-

tween the MA and the IA regarding what purpose evaluation might serve and so IA

staff often feel the evaluation is about checking on them, and the study fails to support

the implementation process or provide reliable information and becomes a kind of

a value judgment.

7. The effectiveness of recommendation implementation is also reduced by the fact there

is no formal system for such implementation neither at the level of Managing Author-

ity nor lower levels of the SOP HRD and EQUAL CIP implementation. Also, the divi-

sion of responsibilities between the MA and the IA is unclear as regards implementa-

tion of recommendations. That said, there appear some attempts at formalising rec-

ommendation implementation, a step in the right direction. They consist in developing

recommendation tables jointly by MA and IA staff, and then monitoring activities re-

lated to the implementation.

8. The reduction in the relevance and effectiveness of the implementation of recommen-

dations implementation has also something to do with the poor activity of the mem-

bers of Evaluation Steering Groups at the stage of planning and dissemination of re-

sults: e.g. such groups do not debate the purposefulness of individual recommenda-

tions.

9. The research process fails to take account of the evaluation of the results of the already

implemented recommendations. Consequently, neither decision-makers nor staff re-

ceive data regarding the purposefulness of evaluation and the use of evaluation results.

All this weakens their involvement in the evaluation process, from planning to the im-

plementation of recommendations.

10. Due to the absence of a system for monitoring the implementation of recommenda-

tions, the staff of the institutions involved in the implementation of the Programmes

not know which changes in the SOP HRD or EQUAL CIP result from recommenda-

11

tions following evaluation. And so the impression is amplified that the effectiveness of

recommendation implementation is low and that evaluation is not an effective tool to

manage a Programme. This brings about an unfavourable climate around evaluation as

an activity which will never enjoy the status of a priority as it is not important enough.

11. Another difficulty in implementing recommendations (financial constraints aside) is

the nature of many suggestions, related to political priorities (like an educational pol-

icy or a policy for combating unemployment).

12

Introduction

Acting on a fundamental recommendation included in this report we are hereby presenting

a brief document reporting on the evaluation of a system for implementing recommendations

by the institutions engaged in the implementation of the Sectoral Operational Programme

Human Resource Development and EQUAL Community Initiative Programme.

The study takes into consideration all the evaluation work carried out thus far at the Depart-

ment for European Social Fund Management as part of the Sectoral Operational Programme

Human Resource Development and EQUAL CIP. The study was carried out from 17 Septem-

ber to 31 October 2007.

The goals of the evaluation were as follows:

- assessment of the evaluation planning and implementation processes as for the rele-

vance of studies commissioned;

- assessment of the usefulness of the recommendations included in the evaluation re-

ports;

- assessment of the extent to which the recommendations from the reports are used by

the institutions involved in the implementation of the SOP HRD and EQUAL CIP;

- assessment of the process of disseminating the results of evaluation and recommenda-

tions;

- assessment of the existing system for implementing recommendations at the Depart-

ment for European Social Fund Management and identification of factors impacting

this process;

- formulation of recommendations aimed to develop a model of effective and efficient

implementation of recommendations for the Human Capital Operational Programme

2007-2013 and better control over the quality of evaluation studies (as for their rele-

vance and usefulness).

This report comprises five sections:

- an abstract with key results and recommendations;

- introduction including brief information on the study idea

- evaluation results, presented as per evaluation criteria of: relevance, usefulness and ef-

fectiveness; this section of the report also includes best practice and results of case

studies showing good practices and benchmarking;

- conclusions and recommendations;

13

- annexes, comprising descriptions of the evaluation systems in question in three EU

Member States as well as sheets related to analysis of evaluation performed as part of

the SOP HRD and EQUAL CIP.

We wish to thank all those who have spoken with us for their kindness and the time they de-

voted to the cause. Special thanks are due to staff members at the MRD Monitoring and

Evaluation Division Ms Joanna Hofman and Ms Paulina Tarsa for their assistance in making

our study work efficient.

Methodology During the study, we have analysed:

- policy papers;

- reports following evaluation studies (with special emphasis

placed on recommendations) and descriptions of objects of

contracts concerning evaluation studies, and

- Internet portals focusing on evaluation in Italy, Finland and

Lithuania.

We have conducted 25 interviews with:

- representatives of the institutions involved in the implemen-

tation of EQUAL CIP and the SOP HRD of all levels;

- members of the Monitoring Committee and Evaluation

Steering Groups;

- evaluators.

A more detailed analysis was performed of four SOP HRD and

EQUAL CIP evaluations:

- Improving staff skills and qualifications: study of training

projects as part of 2004-2006 SOP HRD Measure 2.3

scheme a);

- Assessment of the system for the management and imple-

mentation of the 2004-2006 SOP HRD;

- Current evaluation of EQUAL CIP, and

- Estimation-based assessment of the HC OP.

We have analysed systems for managing recommendations in three

document analysis

in-depth interviews

case studies

benchmarking

14

countries:

- Italy,

- Lithuania, and

- Finland.

15

Results

The results of the study are presented in three sections reflecting the three evaluation criteria.

Each section contains answers to the research questions posed in the draft evaluation.

Relevance of contracted studies Introduction

This section features a presentation of the evaluation results regarding:

- impact of the evaluation planning process on the relevance of the study;

- participation of stakeholders in the planning process, and

- methods of contractor selection.

Evaluation planning Regularity and flexibility

Under the SOP HRD, evaluation is envisaged every several years, annually and every couple

of months by the Department for European Social Fund Management at the Ministry of Re-

gional Development. Under EQUAL CIP, three1 evaluation jobs have been planned, one of

them being cyclical and covering the whole spectrum of issues examination of which was

deemed necessary.

Evaluation plans for several years ahead are approved by the Monitoring Committee. They

contain highly recommended horizontal themes and thematic evaluations. They take account

of the performance of obligatory evaluations (ex-ante, ex-post2) and, to a lesser or larger de-

gree, the needs of the institutions involved in the implementation of the Programmes: the

Monitoring Committee, Managing Authority, Intermediate Bodies, and Implementing Au-

thorities.

Based on such multi-annual plans, annual plans are developed, taking into consideration cur-

rent needs. If a plan provided for the performance of a specific evaluation and the Department

for ESF Management decided it was pointless at that time, the study was abandoned.

1 Selection process assessment, Assessment of Measure 1, Current assessment until 2008. 2 Due to the short implementation period the mid-term evaluation was not necessary or planned.

16

Good Polish practice

Evaluation planning for the whole Programme is made systematic, which facilitates better

evaluation management (securing resources for evaluation performance and planning staff

contribution), while ensuring flexibility which in turn takes account of research needs and so

has an impact on making the evaluation more relevant.

Apart from evaluations planned for several years and annually, ad hoc evaluations are per-

formed too, facilitating better adaptation to current needs. As the collected data suggest,

stakeholders see ad hoc evaluations as most useful.

Narrow circle of stakeholders

In practical terms, the diagnosis of evaluation-related needs is made by a narrow group of

staff at the European Social Fund Management at the Ministry of Regional Development. The

evaluation plan is developed as a result of discussions and agreements mainly within the MA

team. In consequence, research problems are related to the inquisitiveness and interests of

Department staff, their knowledge of the implementation of Programmes and difficulties ap-

pearing on the way.

One weakness of the process of planning specific evaluations is a narrow circle of stake-

holders involved in it. That the theme and scope of the evaluation are defined at the level of

Programme management leads to the fact that the evaluation barely meets the needs of lower-

ranking institutions. Such plans may be consulted with the Evaluation Steering Group and

sometimes with representatives of specific Managing Authorities yet, as shown by daily prac-

tice, their contribution is modest. This leads to the following question: Why is the stake-

holders’ participation in evaluation planning so limited? To answer this, let us have a closer

look at individual stakeholder groups: the Monitoring Committee, Evaluation Steering Group

and lower-ranking institutions.

MC and ESG operation

The EQUAL CIP and SOP HRD Monitoring Committees and Evaluation Steering Groups are

not active in voicing problems to be examined by evaluation.

The reasons are as follows:

- weak representation of various groups in the SOP HRD MC, due to which research

needs vital for individual social groups are not expressed;

17

- lack of vision on the part of the Monitoring Committees, focusing on daily formalities

rather than a long-term policy of improving implementation quality;

- changes in the composition of the Committees (turnover of persons representing indi-

vidual institutions/entities);

- lack of a system where work is done through working groups makes it difficult for ac-

tual discussion between MC members to take place;

- most members of Monitoring Committees have insufficient knowledge of evaluation,

why it should be performed or how the MA uses available evaluation results;

- members of the Evaluation Steering Groups do not treat evaluation as a management

tool and so evaluation is not a priority to them; that is why evaluation ‘loses out’

against other duties which are seen as important and urgent;

- members of the Groups are both people in managerial positions and regular staff

members; the former often devoting their time to other tasks which are, in their view,

more important than evaluation; in turn, the latter are not decision-makers, and

- the participation of experts in the Groups is meagre: in principle they should not be

persons seeking a contract related to evaluating the Programmes; this restriction may

be correct from the ethical point of view, yet it significantly decreases the potential of

the Groups as regards the creative attitude towards planned evaluations.

IB and IA activity

Levels of involvement of representatives of individual institutions implementing the SOP

HRD and EQUAL CIP vary: during the examination we have identified lower-ranking institu-

tions which were not interested in the evaluation requested by the MA, even if its subject mat-

ter overlapped with the area of that institution’s activity. The following are the reasons:

- perceiving evaluation as a tool for control rather than one supporting the implementa-

tion of a Programme;

- low levels of trust shown by the staff of the institutions towards evaluation studies;

- IA or IB staff lacking the sense of being able to really influence the shape of the

evaluation, and

- IB and IA staff lacking the sense of being able to influence the system for managing

and implementing the Programmes; as a result they focus on acting on guidelines

rather than making a creative contribution to the implementation of the Programmes.

- Our interlocutors have questioned the relevance of the following studies:

18

- assessment of the system for the management and implementation of the 2004-2006

HRD SOP: that study was seen as not bringing enough new information by the Man-

aging Authority as well as not useful enough by lower-ranking institutions, and

- study of final beneficiaries of assistance granted under the 2004-2006 SOP HRD six

months after they finished their participation in the project: this study, a combination

of monitoring and evaluation, has not so far brought any conclusions that might be

used in practice, for instance in selecting projects for implementation.

Using results of past studies

When evaluation is planned under the HRD SOP, the results from monitoring of projects con-

ducted by the Implementing Authorities are not given enough attention. As a result, the results

of the evaluation tend to have a low novelty value for the IAs.

Case study

Evaluation of Improving staff skills and qualifications: study of training projects as part

of Measure 2.3 scheme a) SOP HRD 2004-2006

The scope of the study partially overlapped with the scope of monitoring and evaluation con-

ducted by the PARP (through its own study by questionnaire and analysis of implementation

reports). Given the above, the results of the study for the IA were mainly the confirmation of

previous conclusions. However, the substantive complementarity of schemes a) and b) im-

plemented under Measure 2.3 was not examined, which would be recommended from the

IA’s viewpoint. Still, the IA did not voice that need at the evaluation planning stage, al-

though its representatives attended the meetings during which the scope of study was

planned. The contracting party, in turn, did not decide to examine the complementarity of the

schemes because of organisation - and methodology-related reasons (such an examination

would have entailed including in the study the very same respondent groups, which would

have made the performance of the evaluation significantly more difficult).

Case study

Evaluation planning in Finland

A decision concerning the performance of specific non-obligatory evaluation does not result

from the process of programme implementation, but rather from recommendations made by

evaluators addressed to the Ministry of Labour. The evaluators indicate some issues which

might be subject to analysis. The suggestions are followed if the idea is of interest to the

managerial staff of the institution.

19

Contractor selection procedures

Study conceptualisation

The conceptual outline of study is to a large extent developed by the contracting party and

does not give the researchers much freedom. A clear trend can be seen of making questions

and methodological guidelines more detailed. This becomes typical of institutions with solid

evaluation units which are not just the contracting party but also a strategic team pursuing an

evaluation policy (the Department for ESF Management at the MRD where Monitoring and

Evaluation Division operates; the PARP, with a robust evaluation unit also doing its own re-

search work). Making evaluation outlines more detailed is also related to public procurement

procedures.

Detailed precision in data included in the ToR makes sense if:

- there is little time for study performance: precision on the part of the contracting par-

ties saves precious time and lets the evaluators move quickly to the implementation

phase (as was the case, for instance, with the study of Measure 2.3 scheme a), and

- the contractor’s task is to provide very specific data (e.g. on indicator values).

Drawbacks of detailed provisions of the ToR are as follows:

- there is less opportunity for the contractors to be inventive and contribute on the sub-

ject; when the ToR defines research questions and methods in detail, the evaluators

have less to contribute into the bid; as a result, the technical capacity related to study

performance in a short period of time and the price have an increasing impact upon the

selection;

- when the contracting party has developed key questions in great detail, there is a risk

that the results of the evaluation fail to have a novelty value for the contracting party,

since they focus on issues that have already been identified and the evaluation is

merely a deepening of that knowledge; when ToR is more open, the evaluators stand

a chance of discovering something the contracting party was unaware of; this draw-

back is even graver when the object of the evaluation requires a non-routine approach

or when there have been no experiences to which one can refer to while developing

the conceptual outline of the study, and

20

- in actual fact, the assessment of the bids submitted by the evaluators boils down to the

price and the method selection criterion is of less importance; it is so firstly because by

specifying some methods in the DOC the contracting party points to a direction of the

methodology: selecting some other research methods must be subjected to the DOC

requirements; thus, the evaluator’s influence on the final study methodological outline

is only partial; another reason is that adding methods to the minimum specified in the

DOC brings the costs of the study up, and so the evaluator may win points during

methodological assessment only to lose them later when the price criterion is assessed;

the methodological criterion is acutely limited to price assessment when sample size is

a vital factor subject to appraisal; in such a case points are awarded to those contrac-

tors who do more (interviews, questionnaires) at a lower cost; also, expanding the

sample is not always justified from the methodological viewpoint.

Contribution of evaluators

Evaluations where contractors performing them are selected in an open tender specify the

research scope in great detail. The contracting party asks key questions and specifies the

methodological minimum in detail.

The contractors are requested to present their own questions and suggest any additional re-

search methods. In practice, however, they very rarely go beyond the concept defined by the

contracting party. Equally rarely do researchers add anything into the methodology report.

The contractors do not significantly step outside the conceptual outline included in the ToR

and DOC, as confirmed by the analysis of methodological reports we have performed (table

1).

This is because:

- there is a limited time for the performance of the study: because research schedules are

usually tight, extending the study by extra research questions or using additional

methods would entail a risk of not meeting the deadlines set by the contracting party;

- the bid selection criteria do not encourage contractors to expand the conceptual outline

by extra questions and methods as the price has a large impact (mostly around 40-

50%) on the bid selection; although the methodology criterion is applied, it also often

boils down to the price factor (as said above); from the point of view of the research-

ers it is not rational to expand the research field or suggest a wider range of research

21

methods since this increases the cost of the study; when the price is the primary crite-

rion for bid assessment researchers avoid doing so, and

- the period of time in which to submit bids is usually short: 2-3 weeks, which is in

compliance with the Public Procurement Act (defining the time minimum), yet due to

the fact the contractors do not have enough time to study the object of the evaluation

in-depth or develop an offer which would significantly enrich the conceptual outline

described by the contracting party.

Case study

Ex-ante evaluation of the HC OP

The evaluation report contains information that the limited time for conducting the study

necessitated the use of a limited set of research tools and resulted in lack of detailed answers

to all the evaluation questions.

22

Table 1. Suggestions from evaluators concerning the contents of detailed descriptions of the

object of the contract

Evaluation studies subject to

analysis

How many new questions are

there in the methodology report

(MR)

as compared with the DOC?

Do they significantly extend the

thematic scope of the study or

are they merely more detailed

versions of the questions featur-

ing in the DOC?

Current evaluation of EQUAL

CIP

20 The MR gives a structure to the

goals and assumptions of the study

defined in the DOC: it introduces

three evaluation criteria, three

analysis levels and two study

methods. Such three dimensions

make up an assessment model

where the questions featuring in

the DOC fit in. The new questions

appearing in the MR are more

detailed versions of the questions

from the DOC or they stem from

the need to approach the issue at

various levels of analysis.

Improving staff skills and quali-

fications: study of training pro-

jects as part of Measure 2.3

scheme a) SOP HRD2004-2006

Around 15 (In the MR, research

questions from the DOC were

included only in interview scripts.

Nowhere in the MR are research

questions related to the methodol-

ogy).

The MR lacks a link between re-

search questions and methods of

study performance. The report only

gives suggested research methods

and sample selection criteria. The

research questions appearing in the

DOC can be found only in inter-

view scripts annexed to the MR (in

most cases questions are simply

copied and identical to those in the

DOC).

Additional issues presented in the

MR are an extension of the theme

of the system for implementing

Measure 2.3 by reporting and pay-

ment settlement. The MR also

includes questions about assessing

23

Measure 2.3 as regards mutual co-

operation between individual enti-

ties involved in the Programme.

Assessment of the system for the

management and implementa-

tion of the 2004-2006 HRD SOP

Research problems in the MR are

consistent with the DOC. The

DOC includes a set of research

problems: guidelines defining the

evaluation scope.

In the MR research problems have

been made operational in the form

of research questions.

The MR includes all the research

problems whilst the research ques-

tions given in the MR do not cover

all the themes from the research

problems.

Estimation-based assessment of

the HC OP

No new questions appear.

All the questions in the DOC are to

be fund in the MR. The MR brings

order to research questions based

on the following evaluation crite-

ria: relevance, coherence and effec-

tiveness.

Case study

Selection of contractors

Italy: The key criterion for selecting external evaluators is experience and expertise of the

research team, the price considered later.

Finland: Most typically applied selection criteria are: understanding of the institutional setup

to which a given evaluation job is related and knowledge of the ESF (checked by questions

in the competition application form: evaluation to what extent a given institution is familiar

with how the ESF operates). Additional criteria are experience in conducting evaluation as

well as the company’s reliability and financial standing. For each criterion the contractor is

awarded points separately and then the aggregate number of points determines a ranking list

of candidates for evaluation performance. While examining the bids from evaluators also the

cost of conducting the evaluation is taken into consideration, yet it is analysed for the corre-

spondence between the costs and planned activities.

24

Evaluation management

Co-operation with contractor

The relevance of the evaluation study is influenced by relations between all the parties in-

volved in it. They should find an agreement and the evaluators should learn from the contract-

ing party at the very start what the final goal of the study is and what purpose it is supposed to

serve. The basis of such an agreement should be dialogue and mutual openness of the parties

as well as clear communication of any difficulties on the way since such an attitude will en-

sure the good quality of the conducted study, which in turn will translate into the results and

suggested recommendations. The more relevant, unambiguous and specific they are, the more

practicable and useful for the addressees they will be.

The contractor selection made in an open tender has a serious limitation: no direct personal

contact between the contracting party and the bidder. This restriction stems from the Public

Procurement Act, yet it entails the risk that a contractor will be selected with whom the con-

tracting party may experience communication problems and another that each of the parties

may have a different understanding of the formulations in the bid. This has not been the case

so far, yet we are pointing that out as a potential risk.

An important moment in the process of developing an evaluation is meetings with the con-

tractors in the period between when the contract is signed and the study launched, which is

practised for evaluations requested by the MA. Dialogue between the contracting party and

the contractor at the stage when the methodology report is being drafted is a good opportunity

to introduce more precision into the conceptual study outline in great detail. This is also an

opportunity for the evaluator to better understand the contracting party’s needs, a fact boost-

ing the relevance of the study. Still, the bid on the basis of which the contractor was selected

is a limitation.

Monitoring

Monitoring of conducted evaluation studies performed by the contracting party guarantees the

appropriate quality of the examination as well as facilitates the adaptation of the evaluation to

the actual needs and expectations of the institution contracting the work, which in the long run

has an impact on the acceptance or rejection of the recommendations to be presented in the

evaluation report. The awareness that the research process is being watched makes it easier

for the contracting institution to identify itself with the suggested recommendations and as

25

a result with the acceptance of the recommendations and their further implementation. Then

they are not an abstract set of guidelines and tips but rather natural conclusions coming to

light over the course of the study. It is of particular importance in the context of the later im-

plementation of evaluation recommendations.

Various methods are used for monitoring and controlling the quality of the study while it is in

progress. In the case of larger studies, more complex and performed under time pressure, in-

terim reports are a handy tool, relating to narrower issues or closing individual stages of the

study. In other cases, the monitoring methods are most influenced by the working style of the

evaluators themselves. If they provide information on the progress of the study themselves

and engage in consultations to dispel their doubts on a regular basis, they satisfy the funda-

mental need of the contracting party, the sense of security as regards the success of the whole

project. Such evaluation monitoring methods are not mutually exclusive but typically applied

jointly.

The system currently in operation fails to involve other institutions with a potential interest in

study results, apart from the contracting party, in evaluation monitoring. In consequence, only

the contracting party is informed of the course of the evaluation and preliminary results dur-

ing the evaluative exercise and may contact the evaluators about its needs and comments. The

other stakeholders are not sufficiently engaged in the process. It would be recommended that

the key stakeholders (the institutions in charge of activities subject to evaluation) attend some

meetings with the evaluators, particularly when preliminary results of the study are discussed.

The key stakeholders should also voice their opinion on the reports shown to them, at all the

project stages.

Case study

A system for evaluation implementation monitoring in Italy

Monitoring of the processes and results of evaluations of Operational Programmes carried

out in Italy is a task of the National Evaluation Group which operates at the Department of

Developmental Policies at the Ministry of Economic Development. The Group’s goal is to

see to it that evaluations are performed using appropriate resources, as professionally as pos-

sible and in consequence bringing correct results, that is the results should be made objective,

content-related conclusions justified and recommendations as much relevant as possible, thus

useful for those implementing the project. The Group is responsible for making sure that the

26

evaluation process is of good quality, primarily as regards participation of all the parties in

defining evaluation questions, data accessibility and the methodology used in evaluation

studies.

The members of the Group hold regular meetings with evaluators, Evaluation Units and

Management Units during which current issues are taken up related to studies in progress.

Representatives of each of the parties share information and doubts as well as present prob-

lems which appeared during study performance or can voice new suggestions not publicised

before. The point is to offer all the stakeholders a real insight into what is going on over the

course of the evaluation study and an opportunity to become involved in the process. This

aims at ensuring a good quality of the conducted studies. Decisions are made on the basis of

mutual agreement and jointly achieving best solutions. To illustrate the case, we are present-

ing an example modification which was suggested by evaluators during one such meeting. It

concerned a correction of research tools as it turned out that the practice of beneficiaries

sending in questionnaires had been partially abandoned since the rate of returned copies was

too low to draw meaningful conclusions. The evaluators presented their doubts before the

Group and it was agreed that questionnaires should be abandoned and replaced by direct in-

terviews.

The Group also makes sure that evaluators have free access to all necessary documents they

need while conducting the study so that it proceeds unhindered. Past experiences included

situations where work of the evaluation team was halted or made impossible due to, for in-

stance, missing telephone numbers or other data necessary for contacting beneficiaries. To

avoid this, the Group officially asks relevant institutions to prepare data in advance and make

them available.

The Group also carries out meta-evaluative activities consisting in:

- analysing evaluation reports;

- direct observation of the evaluation process (to provide feedback to the involved par-

ties when the process is still in progress), and

- holding systematic meetings with evaluators, Evaluation Units and Management

Units.

The Group collects some information on its own by conducting relevant examinations and

some necessary information is provided directly by Management Units, which send in re-

quired documents. The Group reports directly to the Monitoring Committee, for which it

27

drafts and presents reports. The Monitoring Committee, a unit in charge of overseeing the

implementation of EU operational programmes, also supervises the work of the evaluation

units.

Additionally, a working team operates as part of the Institute for Development of Staff Edu-

cation (ISFOL) at the Ministry of Labour focusing on monitoring of conducted evaluations,

which directly oversees the course of the studies in progress. Its objective is to supervise the

work progress, control the work schedule, introduce possible modifications to the project and

ensure the quality of the study at all the stages of its implementation. The Group meets not

less frequently than every two months. It is a very good solution as the parties have an oppor-

tunity to express their doubts and questions as well as to modify certain research points so

that the study can be as relevant and useful for the parties contracting the evaluation as pos-

sible. Then these are consulted with other participants and following discussion approved or

rejected.

Implementation time

Date selection

Evaluation relevance is conditioned by the time when the study is carried out. Too early study

performance, for example, makes it impossible to collect data and draw conclusions on their

basis. In turn, if an evaluation is carried out too late, its results cannot be used since the deci-

sions have already been made.

Lacking data also means problems with evaluation of projects not yet completed as well as

conducting the study during summer months. It also happens that the very same beneficiaries

are examined simultaneously by several evaluators, which discourages the respondents from

participating in the study and causes data collection problems.

Case study

Ex-ante evaluation of the HC OP

The analysis focused on documents which were being developed and subject to consultation

by social partners. The evaluator was to analyse a document in the course of its development

and the programming team was supposed to take note of the evaluator’s conclusions and rec-

ommendations as they appeared. In practice, it proved difficult as the evaluator’s work on

28

a changing document with little time for making pauses in its development led to a vicious

circle barring reflection and following recommendations.

It should be noted, however, that the study performance time results also from reasons inde-

pendent of the contracting party. These are mainly dates when tendering processes end (which

cannot be precisely predicted) and delays in evaluation performance. These difficulties some-

times cause backlogs due to evaluation reports coming all at once to the Monitoring and

Evaluation Division at the MRD. Over a short period of time, a team of staff from the con-

tracting institution must read and correct them.

Time pressure

When study implementation time is short, there is no safety margin left for unexpected (or

expected) problems. In the case of evaluating ESF projects, a typical problem is obtaining

data concerning final beneficiaries. This is due to two factors: dispersion of databases and

necessity to sign contracts and agreements with Implementing Authorities, 21 in total. Con-

tracts with the Implementing Authorities contain just general provisions concerning the per-

formance of evaluation yet no regulations of detailed issues of making data accessible and co-

operation with evaluators. An attempt at concluding detailed provisions covering such issues

might lead to the institutions making data available stiffening their attitude. The recom-

mended path is to move towards engaging staff of the Implementing Authorities in the evalua-

tion planning process so that they identify with them and see their importance.

Case study

Improving staff skills and qualifications: study of training projects as part of 2004-2006

SOP HRD Measure 2.3 scheme a)

Incomplete databases and information recorded in a format making its use impossible re-

sulted in the failure to attain all the research goals presented in the methodology report.

Time pressure has a negative impact also on co-operation between evaluators and staff of the

institutions involved. The tight schedule of the study, coupled with difficulties with obtaining

data from Implementing Authorities, results in the evaluators requesting large volumes of

documents, regardless of whether they will be useful for them or not. In this way they make

provisions for when during the study a need appears to analyse them: then they will not have

to wait for them risking delay in returning the study report. Meanwhile, the staff of Imple-

29

menting Authorities have more work to do in a short time; they also know that they work is

not always used, which leads to frustration and creates an unfavourable climate around the

study. The tight schedule of the study results in a poorer quality of study reports. Evaluators

emphasise that sometimes they submit poorly drafted reports.

Systemic solution

Evaluation potential at regional level

Evaluation potential at the regional level is weak: there are only few people at Voivodship

Marshal Offices competent in the field of evaluation enough to accurately diagnose the needs

and manage evaluation at all the implementation stages. Equally poor is the potential of local

evaluators: there are few robust evaluation firms.

Currently, division of responsibilities between the institutions involved in the Programme

implementation at various levels is far from clear as regards evaluation. The evaluation work

performed by the MA covers both the entire Programme and individual Measures or even

schemes. The consequences are as follows:

- evaluation is seen as imposed ‘from above’ as a form of control;

- reports include recommendations which the contracting party is unable to carry out as

they are addressed to other institutions (at a different level implementation); if that is

the case, the role of the contracting party is to initiate discussion on opportunities to

implement them (rather than to directly implement them), and

- results of the project monitoring conducted by IAs are not given enough attention.

Managing evaluation at institutional level

Currently, all the tasks related to evaluation planning and management are performed by the

Monitoring and Evaluation Division.

Good Polish practice

Good practice is, for instance, the creation of a separate unit in charge of evaluation within

institutions which are involved in programme implementation. Such a solution lends impor-

tance to evaluation as well as facilitates staff professionalism. The quality of evaluation, per-

formed by the institution itself or outsourced, is also helped by the creation of evaluation

30

units at the Department for ESF Management at the MRD and the PARP, manned with

staff who know the subject well and are highly involved in evaluation performance. Notably,

the evaluation unit at the PARP also performs educational activities targeting staff of the en-

tire institution and aiming at showing why evaluation makes sense, how its results may be

used and how it can be conducted. Educational activities and including a wide group of

stakeholders in the evaluation planning process make PARP staff open to evaluation results.

Similarly, the management of the Department for ESF Management at the Ministry of Re-

gional Development is open to evaluation results. The examples of the MRD and PARP

show that the key factor is a positive attitude of superiors towards evaluation as well as their

appreciation of its importance.

Closing brief

Factors influencing evaluation relevance

The reduced relevance of evaluation as regards the needs of the institutions engaged in the

implementation of EQUAL CIP and the SOP HRD is a consequence of the practised method

of evaluation planning, in particular:

- the way in which research needs are diagnosed;

- not all stakeholders participating in the evaluation planning process;

- practised contractor selection procedure and contribution of evaluators into the final

study outline, and

- time of study implementation.

Good communication between the contracting party and evaluators improves the relevance of

the study.

31

Usefulness of recommendations

Introduction

This section describes evaluation results as regards:

- views on the usefulness of evaluation for the institutions involved in the implementa-

tion of programmes covered by the study, and

- dissemination of results and the form of recommendations.

Institutional usefulness of evaluation

Recommendation usefulness

The usefulness of evaluation conducted by the MA for lower-ranking institutions is linked to

the fact that when there are no other sources of information staff analyse reports looking at

their own needs. Also, reports are a methodological inspiration for planning one’s own re-

search. Results of evaluation studies are useful insofar as staff can use them later during pub-

lic presentations or conferences. They are also a basis for debate in various ministries. They

are then a ‘paper base’, another stimulus to start changing things and a discussion subject.

Still, some recommendations included in evaluation reports are seen by staff of the institu-

tions as useless and impossible to carry out since the solutions on offer do not fit within the

structure of administrative work. The question then arises whether evaluation should perhaps

go beyond set routines. In our view, also such recommendations are necessary as they point

out directions for future development and modifications of the existing structures. However,

the evaluators should make it clear in the reports which recommendations go beyond the ex-

isting regulatory environment, so as not to frustrate the recipients by suggesting changes im-

possible to make over a short period of time.

Case study

Recommendation implementation in Finland

An evaluator concluded that small NGOs found it difficult to participate in projects funded

from the ESF as a large own contribution is required, as is previous experience in implement-

ing large projects. The evaluator suggested that a mechanism be introduced would facilitate

access for small NGOs to ESF funding. His recommendation was followed. An umbrella

project was set up involving as partners nationwide organisations from various sections of

32

the country. Their role was to administer funding and transfer it to small organisations which

were directly responsible for the implementation of activities. In this way the problem was

resolved.

Dissemination of evaluation results

Evaluation recipients

A vital element of each evaluation study is making the results of performed evaluation avail-

able and disseminating them. The member states implementing EU funded projects are

obliged by the European Commission to disseminate and publish evaluation results not just

among directly interested parties (the institutions contracting the evaluation) but also the gen-

eral public. Such activities are to serve knowledge dissemination, promotion of the good prac-

tice developed during the implementation of a project, as well as improvement of general

knowledge of those who in one way or another may be interested in the results, not just direct

beneficiaries of a given project. The point is to provide free access to study results for anyone

who is interested.

The goal is primarily to make the general public sensitive to the issues taken up in the context

of specific projects and encourage the information recipients towards various activities in

which they could participate in the future.

Case study

Dissemination of evaluation results in Italy

In Italy, there is no single procedure for dissemination of evaluation results, all depending on

the intentions of the parties contracting and performing such a study. Sometimes (depending

on what the project is about and who may be particularly interested in the results of such

a study) meetings are held in institutions locally, ‘in the field’, where the project was directly

carried out, and sometimes these are larger-scope meetings: at universities, in research units,

at ministries which commissioned the study etc. It is important that such meetings be well

adapted to the target groups: they must send a transparent message so as not to discourage

the recipients but to make the project attractive to them. To that end, several versions of re-

ports are drafted, for example. The full version goes to the management unit and evaluation

unit, while the other reports are presented in abbreviated versions targeting the general public

(although not always). As part of dissemination of results, some documents are published on

33

the websites of individual institutions, while the whole documentation is generally available

for consultation on the spot, i.e. in internal libraries where one can go and study the study

results in detail.

Communication channels

There is no single procedure for dissemination of evaluation results: information channels are

selected depending on the study. Reports are made public through sending them by electronic

mail to staff of the institutions involved in Programme implementation as well as publishing

them on websites and in a paper version. Despite such solutions, staff of the institutions im-

plementing the SOP HRD and EQUAL do not know which evaluations have been conducted

or their results. They also find it difficult to give examples of recommendations.

There is no operative system for flow of information concerning evaluations in progress or

already completed by an individual institution engaged in Programme implementation. In

theory, the Evaluation Steering Group is supposed to make such information flow. In practice,

however, it does not work because its members give priority to other duties.

Abbreviated versions of reports are not written – unlike, for example, in Italy –addressed to a

wider readership.

Good Polish practice

→ Organising seminars featuring presentation and discussion of study results and recom-

mendations (e.g. results of the evaluation ‘Measure 2.3, Scheme a) of the SOP HRD’ or

‘Measure 2.3, Scheme b) of the SOP HRD’);

→ Publishing reports on websites;

→ Publishing reports in a paper version and sending them to the institutions engaged in Pro-

gramme implementation;

→ Developing ‘recommendation implementation tables’, bringing together recommendations

which the institution wished to act on, a description of tasks to complete and a relevant

schedule.

34

MC’s knowledge of recommendations

The Monitoring Committee members do not receive results of all the evaluations. They also

do not have a full knowledge of which evaluations have been performed or are currently in

progress. When the MC meets, there is little time for becoming familiar with evaluation re-

sults, only a very short time is given for presentation of results, materials related to the results

are too bulky and the MC members have no time to read them. As a result no debate is held as

regards the results and recommendations in the context of making use of them.

Report form

Evaluation reports concerning the SOP HRD and EQUAL CIP vary in form. Not all of them

include a summary, sometimes the evaluators choose to offer an introduction or a brief on

results instead. The section on results is most frequently linked with recommendations: in the

form of a table or as the conclusion-recommendation-conclusion-recommendation pattern.

Conclusions and recommendations are sometimes placed after subsequent sections of analysis

of the results. In three cases conclusions with recommendations take tens of pages and in one

case recommendations make the largest section (17 pages). In the other cases recommenda-

tions take from one page to three; conclusions typically take slightly longer to present, yet not

more than ten pages.

Case study

Assessment of the system for the management and implementation of the 2004-2006

HRD SOP

Excluding attachments, the report is 423 pages long, its summary 39 pages long and conclu-

sions with recommendations 42 take pages. The recommendations are not indicated in the

report but rather ‘woven into’ the conclusions, which makes them difficult to find. As not

every conclusion is accompanied by recommendations, finding the latter requires browsing

through over 40 pages in order to ‘fish’ for them in the text. Most of our interlocutors re-

membered this study primarily because of the high number of pages impossible to read.

Contents of recommendations

Staff of the institutions involved in the implementation of the Programmes tend to think that

one single drawback of recommendations is the fact of their being too general, which is also

a downside they point out as hampering recommendation implementation. However, detailed

35

analysis of the contents of the recommendations included in the reports on evaluation-

research cases indicates that a vast majority of the recommendations do include a detailed

description of how to attain the desired state. Only in few cases are the recommendations gen-

eral or conclusions merely point out that a problem exists yet no solution for problem resolu-

tion is given. Frequently, however, the institutional addressee is not specified in the recom-

mendations (table 2).

Table 2. Contents of recommendations.

Current evaluation

of EQUAL CIP

Improving staff skills

and qualifications:

study of training pro-

jects as part of 2004-

2006 SOP HRD Meas-

ure 2.3 scheme a)

Assessment of the sys-

tem for the manage-

ment and implementa-

tion of the 2004-2006

HRD SOP

Estimation-based as-

sessment of

the HC OP

Do the recommendations include a description of the desired state?

YES YES YES YES

Do the recommendations include a description of how to attain the desired state?

A vast majority of them

YES. Only in one case is

a recommendation a set

of very general guide-

lines.

YES, the final recom-

mendations contain

information on what

measures to take to im-

plement projects under

Measure 2.3 scheme a)

more effectively and

more efficiently.

However, as for some

recommendations the

evaluators take a highly

general approach: e.g.

‘developing special pro-

cedures for assessing re-

submitted applications,’

‘ensuring competent and

efficient advisory and

information support.’

More detailed recom-

mendations are ‘blurred’

due to the fact that the

YES. A vast majority of

recommendations contain

a detailed description

of attaining the desired

state. In few cases the

recommendations are

general or the conclu-

sions merely state there is

a problem without sug-

gesting a solution.

YES, but not all: First,

there is a list of

detailed

recommendations

related to individual re-

search criteria, followed

by a section containing

an opinion of the evalua-

tor on the possibility of

implementing selected

recommendations.

36

addressees have not been

clearly defined.

Are the entities to whom the recommendations are addressed indicated?

For most recommenda-

tions, their addressee(s)

can be easily identified.

Such information is miss-

ing in two cases.

NO – no information as

to who is supposed to

take given actions im-

proving the process.

Mostly NO.

The complex structure of

management in the case

of the HRD SPO as well

as a large number of

entities involved in the

management call for a

clear definition of who is

the addressee of the rec-

ommendations.

In few cases the recom-

mendations relate to

procedures or legal solu-

tions applicable in Po-

land.

Given the nature of the

evaluation, the recom-

mendations are related to

the Programme draft and

they address its authors.

Factors influencing recommendation usefulness

Institutional involvement in evaluation planning

The staff of implementing authorities find evaluations performed as requested by the MRD as

the Managing Authority needed. However, they do not answer questions which are of interest

to the Implementing Authorities. The staff of Implementing Authorities tend to think that the

knowledge gained from those evaluations is largely useless at the implementation level where

they operate. They must try very hard to find something relevant for themselves in the re-

ports.

The Implementing Authorities focus on more detailed information, e.g. what should training

profiles be like (particularly as regards individual regions); signalling potential implementa-

tion problems so as to allow fast allocation of resources between individual activities when

there is no interest in a given activity; identification of needs of the final beneficiaries; deter-

mination of the values of target indicators; how to effectively promote a programme and reach

potential beneficiaries; barriers faced by potential beneficiaries, and implementation difficul-

ties.

37

Case study

In Italy, there operates a ‘network of evaluation and verification units’ (NUVV) which brings

together all the central-level and regional Evaluation Units forming an institutional collabora-

tion network. Its operation consists in providing technical support, evaluation, monitoring

and verification of draft public investment projects undertaken by the government admini-

stration. The network serves the purpose of exchanging documents, information and experi-

ences as well as is a forum for experts. Currently, the network includes 33 units (employing

a total of circa 400 people) operating at twelve central-level institutions, in regions and

autonomous provinces. The network is managed by a General Conference made up of repre-

sentatives of each of the 33 units. In actual fact, the evaluation units are not entirely inde-

pendent as they are anchored in the administrative structure of Management Units. Their par-

tial autonomy has been secured, however, by employing people from outside of public ad-

ministration who are highly competent and experienced as regards evaluation3.

Lack of feedback

Evaluators do not receive information on what happens to the recommendations they have

made. As a result, the recommendation definition process is hardly a learning experience for

them and leads to the repetition in subsequent reports of the same recommendations which,

although rightful and justified, are not carried out due to formal restrictions.

Factors influencing recommendation implementation

Nature of evaluation

Modest-scope ad hoc evaluations are more useful, as they are an answer to a specific problem

and yield fast results. One of the most useful studies mentioned has been one concerning the

PESF system because it responded to the actual needs problems; it was conducted fast (which

was possible because its scale was relatively small); the recommendations were very practical

and specific; they were also made at a time when they could be used.

Whether recommendations are going to be implemented is largely decided by the nature of

the evaluation: if it is transversal and involves many different institutions it will be definitely

harder to carry out. It may happen that an institution considers a given recommendation to be

right, yet its implementation entails extra costs: it may be time-consuming and labour- 3 More information on the Italian evaluation system can be found in the analysis presented in the annexes.

38

intensive, which leads to the non-performance of some activities. Sometimes recommenda-

tions are not implemented which are not deemed cost-effective or whose implementation

costs outweigh the benefits. Also, for recommendations to be followed effectively, they must

be specific and truly useful; ideally, they should inspire, make people reflective and imple-

mentation-ready. A respondent representing the MRD has said that most evaluation studies

inspire them and when it comes to making a full use of recommendations they carry out

around 30-40% of them.

Implementation time

The poor usefulness of the evaluation has been also a result of a brief implementation time.

Before the Programmes were ‘properly launched’ and before they could be evaluated it had

turned out that it ‘did not pay’ to implement changes anymore as implementing them would

require effort, not to be offset by short-term benefits.

Evaluation planning method

Evaluation usefulness is reduced by a long time needed for evaluation planning (drafting de-

tailed ToR takes a lot of time), contractor selection, study performance and elaboration of

results. The direct usefulness of study results at the level of operational management is poor

(the more so as the 2004-2006 implementation period was characterised by high changeability

of guidelines). The problem concerned mainly large studies as the time needed for data collec-

tion and analysis was longer whilst conclusions and recommendations were becoming obso-

lete (which was the case e.g. for the study of EQUAL CIP).

Attitude of decision-makers and staff

The usefulness of the evaluation depends first and foremost on the attitude of the institution

which is the recipient of the study results: whether it is interested in the evaluation results or

not; whether it treats evaluation as a sideline and procedural activity or rather as a develop-

ment tool; whether it has a team of competent individuals able to select a good external

evaluator and be a conversation partner for him/her as well as whether they are able to make

the evaluator deliver a study of the desired quality.

Good international practice

In 2006, ex-ante evaluation of four 2007-2013 operational programmes was conducted in

Lithuania. The evaluation report included 27 general and over 180 specific recommenda-

39

tions. When policy papers were being drafted, two-thirds of the recommendations were taken

into consideration.

As part of evaluation discussions were held with chief programme authors, mainly ministries,

and regular presentations in the Evaluation Steering Group as well as committees in charge

of the preparation of a strategy for EU structural funds for 2007-2013. The regular seminars

were an opportunity for interactive discussion and development of an operational programme

whilst ensuring that the recommendations were carried out as much as possible.

Evaluation is useful only when its recipients are open towards its results, which is a conse-

quence of the institution’s organisational culture on the one hand and the way the very study

is conducted on the other. We have seen a case when the evaluation results were rejected be-

cause the institution staff felt an accompanying sense of threat, stemming from lack of ano-

nymity and a largely diverse interpretation of results. This was the case because:

- the staff received a questionnaire to fill out, a task defined as obligatory by their supe-

riors; the questionnaire included some sensitive questions (for instance, attitude to-

wards managers); the process was email-based, so the respondents were expected to

send the completed questionnaire from their own email account, which restricted their

anonymity;

- although the questionnaire was voluntary in principle, the management reprimanded

those who have not completed it, and

- the IA management received a staff ranking list drafted on the basis of the evaluation

of individual projects; in the report such data were presented in an aggregated form,

while the management had access to detailed data concerning the assessment of indi-

vidual staff members, who did not know the results but were reprimanded during

a meeting with the management for their poor performance.

MC role

In the context of discussing the use of evaluation results and recommendation implementa-

tion, the role of Monitoring Committees should be remembered. MC members have a poor

knowledge and evaluation-related awareness, do not understand the point of evaluation very

much and at Committee sittings devote little time to analysis of evaluation results. Such meet-

40

ings are not conducive to having any in-depth debate since the body is too formalised and

heavy on red tape.

For the MC to be able to actually use evaluation, it should become a forum for discussing

issued relates to the evaluation of individual programmes as well as conclusions and recom-

mendations together with ways of their implementation etc. To make it happen, the members

should first have a solid knowledge concerning evaluation. They should be trained in evalua-

tion as well as shown good practice from Poland and other European countries so that they

believe evaluation makes sense. Also, a formal procedure would be useful for dealing with

recommendations. The MC might, for example, set a standard for its work and assume that

each evaluation should be followed by a detailed response from the institution it concerns.

Quality of recommendations

The usefulness of recommendations is reduced by the evaluators’ poor knowledge of:

- the programme and legal framework regulating its implementation, and

- specific features of the groups which are Programme beneficiaries.

For those reasons recommendations tend to be highly general. In most cases recommenda-

tions do include a description of the desired state but do not indicate how to achieve it.

Also, the evaluators have little time in which to conduct the study. We have analysed the time

that was needed to conduct studies as part of the HRD SOP. Three project types could be

identified: 1) short-term, taking around 7-10 weeks, 2) mid-term, typically lasting from 16 to

20 weeks and 3) long-term, taking more than 24 weeks to complete. Given the fact that the

first week or even two weeks of the study should be devoted to making arrangements with the

contracting party and the preparation of research tools whilst the last week or two focus on

report-related consultations, not much time is left for data collection and analysis as well as

preparing a report in a version ready for consultation.

Case study

Ex-ante evaluation of the HR PO

In the report the contractor indicates that due to a brief period of time in which the study was

to be conducted (11 weeks following the conclusion of the contract) not all the research ques-

tions were answered equally exhaustively. The conclusions in the report summary are given

for two evaluation criteria and take a single sentence.

41

For the two other criteria conclusions are lacking. A description of the results of the study

performed follows the research question–answer/conclusions pattern.

Possible use of recommendations

Inspiring

The usefulness of evaluation cannot be boiled down to just implementation or non-

implementation of some specific recommendations as it also consists in inspiring to act,

showing another point of view concerning a given issue subject to analysis. Several of our

respondents have given such examples of evaluations which were not used directly but inspi-

rational.

Writing recommendations down also serves convincing decision-makers that some changes

are worth introducing, even if not immediately then in the future. The most essential point is

voicing the need for change by an external entity – a neutral external evaluator – on the basis

of the conducted study.

It should be noted that evaluation usefulness does not consist in just immediate implementa-

tion. Even if not implemented, the recommendations have the following role to play:

- pointing to possible directions of action, which in the future may win acceptance of

decision-makers or be an inspiration for development, and

- improve communication inside the programme implementation structure, as many of

them are recommendations formulated directly by the respondents who are staff of the

institutions involved in the implementation of Programmes, beneficiaries and social

partners.

Closing brief

It must be said that the usefulness of recommendations for the institutions involved in the

implementation of Programmes is limited. This is the case although most recommendations

include rather specific guidelines as to what steps to take. The usefulness of recommendations

is restricted because their addressees are not clearly identified: neither evaluators nor the

Managing Authority or Monitoring Committees indicate the addressees. In consequence, no

single institution feels accountable for the formulated recommendations. This is most visible

42

at the level of Implementing Authorities which to change anything are keener to wait for

guidelines from the MA than follow the recommendations in the reports.

Some recommendations fail to acknowledge the regulatory framework of the structural funds,

which the recipients see as a drawback. Still, looking at this from a wider angle, it should be

said that such recommendations may be a source of inspiration for change in the long run.

They can also make a case in the process of lobbying for regulatory changes.

Effectiveness of recommendation implementation

Introduction

This section has presented the results of the study concerning:

- methods of implementing recommendations, in the context of their effectiveness and

functionality;

- the extent to which recommendations are used by the institutions which they concern;

- the role of the Implementing Authority in recommendation implementation;

- views of the staff of the institutions involved in the implementation of the programmes

subject to analysis as regards opportunities and limitations in recommendation imple-

mentation, and

- reasons for failure to implement recommendations.

Recommendation implementation

Recommendation management

There exists no system for managing the recommendations included in evaluation reports.

Most of our interlocutors, both parties contracting studies and potential recipients, realised the

need to develop standards in that regard. One recurring idea was to make evaluative-study

contractors obliged to summarise recommendations on a single page, which would give all the

interested people an opportunity to became familiar with them and create an incentive to read

the entire report.

Recommendation implementation is not subject to monitoring. Likewise, there is no system of

notification concerning the recommendations to be implemented or which have been carried

43

out. A vast majority of respondents from outside of the Managing Authority were unable to

show which changes in the programme (SOP HRD or EQUAL CIP) took place or were a con-

sequence of any evaluation study.

Good Polish practice

→ A pioneering activity was preparing a collection of key recommendations from all the

conducted evaluations to be presented before the Monitoring Committee. It contained all the

information on the source of the recommendation, to what it referred and to whom it was

addressed. It also included an opinion by the persons preparing the collection on the impor-

tance of a given recommendation.

→ During the examination of Measure 2.3, scheme a) some traces could be seen of formalis-

ing the use of recommendations, i.e. preparation of a recommendation table, and then moni-

toring activities related to recommendation implementation. That was an initiative by the

staff of Monitoring and Evaluation Division at the MRD developed jointly by the staff of the

ESF Management Department at the MRD and the PARP.

Case study

Lithuania: The institutions to which recommendations are addressed were mostly indicated

in the report. Specific calculations whether all the recommendations were carried out are not

always made (a formal monitoring mechanism being absent). Officially, recommendation

implementation methods are not described. For 2007-2013, more emphasis is planned to be

placed on recommendation implementation monitoring. Given that evaluation is not audit,

however, recommendation implementation is not going to be rigorously formalised, yet rele-

vant institutions are going to forward information on the recommendation implementation

process to the Evaluation Department. Whether a recommendation implementation schedule

exists depends on the nature of a recommendation. Typically, the time of recommendation

implementation is indicated in the evaluation report. It is not formally established who is

accountable for the performance of the whole implementation process and individual tasks.

In 2007-2013 all the tasks will be taken onboard by the Evaluation Department. It is envis-

aged that in 2007-2013 the institutions offered technical support for implementing evaluation

projects will be obliged to notify the Evaluation Department on a regular basis of the imple-

mentation process and results of recommendation evaluation. They will also have to provide

information on why given recommendations have not been followed (a novelty).

44

Finland: Recommendations from all the evaluation work are compiled into a single docu-

ment and then presented before the Monitoring Committee. Recommendations are rarely

addressed to specific institutions and even if that is the case, they relate to governmental in-

stitutions (most typically related to recommendations to increase funding for a given type of

activity). Recommendations are usually rather general, e.g. changes in the way a policy is

developed in a given area. The Committee examines all the recommendations and ponders

how the ESF implementation system could be made more effective. As for controversial rec-

ommendations or ones difficult to carry out, evaluators are also invited to attend discussions.

The Committee makes decision which of the submitted recommendations are to be followed.

There is no detailed recommendation implementation monitoring system, and so it is difficult

to estimate what percentage of the recommendations are carried out.

Italy: There are no structures dealing with recommendation implementation or uniform prac-

tice in that regard. Institutions being the addressees of the recommendations are directly

mentioned in reports. There is no specific tool ensuring the use of the evaluation results and

implementation of recommendations as well as enforcing/monitoring whether they have been

accepted. Also, as a result, there are no detailed schedules concerning possible implementa-

tion of the recommendations. Good practice related to recommendation implementation in-

cludes the formal obligation imposed on the Evaluation Unit coordinating the study: several

months after the completion of evaluation it must ask the persons in charge of a given pro-

gramme in writing how the process of following recommendations is going in practical

terms. It is a tool mobilising to carry out recommendations, although no specific sanctions

for lack of progress in that area are envisaged.

MA role

The Managing Authority exerts moderate impact upon recommendation implementation. The

expectation is palpable, however, that just the MA should decide as regards making use of

recommendations and provide lower-ranking institutions with relevant guidelines.

Case study

Current evaluation of EQUAL CIP

It is not clearly defined who is accountable for implementing recommendations. Also, there

45

is no system facilitating reflection on the suggested recommendations. From the viewpoint of

IA staff recommendations are of not much use. They are also not fully known as IA staff

have no obligation to read evaluation reports. From all this a picture emerges of a lack of

established task division between the MA and the IA as regards following conclusions from

the evaluation study. The result is recommendations hanging in an organisational vacuum.

Additionally, in their research procedure researchers do not take into account the study on

recommendation implementation and the results of change, although it is a cyclical evalua-

tion.

Use of recommendations

As until now no recommendation implementation system has been operational not a single

respondent was able to indicate how much or little the recommendations are used. The re-

spondents also found it difficult to name recommendations which were addressed to their in-

stitutions or ones which have been followed.

Case study

In the four analysed evaluation reports constituting case studies in this study recom-

mendations appear related to:

Promotion: necessity to strengthen or integrate the information and promotion policy (report

on the evaluation of EQUAL and Measure 2.3 of the HRD SOP).

Organisational structure: change of the structure and/or tasks of institutions engaged in the

programme implementation (HC OP, Assessment of SOP HRD implementation).

Payments: simplification of a financial settlement system, accelerating the process of expen-

diture and payment; reduction of red tape through, for example, change of procedures

(evaluation of Measure 2.3, assessment of SOP HRD implementation).

Applications submitted as part of programmes: improving the transparency of the applica-

tion evaluation criteria and changes in the competition procedure (assessment of SOP HRD

implementation, evaluation of Measure 2.3).

Closing brief

No effective way of recommendation implementation has been developed for entire Pro-

grammes. The methods of disseminating evaluation results do not guarantee that a debate will

46

be held on the purposefulness of carrying out recommendations. However, first steps have

been taken towards the creation of a recommendation implementation system. The institutions

which have undertaken the task boast strong evaluation units, the conclusion being that

knowledge of evaluation is a vital factor influencing the extent to which evaluation results are

used.

Also, whether an institution is going to follow recommendations depends very much on the

attitude of its staff towards evaluation. During the study performance we have identified both

institutions which are open towards change and treat evaluation as a tool supporting the man-

agement process and such which see evaluation as a control mechanism.

Also, there is no coherent policy for disseminating data from individual evaluation studies.

No clear definition of the stakeholder group at the stage of study planning results in uncer-

tainty to whom the study results should be addressed.

Using the evaluation results depends on the assessment of the quality of the evaluation report.

The general view of the staff of the institutions we have analysed is that many evaluation re-

ports have not been prepared diligently enough, a problem the evaluators are also aware of.

The reason seems to be too little time to perform the study.

47

Con

clus

ions

and

rec

omm

enda

tions

The

conc

lusi

ons

and

reco

mm

enda

tions

hav

e be

en n

umbe

red

depe

ndin

g on

the

impo

rtanc

e of

the

prop

osed

sol

utio

ns. A

ll th

e su

gges

tions

are

ad-

dres

sed

to th

e M

onito

ring

and

Eval

uatio

n U

nit a

t the

Dep

artm

ent f

or E

SF M

anag

emen

t as i

t is o

ur re

com

men

datio

n th

at th

is b

ody:

- be

in c

harg

e of

eva

luat

ion

coor

dina

tion

at th

e Pr

ogra

mm

e le

vel,

and

- pur

sue

an a

ctiv

e ed

ucat

iona

l pol

icy

as re

gard

s eva

luat

ion

vis-

à-vi

s oth

er in

volv

ed in

stitu

tions

.

Con

clus

ions

/ Iss

ues

At

pres

ent,

the

them

e of

con

tract

ed

eval

uatio

ns a

nd t

heir

goal

s ar

e no

t

cons

iste

nt w

ith th

e sc

ope

of re

spon

si-

bilit

ies

of t

he c

ontra

ctin

g in

stitu

tion,

stem

min

g fr

om i

ts p

lace

in

the

Pro-

gram

me

impl

emen

tatio

n st

ruct

ure.

This

lim

its t

he p

ossi

bilit

y of

the

rec

-

omm

enda

tions

bei

ng c

arrie

d ou

t di

-

rect

ly b

y th

e co

ntra

ctin

g pa

rty.

The

cont

ract

ing

party

ha

s lit

tle

say

as

rega

rds

use

of t

he r

ecom

men

datio

ns

by o

ther

ins

titut

ions

inv

olve

d in

the

impl

emen

tatio

n.

Rec

omm

enda

tions

1. It

is re

com

men

ded

that

the

leve

l of e

valu

atio

n of

the

Ope

ratio

nal P

rogr

amm

e fo

r whi

ch

indi

vidu

al i

nstit

utio

ns i

mpl

emen

ting

the

Prog

ram

me

are

resp

onsi

ble

be c

lear

ly d

efin

ed.

The

eval

uatio

n le

vel

shou

ld b

e co

nsis

tent

with

the

rol

e of

the

ins

titut

ion

in t

he P

ro-

gram

me

man

agem

ent

and

impl

emen

tatio

n st

ruct

ure.

In

the

reco

mm

ende

d m

odel

, w

e

sugg

est t

he fo

llow

ing

solu

tions

:

- Th

e ob

ligat

ion

to c

ondu

ct e

valu

atio

n at

the

leve

l of

the

entir

e pr

ogra

mm

e (e

.g.

ex-a

nte,

per

iodi

cal,

ex-p

ost,

trans

vers

al e

valu

atio

n or

eva

luat

ion

rela

ted

to t

he

impl

emen

tatio

n sy

stem

) sho

uld

lie w

ith th

e M

anag

ing

Aut

horit

y;

- Ev

alua

tion

at th

e le

vel o

f prio

ritie

s fo

r the

nat

iona

l com

pone

nt s

houl

d be

a ta

sk

of In

term

edia

te B

odie

s (a

t ind

ivid

ual M

inis

tries

) in

char

ge o

f the

ir im

plem

enta

-

tion;

- Ev

alua

tion

at th

e le

vel o

f prio

ritie

s fo

r the

regi

onal

com

pone

nt s

houl

d be

a ta

sk

of I

nter

med

iate

Bod

ies

(inte

r al

ia V

oivo

dshi

p M

arsh

al O

ffic

es a

nd V

oivo

dshi

p

Labo

ur O

ffic

es)

- Ev

alua

tion

at th

e le

vel o

f Mea

sure

s an

d pr

ojec

ts s

houl

d be

a re

spon

sibi

lity

of 2

nd

Eva

luat

ion

syst

em

48

The

eval

uatio

n pl

anni

ng m

etho

d do

es

not

guar

ante

e th

e re

leva

nce

of t

he

cont

ract

ed

stud

ies,

whi

ch

is

why

eval

uatio

n re

sults

are

not

muc

h of

a

nove

lty f

or t

he c

ontra

ctin

g pa

rties

.

Cur

rent

ly a

vas

t pa

rt of

eva

luat

ion

is

plan

ned

on t

he b

asis

of

the

stud

y

obje

ct; t

he s

tarti

ng p

oint

see

ms

to b

e

the

ques

tion

Wha

t to

exam

ine

(whi

ch

part

of

the

prog

ram

me)

? ra

ther

tha

n

Wha

t ki

nd o

f pr

oble

m a

re w

e su

p-

pose

d to

exa

min

e? w

hile

the

eva

lua-

tion

plan

ning

sta

ge i

s of

key

im

por-

degr

ee In

term

edia

te B

odie

s (or

1st d

egre

e, in

the

abse

nce

of IB

II)

2. E

valu

atio

n co

ordi

natio

n at

all

leve

ls s

houl

d be

a ta

sk o

f the

Man

agin

g A

utho

rity.

The

HC

OP

Man

agin

g A

utho

rity

shou

ld:

- co

ordi

nate

all

the

eval

uatio

n ac

tiviti

es p

erfo

rmed

as p

art o

f the

Pro

gram

me

at th

e

cent

ral a

nd re

gion

al le

vels

(e.g

. mon

itorin

g, c

oher

ence

of e

valu

atio

n pl

ans,

pres

-

enta

tion

of re

sults

at M

C m

eetin

gs, d

isse

min

atio

n of

resu

lts b

y pl

acin

g th

em o

n

the

web

site

, and

age

ncy

in e

xcha

nge

of e

xper

ienc

es e

.g. b

y or

gani

sing

wor

king

mee

tings

), an

d

- pr

ovid

e co

nten

t-rel

ated

ass

ista

nce

to th

e st

aff

in c

harg

e of

eva

luat

ion

in lo

wer

-

rank

ing

inst

itutio

ns a

t bot

h ce

ntra

l and

reg

iona

l lev

els

(suc

h su

ppor

t sho

uld

in-

clud

e, fo

r ins

tanc

e, tr

aini

ng a

nd st

udy

visi

ts).

3. E

valu

atio

ns s

houl

d be

pla

nned

and

per

form

ed b

y al

l th

e in

stitu

tions

inv

olve

d in

the

impl

emen

tatio

n of

the

Prog

ram

mes

. Eva

luat

ions

sho

uld

be p

lann

ed in

a v

arie

ty o

f tim

e

pers

pect

ives

, jus

t as

it is

cur

rent

ly d

one

for t

hose

con

duct

ed b

y th

e M

A. L

ong-

term

pla

n-

ning

sho

uld

incl

ude

trans

vers

al s

tudi

es, s

hort-

term

pla

nnin

g sh

ould

foc

us o

n st

udie

s of

resu

lts a

nd im

pact

(m

ore

smal

l-sca

le s

tudi

es s

houl

d be

con

duct

ed a

s pa

rt of

the

HC

OP,

e.g.

cov

erin

g at

leas

t 2-3

regi

ons)

, and

ad

hoc

stud

ies

on th

e m

anag

emen

t and

impl

emen

-

tatio

n sy

stem

s. Th

e pl

anni

ng p

roce

ss s

houl

d in

clud

e th

e vo

ice

of a

lar

ge s

take

hold

er

grou

p. I

t sho

uld

not b

e ex

pect

ed, h

owev

er, t

hat t

he s

take

hold

ers

will

be

mor

e ac

tive

in

that

rega

rd a

ny ti

me

soon

. Tha

t is

why

it is

reco

mm

ende

d th

at a

ctiv

ities

be

intro

duce

d to

supp

ort a

ckno

wle

dgin

g th

eir

need

s. A

lso,

eva

luat

ion

plan

ning

sho

uld

see

mor

e fo

cus

on

defin

ing

the

prob

lem

to b

e st

udie

d (w

hich

sho

uld

be th

e st

artin

g po

int).

Sta

rting

from

the

prob

lem

rath

er th

an o

bjec

t of t

he s

tudy

is th

e ke

y. A

nd s

o w

e su

gges

t tha

t lon

g- a

nd m

id-

term

eva

luat

ion

plan

s be

subj

ect t

o ex

-ant

e ev

alua

tion

requ

este

d by

the

Man

agin

g A

utho

r-

Prog

ram

me

eval

uatio

n sy

stem

Plan

ning

49

tanc

e in

th

e co

ntex

t of

ev

alua

tion

rele

vanc

e an

d us

eful

ness

for t

he in

sti-

tutio

ns im

plem

entin

g th

e Pr

ogra

mm

e.

That

is

why

it

is v

ital

that

bro

ades

t

poss

ible

par

ticip

atio

n of

sta

keho

lder

s

in th

e pl

anni

ng p

roce

ss b

e en

visa

ged.

How

ever

, sta

keho

lder

invo

lvem

ent i

n

eval

uatio

n pl

anni

ng a

t the

leve

l of t

he

entir

e op

erat

iona

l pr

ogra

mm

e is

cur

-

rent

ly w

eak.

The

Mon

itorin

g C

omm

it-

tee,

Eva

luat

ion

Stee

ring

Gro

up a

nd

repr

esen

tativ

es o

f in

divi

dual

Int

erm

e-

diat

e B

odie

s an

d Im

plem

entin

g A

u-

thor

ities

con

sult

eval

uatio

n ra

ther

than

plan

it, a

nd to

a li

mite

d ex

tend

at t

hat.

This

is m

ainl

y be

caus

e:

- ev

alua

tion

is n

ot t

reat

ed a

s a

tool

supp

ortin

g th

e m

anag

emen

t of

th

e

Prog

ram

mes

, and

- in

tern

al w

orki

ng p

roce

dure

s at

the

MC

and

ESG

are

dys

func

tiona

l.

ity (

as a

n in

stitu

tion

coor

dina

ting

eval

uatio

ns p

erfo

rmed

as

part

of th

e Pr

ogra

mm

e). I

ts

obje

ctiv

e w

ould

be

as fo

llow

s:

- im

prov

ing

the

rele

vanc

e of

the

plan

ned

eval

uatio

n th

roug

h a

mul

tifac

eted

dia

g-

nosi

s of n

eeds

, and

- im

prov

ing

eval

uatio

n us

eful

ness

for

low

er-r

anki

ng i

nstit

utio

ns a

nd t

he M

C

thro

ugh

invo

lvin

g th

eir r

epre

sent

ativ

es in

pla

nnin

g.

- Th

e ex

-ant

e ev

alua

tion

shou

ld a

lso

offe

r ans

wer

s to

the

follo

win

g qu

estio

ns:

- To

wha

t ext

ent d

o pl

anne

d ev

alua

tions

mee

t the

mos

t ess

entia

l nee

ds (b

oth

real

-

ised

and

not

real

ised

) of t

he in

stitu

tions

whi

ch re

ques

t the

m?

- W

ill (a

nd to

wha

t ext

ent)

the

resu

lt of

the

eval

uatio

n be

use

ful f

or o

ther

ent

ities

invo

lved

in P

rogr

amm

e im

plem

enta

tion?

- Ar

e th

e pl

anne

d ev

alua

tions

sepa

rabl

e (in

oth

er w

ords

: do

thei

r sco

pes n

ot o

ver-

lap)

?

- Th

e su

gges

ted

eval

uatio

n sh

ould

be

base

d on

qua

litat

ive

met

hods

of d

ata

colle

c-

tion

and

anal

ysis

(in

divi

dual

or

grou

p in

terv

iew

s w

ith s

take

hold

ers

and

docu

-

men

t ana

lysi

s). I

t is

to b

e ex

pect

ed th

at, a

s a

fast

and

sm

all-s

cale

exe

rcis

e, it

is

goin

g to

be

effe

ctiv

e an

d ef

ficie

nt.

4. T

he e

valu

atio

n pl

anni

ng p

roce

ss s

houl

d to

a la

rger

ext

ent e

nsur

e th

at a

wid

er g

roup

of

stak

ehol

ders

is ta

ken

into

con

side

ratio

n. T

his

can

be d

one

once

the

follo

win

g co

nditi

ons

have

bee

n m

et:

- R

epor

ts fr

om e

valu

atio

n st

udie

s sh

ould

incl

ude

reco

mm

enda

tions

con

cern

ing

fu-

ture

eva

luat

ion,

inc

ludi

ng s

ugge

sted

res

earc

h ar

ea a

nd q

uest

ions

. Tha

nks

to i

t

sugg

estio

ns n

ot ju

st fr

om e

valu

ator

s bu

t als

o w

ide

stak

ehol

der g

roup

s w

ho p

ar-

ticip

ated

in p

ast s

tudi

es c

ould

be

cons

ider

ed;

- th

e co

ntra

ctin

g pa

rty s

houl

d re

quire

of

the

eval

uato

rs th

at s

ugge

stio

ns b

e pu

t in

Eva

luat

ion

plan

ning

50

writ

ing

for f

urth

er re

sear

ch p

robl

ems t

oget

her w

ith a

just

ifica

tion,

and

- th

e pa

rty c

ontra

ctin

g th

e ev

alua

tion

shou

ld m

ake

the

area

of

the

plan

ned

stud

y

subj

ect

to c

olle

ctiv

e re

flect

ion

by t

he m

embe

rs o

f Ev

alua

tion

Stee

ring

Gro

up

(taki

ng p

artic

ular

inte

rest

in th

ose

who

m th

e st

udy

conc

erns

).

5. A

n Ev

alua

tion

Stee

ring

Gro

up s

houl

d be

est

ablis

hed

in e

ach

regi

on. T

he G

roup

sho

uld

deal

with

bot

h ev

alua

tion

plan

ned

as p

art o

f the

HC

OP

regi

onal

com

pone

nt a

nd R

egio

nal

Ope

ratio

nal

Prog

ram

mes

. The

Gro

up w

ould

be

com

pose

d of

rep

rese

ntat

ives

of

inst

itu-

tions

invo

lved

in th

e im

plem

enta

tion

of th

e Pr

ogra

mm

es a

s w

ell a

s a

repr

esen

tatio

n of

soci

al p

artn

ers a

nd e

xper

ts (n

ot in

tere

sted

in p

erfo

rmin

g th

e ev

alua

tion)

.

In c

ases

whe

n se

para

te S

teer

ing

Gro

ups

wer

e se

t up

in a

regi

on, o

ne fo

r HC

OP

regi

onal

com

pone

nt a

nd t

he o

ther

one

for

the

RO

P, i

t w

ould

be

reco

mm

ende

d th

at t

he G

roup

s

rem

ain

in to

uch

for t

he s

ake

of b

ette

r com

plem

enta

rity

of th

e im

plem

ente

d Pr

ogra

mm

es,

thro

ugh

a co

mpl

emen

tary

eva

luat

ion

syst

em.

6. T

he m

onito

ring

and

eval

uatio

n fu

nctio

ns s

houl

d be

inte

grat

ed s

o th

at m

onito

ring

re-

sults

are

mor

e in

tens

ely

used

in e

valu

atio

n pl

anni

ng a

nd d

ata

anal

ysis

.

To th

is e

nd, t

he fo

llow

ing

is re

com

men

ded:

- in

the

inst

itutio

ns in

volv

ed in

the

impl

emen

tatio

n of

the

Prog

ram

mes

org

anis

a-

tiona

l uni

ts s

houl

d op

erat

e de

alin

g w

ith m

onito

ring

and

eval

uatio

n (w

hich

is o

f-

ten

the

case

alre

ady

now

);

- ev

alua

tors

sho

uld

to a

larg

er e

xten

t use

the

qual

ity d

ata

from

the

perio

dica

l re-

ports

on

proj

ect i

mpl

emen

tatio

n, a

nd

- su

perio

r ins

titut

ions

(MA

and

IB) s

houl

d en

joy

dire

ct a

cces

s to

perio

dica

l rep

orts

on p

roje

ct im

plem

enta

tion.

Bec

ause

of

the

exis

ting

casc

ade

mon

itorin

g sy

stem

the

supe

rior

inst

itutio

ns r

ecei

ve a

lread

y ag

greg

ated

dat

a fr

om i

nstit

utio

ns o

f

Prog

ram

me

eval

uatio

n sy

stem

Eva

luat

ion

syst

em

51

low

er im

plem

enta

tion

rank

. As

a re

sult,

the

MA

doe

s no

t hav

e a

full

and

dire

ct

pict

ure

of P

rogr

amm

e im

plem

enta

tion

at th

e pr

ojec

t lev

el, i

n pa

rticu

lar o

f pro

b-

lem

s whe

n th

ey a

ppea

r.

To im

prov

e th

e sy

stem

for m

onito

ring

the

impl

emen

tatio

n of

the

Prog

ram

mes

at t

he p

ro-

ject

leve

l and

to in

corp

orat

e it

into

the

eval

uatio

n sy

stem

it m

ight

be

cons

ider

ed w

heth

er

an in

depe

nden

t ext

erna

l con

tract

or m

ight

be

requ

este

d to

dev

elop

ana

lyse

s of

per

iodi

cal

repo

rts. S

uch

anal

yses

sho

uld

be s

uppl

emen

ted

with

sev

eral

cas

e st

udie

s, ai

med

at c

ol-

lect

ing

addi

tiona

l in

form

atio

n co

ncer

ning

the

pro

blem

s in

dica

ted

in t

he r

epor

ts.

The

anal

yses

wou

ld se

rve:

- le

arni

ng a

bout

the

Prog

ram

me

impl

emen

tatio

n st

atus

at t

he p

roje

ct le

vel a

nd a

p-

pear

ing

diff

icul

ties;

- m

akin

g a

wid

e gr

oup

of s

take

hold

ers

quic

kly

fam

iliar

with

the

Prog

ram

me

im-

plem

enta

tion

stat

us a

t th

e pr

ojec

t lev

el w

ithou

t th

em d

evot

ing

time

to r

eadi

ng

tens

of p

erio

dica

l rep

orts

on

proj

ect i

mpl

emen

tatio

n;

- m

akin

g th

e co

nclu

sion

s m

ore

obje

ctiv

e as

the

anal

ysis

wou

ld b

e m

ade

by a

n ex

-

tern

al e

valu

ator

, so

meo

ne m

ore

impa

rtial

tha

n, f

or i

nsta

nce,

sta

ff d

irect

ly r

e-

spon

sibl

e fo

r Pro

gram

me

impl

emen

tatio

n, a

nd

- th

ey w

ould

als

o po

int o

ut a

reas

of f

utur

e ev

alua

tion

stud

ies (

whi

ch w

ould

dee

pen

and

broa

den

anal

yses

per

form

ed a

t the

mon

itorin

g le

vel).

7. D

epar

tmen

t hea

ds a

nd d

ivis

ion

dire

ctor

s sh

ould

sit

on E

valu

atio

n St

eerin

g G

roup

s. In

this

way

eva

luat

ion

coul

d be

com

e m

ore

usef

ul i

n m

anag

ing

the

Prog

ram

mes

. Th

ey

shou

ld, h

owev

er, r

ecei

ve t

rain

ing

in e

valu

atio

n so

tha

t ca

n ef

fect

ivel

y ta

ke p

art

in t

he

wor

k of

the

Gro

up.

8. I

t is

rec

omm

ende

d th

at w

hen

the

met

hodo

logy

rep

ort

is b

eing

dev

elop

ed e

valu

ator

s

Eva

luat

ion

syst

em

Eva

luat

ion

plan

ning

52

The

low

eff

ectiv

enes

s of

rec

omm

en-

datio

n im

plem

enta

tion

is t

o a

larg

e

exte

nt a

res

ult

of b

arrie

rs a

s re

gard

s

invo

lvem

ent

of s

take

hold

ers

in t

he

eval

uatio

n pr

oces

s an

d of

sta

ff f

rom

the

invo

lved

ins

titut

ions

in

re

com

-

men

datio

n im

plem

enta

tion.

Th

ese

barr

iers

ar

e du

e to

th

e st

ill

poor

mee

t rep

rese

ntat

ives

of t

he k

ey in

stitu

tions

invo

lved

in P

rogr

amm

e im

plem

enta

tion.

Suc

h

a re

quire

men

t sho

uld

be w

ritte

n do

wn

in th

e To

R. T

he c

ontra

ctin

g pa

rty s

houl

d al

so a

l-

low

for

som

e tim

e in

the

stud

y sc

hedu

le to

be

devo

ted

to s

uch

conv

ersa

tions

con

duct

ed

by e

valu

ator

s. Th

ey w

ould

serv

e:

- le

arni

ng th

e ne

eds

of th

e ke

y st

akeh

olde

rs a

nd p

ossi

bly

taki

ng th

em in

to c

onsi

d-

erat

ion

in th

e m

etho

dolo

gy re

port;

- id

entif

ying

pot

entia

l diff

icul

ties

in s

tudy

per

form

ance

(e.g

. diff

icul

ties

in a

cces

s-

ing

tele

addr

ess d

ata

and

resp

onde

nt re

sist

ance

), an

d

- cr

eatin

g a

favo

urab

le c

limat

e ar

ound

the

stu

dy,

whi

ch s

houl

d le

ad t

o a

mor

e

open

atti

tude

tow

ards

reco

mm

enda

tions

.

9. It

is re

com

men

ded

that

the

key

stak

ehol

ders

be

invo

lved

in th

e ev

alua

tion

mon

itorin

g

proc

ess.

Rep

rese

ntat

ives

of t

he in

stitu

tions

in c

harg

e of

the

impl

emen

tatio

n of

task

s su

b-

ject

to e

valu

atio

n sh

ould

atte

nd m

eetin

gs w

ith e

valu

ator

s w

hen

prel

imin

ary

stud

y re

sults

are

disc

usse

d. T

he k

ey s

take

hold

ers

shou

ld a

lso

rece

ive

inte

rim r

epor

ts f

or c

onsu

ltatio

n.

All

this

wou

ld im

prov

e ev

alua

tion

usef

ulne

ss a

nd h

elp

crea

te a

favo

urab

le c

limat

e ar

ound

the

stud

y.

10. A

ctiv

ities

mus

t be

inte

nsifi

ed w

hich

aim

at p

rom

otin

g ev

alua

tion

as a

tool

sup

porti

ng

the

impl

emen

tatio

n of

the

Prog

ram

mes

. To

this

end

, it w

ould

be

wor

thw

hile

to:

- pr

omot

e qu

ality

and

use

ful e

valu

atio

ns; p

oint

out

the

chan

ges m

ade

as a

resu

lt of

eval

uatio

n;

- of

fer t

rain

ing

aim

ed a

t mak

ing

eval

uatio

n re

cipi

ents

(dec

isio

n-m

aker

s an

d re

gu-

lar

staf

f, m

embe

rs o

f M

onito

ring

Com

mitt

ees)

aw

are

that

eva

luat

ion

is n

ot a

n-

othe

r co

ntro

l to

ol a

nd c

an b

e us

eful

if

it is

app

roac

hed

in t

he r

ight

way

; th

is

wou

ld m

ake

them

mor

e aw

are

reci

pien

ts o

f ev

alua

tion

repo

rts,

whi

ch s

houl

d

Mon

itori

ng

Edu

catio

n

53

know

ledg

e of

eva

luat

ion,

fea

r of

as-

sess

men

t an

d re

luct

ance

to

war

ds

chan

ge.

Ther

e is

no

reco

mm

enda

tion

impl

e-

men

tatio

n sy

stem

in p

lace

. No

mon

i-

torin

g of

rec

omm

enda

tion

impl

emen

-

tatio

n is

per

form

ed, a

nd th

e re

sults

of

reco

mm

enda

tion

impl

emen

tatio

n ar

e

not

verif

ied

at t

he s

ubse

quen

t st

ages

of th

e st

udy.

trans

late

into

bet

ter u

se o

f eva

luat

ion

resu

lts; t

hey

wou

ld a

lso

be m

ore

awar

e of

rese

arch

nee

ds, w

hich

wou

ld m

ost p

roba

bly

lead

to th

em b

eing

mor

e ac

tive

in

the

eval

uatio

n pl

anni

ng p

roce

ss; t

he tr

aini

ng sh

ould

incl

ude

pres

enta

tion

of g

ood

prac

tice

as re

gard

s use

of e

valu

atio

n re

sults

; the

trai

ning

sho

uld

be c

ondu

cted

by

both

the

Man

agin

g A

utho

rity

(as

the

eval

uatio

n co

ordi

nato

r at

the

leve

l of

the

entir

e Pr

ogra

mm

e) a

nd in

tern

al e

valu

atio

n un

its, a

nd

- di

rect

ly e

ngag

e st

aff o

f ent

ities

pot

entia

lly in

tere

sted

in e

valu

atio

n re

sults

in d

e-

finin

g re

sear

ch a

reas

(pa

rtici

patio

n in

the

stu

dy a

t th

e pl

anni

ng s

tage

will

im

-

prov

e its

rele

vanc

e an

d, c

onse

quen

tly, u

sefu

lnes

s).

11. P

rofe

ssio

nal t

rain

ing

is a

wor

thw

hile

opt

ion

as is

the

deve

lopm

ent o

f st

aff

read

y to

wor

k in

the

area

of e

valu

atio

n st

udie

s. Su

ch tr

aini

ng s

houl

d ne

verth

eles

s pr

epar

e st

aff n

ot

to c

reat

e ex

cess

ivel

y de

taile

d To

R a

s re

gard

s m

etho

dolo

gica

l too

ls b

ut ra

ther

equ

ip th

em

with

the

abili

ty to

dia

gnos

e re

sear

ch p

robl

ems

and

then

com

pete

ntly

rec

eive

eva

luat

ion

resu

lts a

nd w

ork

with

reco

mm

enda

tions

.

12. T

he in

stitu

tion

cont

ract

ing

eval

uatio

n sh

ould

be

char

ged

with

the

task

of t

rans

ferr

ing

and

diss

emin

atin

g re

com

men

datio

ns t

o en

titie

s to

whi

ch t

he e

valu

ator

spe

cific

ally

ad-

dres

ses

them

. Thi

s sh

ould

be

the

case

eve

n if

thes

e en

titie

s do

not

rep

ort t

o it

(in w

hich

case

the

rec

omm

enda

tions

can

be

used

for

lob

byin

g fo

r ch

ange

). R

ecom

men

datio

ns

shou

ld a

lso

enta

il au

thor

ity to

mak

e de

cisi

ons

in th

e ar

ea c

over

ed b

y th

e re

com

men

da-

tions

; oth

erw

ise

frus

tratio

n of

low

er-r

anki

ng st

aff w

ill p

reva

il.

13.

Ther

e is

a n

eed

to m

odify

the

for

m i

n w

hich

Im

plem

entin

g A

utho

ritie

s ar

e gi

ven

guid

elin

es fo

llow

ing

from

the

reco

mm

enda

tions

incl

uded

in th

e ev

alua

tion

repo

rts. U

ntil

now

, the

y ha

ve b

een

hard

ly d

istin

guis

habl

e in

a fl

ow o

f mai

l add

ress

ed to

a g

iven

inst

itu-

Edu

catio

n

Dis

sem

inat

ion

of r

esul

ts

Dis

sem

inat

ion

of r

esul

ts

54

tion.

Suc

h gu

idel

ines

mus

t be

give

n hi

gh p

riorit

y an

d a

new

dis

tinct

ive

form

.

14. D

isse

min

atio

n of

eva

luat

ion

resu

lts s

houl

d al

way

s ta

ke th

e fo

rm o

f mee

tings

. Inv

ita-

tions

shou

ld b

e ex

tend

ed to

repr

esen

tativ

es o

f the

inst

itutio

ns w

hich

the

reco

mm

enda

tions

conc

ern

so a

s to

allo

w th

em to

com

men

t and

exp

ress

thei

r opi

nion

as

to w

heth

er th

e re

c-

omm

enda

tions

fro

m t

he r

epor

t ar

e pr

actic

able

. Afte

r th

e m

eetin

g th

e ev

alua

tors

sho

uld

mak

e th

e re

com

men

datio

ns m

ore

prac

ticab

le, t

hank

s to

whi

ch a

t a la

ter

stag

e th

ey w

ill

stan

d a

bette

r ch

ance

of

bein

g fo

llow

ed th

an w

ould

hav

e be

en th

e ca

se w

ithou

t the

pre

-

limin

ary

‘con

sulta

tions

’. Fo

r mee

tings

con

cern

ing

eval

uatio

n re

sults

, it i

s wor

thw

hile

to:

- se

nd th

e st

udy

resu

lts in

adv

ance

so th

at th

e pa

rtici

pant

s hav

e en

ough

tim

e to

be-

com

e fa

mili

ar w

ith t

hem

and

con

sult

deci

sion

-mak

ers

on t

he r

ecom

men

datio

n

impl

emen

tatio

n op

portu

nitie

s (id

eally

a w

eek

prio

r to

the

mee

ting)

, and

- in

vite

a b

road

par

ticip

ant

grou

p: r

epre

sent

ativ

es o

f th

e in

stitu

tions

inv

olve

d in

prog

ram

me

impl

emen

tatio

n (a

lso

pers

ons

from

out

side

of t

he E

SG),

bene

ficia

r-

ies,

MC

repr

esen

tativ

es, e

valu

ator

s (a

s w

ell a

s ot

her t

eam

s w

ho d

id n

ot p

erfo

rm

a gi

ven

stud

y).

- D

urin

g th

e m

eetin

g, it

is r

ecom

men

ded

that

the

MA

pro

vide

som

e co

mm

enta

ry

for

the

reco

mm

enda

tions

: w

hich

rec

omm

enda

tions

cou

ld b

e ca

rrie

d ou

t in

the

curr

ently

app

licab

le l

egal

con

text

. If

rec

omm

enda

tion

cann

ot b

e fo

llow

ed,

it

wou

ld b

e us

eful

to sa

y w

hy.

15. A

‘rec

omm

enda

tion

impl

emen

tatio

n ta

ble’

sho

uld

be d

evel

oped

incl

udin

g: a

des

crip

-

tion

of th

e de

sire

d st

ate,

task

s to

be

perf

orm

ed to

geth

er w

ith a

sch

edul

e, a

list

of p

erso

ns

in c

harg

e of

task

per

form

ance

. Suc

h ta

bles

sho

uld

be p

repa

red

by in

terin

stitu

tiona

l tea

ms

so th

at r

ecom

men

datio

n im

plem

enta

tion

does

not

res

embl

e a

tech

nocr

atic

pro

cedu

re b

ut

stem

s fr

om th

e st

aff’

s co

nvic

tion

that

the

obje

ctiv

e is

rele

vant

and

the

way

in w

hich

the

Dis

sem

inat

ion

of r

esul

ts

Rec

omm

enda

tion

impl

emen

tatio

n

55

The

qual

ity

of

eval

uatio

n re

ports

rais

es c

once

rns.

Con

tract

ing

parti

es

com

plai

n th

at t

hey

are

poor

ly p

re-

pare

d. R

ealis

ing

this

, th

e ev

alua

tors

say

that

the

rea

son

is t

hat

they

hav

e

little

tim

e be

twee

n w

hen

the

cont

ract

is s

igne

d an

d w

hen

the

first

ver

sion

of

the

repo

rt m

ust b

e pr

esen

ted.

chan

ge is

mad

e is

righ

t. Th

e te

ams

shou

ld in

clud

e: re

pres

enta

tives

of t

he in

stitu

tion

con-

tract

ing

the

eval

uatio

n, a

repr

esen

tatio

n of

all

the

inst

itutio

ns w

hich

the

reco

mm

enda

tions

conc

ern

(at

best

the

se s

houl

d be

at

the

sam

e tim

e m

embe

rs o

f th

e Ev

alua

tion

Stee

ring

Gro

up) a

nd th

e M

onito

ring

Com

mitt

ee.

As

a su

perio

r ent

ity in

Pro

gram

me

impl

emen

tatio

n, th

e M

anag

ing

Aut

horit

y sh

ould

als

o

have

the

initi

ativ

e as

rega

rds

the

tabl

es. M

oreo

ver,

the

colle

cted

dat

a su

gges

t tha

t sta

ff a

t

low

er-r

anki

ng in

stitu

tions

exp

ect j

ust t

hat.

16. T

he o

pera

tiona

l sta

ndar

d sh

ould

be

that

the

inst

itutio

ns to

whi

ch th

e re

com

men

datio

ns

refe

r pr

esen

t th

e M

onito

ring

Com

mitt

ee w

ith t

heir

view

on

the

stud

y re

sults

and

sug

-

gest

ed c

hang

es. T

he M

A sh

ould

pre

sent

the

Mon

itorin

g C

omm

ittee

with

reco

mm

enda

tion

impl

emen

tatio

n pl

ans a

nd th

en im

plem

enta

tion

repo

rts.

17. I

nfor

mat

ion

shou

ld b

e di

ssem

inat

ed c

once

rnin

g w

hich

reco

mm

enda

tions

are

goi

ng to

be c

arrie

d ou

t and

whi

ch h

ave

been

alre

ady.

A s

yste

m fo

r eva

luat

ion,

and

pos

sibl

y m

oni-

torin

g, s

houl

d in

clud

e ev

alua

tion

of t

he r

esul

ts o

f th

e im

plem

ente

d re

com

men

datio

ns.

Info

rmat

ion

on th

e cu

rren

tly im

plem

ente

d re

com

men

datio

ns s

houl

d be

als

o tra

nsfe

rred

to

the

rese

arch

ers

so th

at th

ey c

an im

prov

e th

eir

wor

king

met

hods

and

be

bette

r in

form

ed

auth

ors o

f res

earc

h pr

ojec

ts in

the

futu

re.

18. T

he c

ontra

ctin

g pa

rty p

rovi

ding

eva

luat

ors

with

fee

dbac

k co

ncer

ning

the

usef

ulne

ss

of re

com

men

datio

ns a

nd w

hat h

appe

ns to

them

nex

t cou

ld b

e a

good

pra

ctic

e m

otiv

atin

g

eval

uato

rs a

nd le

ndin

g m

ore

mea

ning

to th

eir

wor

k. S

uch

prac

tice

wou

ld a

lso

faci

litat

e

build

ing

bette

r co

-ope

ratio

n be

twee

n th

e co

ntra

ctor

and

the

cont

ract

ing

party

as

wel

l as

impr

ove

eval

uatio

n re

leva

nce

and

usef

ulne

ss.

Rec

omm

enda

tion

impl

emen

tatio

n

Impl

emen

tatio

n m

onito

ring

Feed

back

56

The

poor

ev

alua

tion

usef

ulne

ss

is

influ

ence

d by

th

e w

ay

eval

uatio

n

resu

lts a

re d

isse

min

ated

. R

epor

ts a

re

too

long

, whi

ch is

a m

ajor

res

trict

ion

if on

e w

ants

to le

arn

the

resu

lts o

f the

stud

y as

the

stru

ctur

al f

unds

are

a

dom

ain

of p

eopl

e bu

rden

ed w

ith a

heav

y w

orkl

oad.

19. M

ore

time

shou

ld b

e gi

ven

for e

valu

atio

n pe

rfor

man

ce. T

his

wou

ld im

prov

e th

e qu

al-

ity o

f ev

alua

tion

repo

rts,

save

the

sta

ff o

f th

e co

ntra

ctin

g in

stitu

tions

tim

e ne

eded

for

corr

ectin

g po

orly

dra

fted

(due

to li

mite

d tim

e) re

ports

.

20. W

e re

com

men

d th

at e

valu

atio

n re

ports

be

shor

ter.

They

mus

t be

frie

ndly

to th

e us

er,

adm

inis

trativ

e st

aff,

who

doe

s not

hav

e m

uch

time

for r

eadi

ng. A

nd so

:

- ev

alua

tion

repo

rts sh

ould

be

shor

ter,

not e

xcee

ding

100

pag

es; a

n in

tegr

al p

art o

f

the

repo

rt m

ust b

e its

sum

mar

y (id

eally

not

long

er th

an 1

0 pa

ges

and

form

ing

a

sepa

rate

doc

umen

t as

suc

h ar

e ea

sier

to

dow

nloa

d fr

om t

he w

ebsi

te),

whi

ch

mea

ns a

larg

er re

ader

ship

;

- th

e re

port

shou

ld b

e ac

com

pani

ed b

y a

one-

page

brie

f inc

ludi

ng k

ey re

sults

and

reco

mm

enda

tions

(to

be

rece

ived

by

the

mem

bers

of

Mon

itorin

g C

omm

ittee

s

and

ESG

s);

- st

udy

resu

lts s

houl

d be

pre

sent

ed in

a ta

ble

incl

udin

g m

ain

prob

lem

s an

d a

de-

scrip

tion

of a

ctiv

ities

aim

ed a

t res

olvi

ng th

em, a

nd

- re

com

men

datio

ns s

houl

d in

clud

e a

desc

riptio

n of

the

desi

red

stat

e an

d a

sugg

es-

tion

(sug

gest

ions

) for

a w

ay (w

ays)

to a

chie

ve th

at s

tate

, and

so

they

sho

uld

an-

swer

the

follo

win

g qu

estio

ns:

- W

hat s

houl

d re

ality

look

like

? / W

hat c

hang

e sh

ould

be

mad

e?

- W

hat

shou

ld b

e do

ne t

o ac

hiev

e th

e de

sire

d st

ate?

/ W

hat

shou

ld b

e do

ne t

o

mak

e th

e de

sire

d ha

ppen

?

New

way

s co

uld

be fo

und

to p

rese

nt th

e st

udy

resu

lts u

sing

a v

arie

ty o

f for

ms

for d

iffer

-

ent r

ecip

ient

gro

ups (

e.g.

key

stak

ehol

ders

, fin

al b

enef

icia

ries a

nd th

e ge

nera

l pub

lic).

21. T

he u

sefu

lnes

s of

rec

omm

enda

tions

sho

uld

not b

e ju

dged

onl

y fo

r th

eir

fast

impl

e-

men

tatio

n po

tent

ial.

It is

reco

mm

ende

d th

at th

e re

ports

not

avo

id re

com

men

datio

ns w

hich

Qua

lity

of r

epor

ts

Dis

sem

inat

ion

of r

esul

ts

Qua

lity

of r

epor

ts

57

The

exis

ting

syst

em

for

sele

ctin

g

eval

uatio

n co

ntra

ctor

s do

es n

ot g

uar-

ante

e hi

gh

rele

vanc

e of

ev

alua

tion

stud

ies.

go b

eyon

d th

e cu

rren

t reg

ulat

ory

envi

ronm

ent a

nd s

o th

eir i

mpl

emen

tatio

n is

not

pos

sibl

e

at g

iven

tim

e. S

uch

reco

mm

enda

tions

sho

uld

be tr

eate

d as

a s

ignp

ost p

oint

ing

out p

ossi

-

ble

dire

ctio

ns f

or c

hang

e, in

clud

ing

thos

e of

lega

l nat

ure.

It i

s re

com

men

ded,

how

ever

,

that

the

repo

rts m

aint

ain

a ba

lanc

e be

twee

n re

com

men

datio

ns th

at c

an b

e im

plem

ente

d

unde

r the

exi

stin

g la

w (p

rovi

ding

gui

delin

es a

s to

wha

t can

be

chan

ged

alre

ady

now

) and

thos

e w

hich

req

uire

leg

isla

tive

amen

dmen

ts.

If t

he r

epor

t in

clud

es r

ecom

men

datio

ns

goin

g be

yond

the

curr

ent r

egul

ator

y en

viro

nmen

t, th

ey s

houl

d be

acc

ompa

nied

by

a no

te

sayi

ng th

at th

ey d

o no

t fit

in th

e cu

rren

t leg

al sy

stem

.

22. A

s ou

r ex

perie

nce

in e

valu

atin

g EU

-fun

ded

prog

ram

mes

is s

till

mod

est,

eval

uatio

n

stud

ies

are

not

rout

ine

jobs

and

req

uire

a c

usto

mis

ed a

ppro

ach.

Whe

n th

e bi

ds a

re a

s-

sess

ed, a

ttent

ion

shou

ld b

e fo

cuse

d on

the

stud

y ou

tline

, met

hodo

logy

as

wel

l as

the

ex-

perti

se o

f the

team

of e

valu

ator

s. Th

e pr

ice

shou

ld a

lso

be c

onsi

dere

d, y

et –

follo

win

g th

e

Finn

ish

exam

ple

– it

shou

ld b

e an

alys

ed lo

okin

g at

the

corr

espo

nden

ce b

etw

een

the

cost

s

and

plan

ned

activ

ities

, not

just

the

amou

nt it

self.

The

reco

mm

ende

d so

lutio

n go

es b

eyon

d

the

curr

ent

regu

lato

ry e

nviro

nmen

t (P

ublic

Pro

cure

men

t A

ct),

how

ever

it

does

sho

w

a di

rect

ion

for d

esira

ble

legi

slat

ive

chan

ges.

23. D

urin

g th

e se

lect

ion

of e

valu

ator

s, tw

o-st

age

tend

erin

g w

ith n

egot

iatio

ns s

houl

d be

used

mor

e of

ten.

Thi

s pr

oced

ure

allo

ws

the

cont

ract

ing

party

to

beco

me

fam

iliar

with

dive

rse

rese

arch

idea

s on

the

basi

s of

whi

ch a

mor

e de

taile

d D

OC

will

be

draf

ted.

An-

othe

r go

od s

ide

of th

e pr

oced

ure

is d

ialo

gue

betw

een

the

cont

ract

ing

party

and

the

bid-

ders

, whi

ch m

ight

tran

slat

e in

to b

ette

r rel

evan

ce o

f the

stu

dy. M

akin

g us

e of

neg

otia

tions

is r

ecom

men

ded

whe

n la

rge-

scal

e ev

alua

tion

stud

ies

are

perf

orm

ed, p

lann

ed l

ong-

term

and

conc

erni

ng im

pact

, as

such

stu

dies

in p

artic

ular

cal

l for

non

-sta

ndar

d co

ncep

tual

isa-

tion.

Eva

luat

or se

lect

ion

Eva

luat

or se

lect

ion

58

24. M

ore

room

for

man

oeuv

re s

houl

d be

lef

t to

the

eva

luat

ors

so t

hat

they

can

sel

ect

rese

arch

met

hods

mor

e fr

eely

. Tha

t is

why

the

Des

crip

tion

of th

e O

bjec

t of t

he C

ontra

ct

does

not

hav

e to

con

tain

a d

escr

iptio

n of

the

met

hodo

logi

cal m

inim

um. T

his

shou

ld b

e

bene

ficia

l for

con

tract

or s

elec

tion

as th

e co

ntra

ctin

g pa

rty w

ill b

e ab

le m

ake

the

choi

ce

from

a m

ore

varie

d of

fer:

afte

r al

l th

e bi

ds m

ight

inc

lude

inn

ovat

ive

solu

tions

or

ones

prov

ing

that

a g

iven

team

of

eval

uato

rs is

mor

e kn

owle

dgea

ble

in th

e su

bjec

t. A

n ex

tra

bene

fit w

ould

be

less

tim

e sp

ent b

y th

e st

aff o

f the

con

tract

ing

inst

itutio

n on

DO

C p

repa

-

ratio

n.

25. T

he c

ontra

ctin

g in

stitu

tion

shou

ld re

serv

e fo

r its

elf a

righ

t in

the

cont

ract

to a

ppro

ve

of th

e ex

perts

sug

gest

ed b

y co

ntra

ctor

s. A

ctua

l tea

m le

ader

s sh

ould

be

pers

ons

expe

ri-

ence

d in

per

form

ing

sim

ilar e

valu

atio

n w

ork

in P

olan

d.

26. T

o im

prov

e th

e qu

ality

of t

he s

ubm

itted

bid

s, th

e pe

riod

for t

he s

ubm

issi

on s

houl

d be

exte

nded

to 3

-4 w

eeks

, so

that

pot

entia

l bid

ders

hav

e m

ore

time

to b

ecom

e fa

mili

ar w

ith

the

Prog

ram

me

and

deve

lop

a st

udy

outli

ne in

clud

ing

mor

e or

igin

al s

ugge

stio

ns th

an is

the

case

now

.

27. W

hen

cont

ract

ors

are

sele

cted

pre

fere

nce

shou

ld b

e gi

ven

to in

terd

isci

plin

ary

team

s,

thus

incr

easi

ng th

e lik

elih

ood

of th

e se

lect

ed te

am b

eing

abl

e to

dra

w m

ore

mul

tifac

eted

conc

lusi

ons a

nd re

com

men

d m

ore

prac

tical

cha

nges

.

Eva

luat

or se

lect

ion

Eva

luat

or se

lect

ion

Eva

luat

or se

lect

ion

Eva

luat

or se

lect

ion