Paolo Severati Isfol – ESF Evaluation Unit

ESF Evaluation Partnership MeetingBrussels, 19 November

Introduction: evaluation context in Italy Production: building a favourable

environment to outcome measurement and impact evaluation, many efforts and good results

Dissemination: significant efforts, but Italy can and should do better

Use: a highly critical point Conclusions

Experiences in evaluation of regional policies and no experiences (or, better, no demand) in evaluation of national policies

Evaluation is confined to the Structural Funds world

At regional level:

In many cases there is no clear programme theory

Sometimes outcomes are not declared and then are not measured; outputs and results are not recurrently and rigorously measured

More generally:

Great attention to inputs and to the formal respect of rules and laws on the ‘correct’ use of funds

Accountability is not always and everywhere considered an inescapable obligation

Scarce demand of evaluation

Generally the bulk of the demand of evaluation is due to exogenous pressures

Evaluation in too many cases, still today, is perceived as an obligation

Policy maker fear impact evaluation findings

Italy is made up of 21 ESF Regional Managing Authorities (19 Regions and 2 Autonomous Provinces); 5 of them belong to Convergence Objective (Basilicata, phasing out)

Local monitoring systems are very heterogeneous and evaluation culture is not evenly developed

Outcomes measurement

Impact evaluations

Where: Objective 3 (all the regions); Objective 1 (all the regions with the exception of Calabria)

When: near the mid-term evaluation and the updating of the mid term evaluation (Objective 3); at the end of the programming period (2006 and 2008) in Objective 1 Regions

By whom: in Objective 3, regional unit (regional research centres, regional agencies); in Objective 1, ISFOL (2006) and regional unit (Sicilia, Basilicata;2008)

Where: Objective 3

When: mid-term evaluation and updating of mid-term evaluation

By whom: ISFOL (national level); Emilia Romagna, Friuli Venezia Giulia (regional level)

At regional level, large and consistent delay in the production of data

On going evaluation, no obligation (evaluation plans were defined at the beginning of the programming period)

No impact evaluation realised

A national beneficiary survey conducted by ISFOL has started. The esteems will be significant (at least) at a regional level. Each region will dispose of the micro data referring to its interventions

Internal control groups will be complemented by external control groups extracted from administrative datasets coming from PES (Public Employment Services)

Impact evaluations will be realised by ISFOL where it will be possible

A technical group coordinated by Isfol and participated by all regional authorities, devoted to the building up of common instruments, the exchange of methods and approaches to evaluation

More and better data for impact evaluations

Evaluation reports were discussed in the Steering and Monitoring Committee and in the Evaluation Technical Groups

Paper copies of the evaluation reports had a good circulation among the insiders. Electronic versions of the same reports were freely available on the web (regional sites and UVAL-DPS site).

Instruments used in 2000-2006 ESF programming period still remain in the current programming period

Anyway, more efforts have to be made to reach

the public opinion: 1. Press releases on the evaluation results and policy

recommendations. Interviews and video-interviews to the evaluators. Short synthesis of the evaluations for citizens, public officers, project managers and social partners ...

2. More publication in English: the potential reader is possibly someone who lives in other European countries

A recent, important experience

Italy was one of the four countries that cooperated with the DG EMPL (ESF Coordination division) in order to realise a study on the return on the ESF investment in the human capital

The data used for impact evaluations at a regional and national level were used by an independent evaluator for new evaluations at a European level

This experience showed to be very useful

In the last programming period, generally evaluations were ignored. But there were some exceptions

Two examples: 1. one good example (Emilia Romagna) of

effective use of the major evaluation findings unfortunately had short life

2. one bad example: Isfol recommendations (distinction of preventive and curative approach measures) contained in the mid term evaluation of CSF Obj.3 were not respected: organisational needs expressed by some relevant regions prevailed

In the last programming period too strict rules on evaluation (too much stick) were stated

In the current programming period: too loose rules on evaluation (too much carrot) are into force

To promote a new contractual relationship, implementation and reporting aimed at exchange of results between Member States (Regions) and the Commission

To adopt a prospective impact evaluation: designing impact evaluation in tandem with policy design. A mutual commitment join policy makers and evaluators

A significant effort was made to design an ambitious evaluation infrastructure, usable at national and regional level

Unfortunately, monitoring and evaluation culture is not evenly diffused among the Italian regions

Large and consistent delay in the production of data at regional level

Next year new micro-data will be usable for impact evaluations

Here the item is how to involve policy makers in monitoring (surveying). In fact, at the moment:

some regions do not collect micro data on projects and beneficiaries

some others do not use micro data to start surveys

No additional rules are necessary to change this embarrassing situation. It is ‘only’ needed to enforce the current ones. Incentives and sanctions to convince policy makers to be more transparent in the use of public funds

Thank you very much

For contacts e-mail to : p.severati@isfol.it