Result orientation challenges – the Polish approach

Urszula RomańskaDepartment for Coordination of Development Strategies

and PoliciesMinistry of Infrastructure and Development

1. Participatory model – the process went hand in hand with programming of the PA and OPs.

2. Intervention logic was at the centre of our attention – correct identification and definition of desired results taking into account the Europe 2020 strategy.

3. Standardised approach to defining the desired results – started from the specific objectives of the PA within thematic objectives, defining the internal and external factors influence the final result:

a. how should a TO be used to achieve the results? specific objectives of the PA

b. linking the EU targets and priorities with the PL strategic documents

c. creating a matrix of logical intervention for each thematic objective

d. strengthening the links between the diagnosis and the specific objectives intervention logic defining the input of cohesion policy to the desired results

Result-orientation in the partnership

agreementExtensive use of ex ante evaluation of the PAExtensive use of ex ante evaluation of the PA

EDUCATION SYSTEM DOES NOT MEET THE NEEDS OF THE

LABOUR MARKET

INCOMPLETE USE OF LABOUR MARKET

RESOURCES

8.7

8.5.

8.6.

8.11

8.9

8.8.

DEVELOPMENT OF ENDOGENOUS

POTENTIAL

INCREASE IN ADAPTABILITY OF PROFESSIONALLY

ACTIVE PEOPLE AND ENTERPRISES

IMPROVEMENT IN THE HEALTH CONDITION OF

LABOUR MARKET RESOURCES

MORE EFFICIENT USE OF LABOUR

MARKET RESOURCES

INCREASE IN EMPLOYMENT

INDICATOR:

1) AGE 20-64 W / M: 57,6%/72,1%

65,2%/82,6%

2) AGE 55-64 W / M 31,0%/51,3%

32,5%/65,2%

3) RURAL AREAS: 50,3% 57,9%

NEEDS ACTIONS (IPs) RESULTS OBJECTIVES

8.10.

8.2

IMPROVEMENT OF CHANCES FOR

EMPLOYMENT OF THE UNEMPLOYED AND JOB-

SEEKERS

3%3%

28%28%

25%25%

22%22%

1%1%

6%6%

10%10%

6%6%

THEMATIC OBJECTIVE 8 IN THE PA4.8 bln €

(4 778,7mln € ESF + 200,8mln € ERDF)

A three-step system of indicators:1.10 development indicators in the PA – macroeconomic indicators (GDP p.c., debt), human development index, etc. + E2020 targets relating to employment, higher education and drop-outs; without baseline values, only milestones for 2020 and target for 2023

2.19 strategic result indicators in the PA – the other E2020 targets and other strategic results indicators stemming from the national strategies,

a. 1-2 per TO (e.g. World Bank’s ease of doing business for TO11)b. with baseline values (2010, 2012 2013 depending on the data availability) and

targets for 2020 and 2030c. an estimated input of ESIF (ranging from 0,46% up to 93%)d. „STRATEG” – an open digital system of strategic result indicators, based on

public statistics as well as some data from the national and regional OPs

3.Strategic result indicators in the operational programmes – usually taken from the PA, in the regional OPs not limited to the impact of that particular OP.

4.COM forced us to regionalise the Europe 2020 targets – to be finalised in 2015

Result-orientation in the partnership agreement

A coherent approach to result indicators in the PAA coherent approach to result indicators in the PA

PA strategic result indicator: GERD relative to GDP

Baseline value 2012 0,90%

Europe 2020 targetMilestone v. 2020 1,70%

Target value 2023 1,94%

ESIF input into the change

23% Direct impact of TO1 expenditures

ESI Funds’ intervention in 2014-2020 should

result in an increase in growth rate of

expenditure on R&D (esp. In the private

sector)

TO 1

TO 1

REGION BASELINE VALUETARGET VALUE

BENCHMARK COHERENT WITH

THE PA% BENCHMARK

dolnośląskie 0,55 - 1,43 n/a

kujawsko-pomorskie 0,43 0,89 1,02 88%

lubelskie 0,65 0,95 2,45 39%

lubuskie - - 0,46 n/a

łódzkie 0,78 1,10 1,73 64%

małopolskie 1,06 1,45 2,99 48%

mazowieckie 1,37 2,00 2,73 73%

opolskie 0,26 - 0,47 n/a

podkarpackie - - 2,41 n/a

podlaskie 0,44 0,97 0,90 108%

pomorskie 0,73 1,30 2,42 54%

śląskie 0,52 1,30 1,44 90%

świętokrzyskie 0,38 0,48 0,71 67%

warmińsko-mazurskie 0,49 1,14 1,14 100%

wielkopolskie 0,64 0,70 1,97 35%

zachodniopomorskie 0,37 - 0,93 n/a

the ROPs’ input to the PA target is only 1,36% PKB instead the 1,96% set in the PA

PA strategic result indicator: GERD relative to GDP

1. Participation of national and regional authorities in the process of ex ante evaluation of the PA

2. Regular information on the progress in negotiations of the EU legislative package and about, their implications, actions to be taken by the ministry, workshops, etc.

3. Analysis of coherence between the strategic result indicators in the PA and regional OPs (as a part of the PA’s ex ante evaluation process)

4. Analysis of coherence and consistency of performance frameworks across all OPs

5. Explanatory and guiding materials: a. a handbook on programming and implementation of cohesion policy 2014-2020; b. the ministry’s commentary to the OP template – guidance how to fill it in;c. a common list of key indicators;

Result-orientation in the partnership agreement

Translating the PA approach to the national and regional OPsTranslating the PA approach to the national and regional OPs

Common list of key indicators (CLKI)

1. 2007-2013 – lack of coherence across the OPs:a. the same actions/changes measured by different indicators

b. indicators divergent in terms of definition, names, estimation methods, lack of historical data etc.:• limited scope for aggregation of data• limited process of institutional learning• limited possibilities to present the evidence of policy’s effectiveness

c. too many of them and insufficient link with the actions taken

2. 2014-2020 – new challenges:a. programming documents more result-oriented

b. the EU list of common indicators

c. performance framework and reserve = a more rigorous approach to indicators much needed.

Challenges to be addressed in 2014-2020 periodChallenges to be addressed in 2014-2020 period

Common list of key indicators (CLKI)

Common list of key indicators – a coherent set of indicators, complementary to the strategic results indicators of the PA and OPs:

1.Indicators which are gathered from projects and aggregated at the OPs’ level and the PA level using the system of electronic data exchange

2.All EU common indicators for the ERDF, the CF and the ESF + other commonly used indicators:

a. direct result indicators – to be used in tables 3 & 4 of the OP template b. output indicators – to be used in table 5 of the OP template

3.Each indicator with an extensive fiche (name, unit, type, other linked indicators, IP/IPs, definition and measurement method, interpretative notes, category of intervention)

= ca. 350 indicators, 4.A template for each OP on choice and methodology for estimation of all indicators used in it – for internal use only, not to be presented to COM.

What is the CLKI?What is the CLKI?

Result and output indicators in the OPs

1. Its description should clarify how the envisaged types of actions (+ main target groups & areas) will lead to achieving the specific objectives and result indicators.

2. result indicators (tables 3 & 4) – quantifying specific objectives, coherent with the PA, max 1-2 per priority axis. Two types:

a. strategic result indicators – from public statistics/evaluations, baseline 0b. direct result indicators – from the CLKI, OP’s monitoring system, baseline

can’t be 0+ for the ESF – from the CLKI,either a direct result indicator or a long-term

result indicator

3. output indicators (table 5) – quantifying the actions to be taken and the expected change/needs:a. a majority of projects covered by a given IP should generate the output

indicators included in that table. b. derived only from the CLKI, obligatory use of the EU common indicators.

4. COM forced us monitor the number of jobs created by each project.

Indicators at a priority axis levelIndicators at a priority axis level

Performance framework

1.Implications of performance framework and reserve should be taken into account when designing the internal structure of an OP limiting the number of thematic objectives on a priority axis

2.Three elements in the performance framework – an ideal solution adding an extra (controllable!) KIS to have the three elements

•Strike the right balance between performance and minimising the risk

•50% of financial allocation: on the IP level or types of projects within IP?

•How to use of KIS? the link with the output indicator is not compulsory

•A moving target – the draft national OPs were sent to COM in January 2014; changes in the final Partnership Agreement = changes in the national and regional OPs

Construction of a performance frameworkConstruction of a performance framework

Challenges:Challenges:

Performance framework

Setting milestone values for financial indicator:Setting milestone values for financial indicator:

Performance framework

Using the method proposed for setting milestones but slightly modified:

•In point 1 we take simply 100% EU allocation for a given OP instead of the long formula

•Next we go through points 2-4

•MAs are advised to decrease the target value on each priority axes by few % because in 2024 expenditures can still be certified. The decrease below 100% has to be based on historical data regarding absorption level at the end of programming period for a given type of investments.

•Making ends meet – programming teams and financial management teams;•Lack of historical data to estimate milestones and targets;•IPs with one or few projects – what should be done with the reserve?

Challenges:Challenges:

Setting target values for financial indicator:Setting target values for financial indicator:

Performance framework

1.Only output indicators from the CLKI and only those presented in the table 5

2.Ensuring a causal effect:a. between milestones and targets of a given output indicatorsb. between output indicators/KIS and financial indicator (?)

Ensuring consistency across programmes:1.which level is appropriate? – IP or type of projects? The financial indicator relates to the whole allocation of a priority axis BUT the output indicators and KIS relate to more than 50% of the allocation

2.which elements should be consistent? – the type & name of the indicator, EUR exchange rate, estimation methods (unit costs), price dynamics, financial instruments, state aid intensity:

• ERDF/CF fully implemented operations vs. partially implemented in the ESF• Lack of data at regional level (esp. ESF indicators)

Challenges:Challenges:

Setting milestone and target values for output indicators:Setting milestone and target values for output indicators:

Performance framework

1. A template to present the methodology in a standardised manner: a. the necessary data required by the CPR and the IR 215/2014b. the additional information required by the ministry:

• on practical implementation mechanisms which will ensure that the implemented projects will lead to achievement of milestones and targets;

• on possible external factors that may influence the estimation of milestone and target values Article 5(6) of the IR 215/2014

c. it will become an official annex to the OPs.

2. Additional annual milestones for 2016, 2017, 2019-2022a. schedule for the OP implementation:b. monitored on a regular basisc. underperformance leads to an explanation and an action plan

3. On-going evaluation for reallocation of the performance reserve – to be completed in a coordinated manner by all MAs in the 1st quarter of 2019

Additional PL actions:Additional PL actions:

Thank you for your attention