2023 metų balandžio mėnesio 10 diena

Agnė Paliokaitė, Senior Policy Researcher, Public Policy and Management Institute, Lithuania

COHESION POLICY SUPPORT TO INNOVATION IN LITHUANIA:

Lessons Learnt for Evaluation

Presentation for DIRECTORATE-GENERAL REGIONAL POLICY

"EVALUATION NETWORK MEETING"Brussels, 14 April 2011

ROADMAP • Aims and challenges• Approach• Conclusions• Added value, strengths and

limitations

Insights, based on 4 studies carried by PPMI in 2010: - Two system level evaluations for the Knowledge

Economy Forum and the Prime Minister’s Office; - ERAWATCH country report; - SF indicators’ system evaluation for the Ministry

of Finance (RTDI measures case study).

WHY ‘SYSTEM’ EVALUATION?• System (portfolio) evaluation – evaluating policy portfolios,

not individual programmes.• Retrospective: new political will put innovation high on the

political agenda in 2009/2010. New ideas - need for revisiting the incrementally developed policy mix.

• Prospective: need for rethinking the future priorities in the context of ‘Progress Strategy Lithuania 2030’ and the new 2014-2020 Structural Funds period.

AIMS OF SYSTEM EVALUATION

• To analyse the extent to which SF funded innovation policy portfolio/mix reflects specific conditions and levels of the National Innovation System (NIS).

• To analyse how the financial proportions fit to the policy agenda (the preferred ‘routes’).

• To present preliminary insights on effectiveness in achieving set targets.

• To draw conclusions on governance & monitoring system.

sdafasdfasdfasdf

2023 metų balandžio mėnesio 10 diena

EVALUATION FRAMEWORK

Hypotheses about bottlenecks Conclusions

Hypotheses about bottlenecks Conclusions

1. Innovation system ‘health’: market, capability, institutional, network, system,

and governance failures

1. Innovation system ‘health’: market, capability, institutional, network, system,

and governance failures

2. Intervention logic and policy mixes2. Intervention logic and policy mixes

3. Extent to which outputs and results are achieved, critical factors

3. Extent to which outputs and results are achieved, critical factors

Innovation Policy and

Governance development

Innovation Policy and

Governance development

Based on: Arnold E. Evaluating research and innovation policy: a systems world needs systems evaluations, Research Evaluation, volume 13(1), 2004

CHALLENGES AND LIMITATIONS

• Timing: low absorption of funds at the time of evaluation (most measures started operation in 2009-2010).

• Small scale evaluations. Hence, inability to apply quantitative approach. • A ‘moving object’: innovation policy and governance reform

(LIS 2010-2020, SITA); changes in the system of SF objectives. • Inability to rely on the system of quantitative indicators .• Innovation policy specific: M&E exceptionally difficult for

innovation programmes: inherently qualitative and diffuse nature of innovation benefits. Long cause-effect chain.

QUALITATIVE APPROACH

DATA COLLECTION• Semi structured interview

programme with stakeholders and target groups (~30 in total);

• Desk research: literature review, secondary and administrative data;

• Expert panels (focus groups);• Triangulation principle applied

for avoiding subjectivity and partiality of the data as well as guaranteeing impartial conclusions.

ANALYTICAL TOOLS• Assessment of the innovation system and

the RTDI policy mix using the ‘system failures’ framework;

• Logical models and reconstruction of the policy intervention logic;

• Meta-analysis of previously carried out studies and analysis of trends in the theoretical debate;

• Data integrating methods: scenarios and road-mapping;

• Comparative analysis / benchmarking of other countries’ experience;

• Risk analysis, critical factors and analysis of policy options.

sdafasdfasdfasdf

2023 metų balandžio mėnesio 10 diena

RESULTS 1: INTERVENTION LOGIC 2007-2013

Younger researchers

More researchers

Public R&D infrastructure quality

and access to business

Higher R&D collaboration between

public and private sectors

Higher public R&D potential and capacity

Higher researchers

mobility

Higher private sector R&D capacity and

potential

Higher value added in the economy

Higher private R&D investments

Better qualified researchers

More and better researchers in public sector

Better innovation

support services

Stronger clusters More

researchers in

business

Better private R&D

infrastru

cture

More business

R&D projects

ESF ERDF

RESULTS 2: NIS ‘HEALTH’ ASSESSMENT

0

0.5

1

1.5

2

2.5

3Financial resources

Absorptive capacity

Clusters vitality

SME establishment

Know ledge infrastructure and quality

Governance quality

Innovation support services

Netw orking

General framew ork conditions

Incentives for innovation andentrepreneurship

Innovation demand

Capacities in f irms

Needs Relevance of policy mix

• Market failure (productive sector)

• Institutional failure (knowledge infrastructure)

• Capabilities failure• Networking failure• Framework

conditions• Governance failure• Demand side

(absorptive capacity)

sdafasdfasdfasdf

2023 metų balandžio mėnesio 10 diena

F

irm

s

N

GO

s

HE

IS, P

RIs

Strengthening public R&D system Public private R&D collaboration-

Investments in private R&D base

Investments into productivity

R&D in

business, € 162.2m

Clusters & innovation

support services

€ 92.7mResearchers in business:

€ 9.3m

RTDI Networks

€ 6.23m

Direct support to companies, € 732.4m (MoE):

-Access to capital (€ 415m )

- process inovelties (€118.2m)

-E-business, investments into production technologies

“Valleys”, national complex programmes

€ 678,6m (MoES)

Source: PPMI, Knowledge Economy Forum, 2010

• Heavily expanding and versatile, but ‘linear’ logic persists’.

• Mainly follows two routes: (1) to strengthen public R&D base, and (2) to invest in R&D in R&D performing firms. Lack of critical mass to implement some objectives placed high on political agenda (e.g. R&D collaboration).

RESULTS 3: POLICY MIX ‘ROUTES’

EVALUATION OF SF MONITORING SYSTEM

• Quantitative (statistical analysis) as well as qualitative (logical models and consensus building activities).

• SMART framework (specific, measurable, achievable, timed..)

‘HORIZONTAL’ EVALUATION

‘VERTICAL’ EVALUATION

~ 1000 indicators

~ 150 indicators

INDICATORLEVEL OF ACHIEVEMENT

BEFORE 2015REMARKS

1 OBJECTIVE: TO STRENGTHEN PRIVATE AND PUBLIC R&D BASE

PRIVATE INVESTMENTS (million EUR) - R LOWOBSTACLES FOR PRIVATE INVESTMENTS, THUS THIS

INDICATOR CAN ONLY BE APPLIED AT IMPACT LEVEL

R&D CENTRES CREATED AND FUNCTIONAL – R HIGH NO THREATS

NUMBER OF R&D BASE DEVELOPMENT PROJECTS– P

MEDIUM

INDICATOR ACHIEVED WILL BE TWICE LOWER AS PLANNED, HOWEVER THIS DOES NOT REFLECT THE REAL DECREASE OF ALLOCATED RESOURCES (-300 MILLION EUR FROM THE “VALLEYS” TO FINANCIAL ENGINEERING MEASYRES).

2 OBJECTIVE - TO INCREASE PUBLIC SECTOR R&D EFFECTIVENESS AND ACCESSIBILITY TO COMPANIES

NUMBER OF GENERAL WORK PLACES CREATED IN THE R&D SECTOR - R

MEDIUM NEW MEASURES ARE BEING CREATED

NUMBER OF COOPERATION CONTRACTS SEIGNED BETWEEN PUBLIC AND PRIVATE SECTOR INSTITUTIONS- R

HIGH NO THREATS

NUMBER OF R&D PROJECTS- P LOWNEW MEASURES TO ENSURE ACHIEVEMENT OF THE

INDICATOR VALUE ARE BEING CREATED

3 OBJECTIVE: TO INCREASE R&D ACTIVITY IN PRIVATE SECTOR

PRIVATE INVESTMENTS (million EUR) - R HIGH NO THREATS

NUMBER OF R&D PROJECTS (R&D ACTIVITY IN COMPANIES) – P

HIGH NO THREATS

4 OBJECTIVE – TO INCREASE BUSINESS AND SCIENCE COLLABORATION, INTENSIFY THE KNOWLEDGE FLOWS

NEW BORN TECHNOLOGY INTENSIVE COMPANIES- R

HIGH THIS IS AN IMPACT LEVEL INDICATOR

R&D AND INNOVATION ENVIRONMENT IMPROVEMENT PROJECTS- P

HIGH NO THREATS

KEY CONCLUSIONS1. Structural gap: Lack of innovation absorptive capacity in business and

society; limited local market: the key barrier to knowledge intensive firms.

2. Policy myopia 1: Excessive focus on supply side measures and on ‘supporting the winners’ can be contradictory to the systemic characteristics of NIS

3. Policy myopia 2: Quantitative targets will be met 99 percent, but it does not mean achievement of qualitative objectives.

4. Risk-averse approach to implementation due to limited capacity to evaluate innovation projects.

5. Hypothesis: only a minor part of economy benefits from innovation measures. Financially marginal “soft” measures are important for behavioral additionality: project pipeline building, innovation brokering

Governance allowing quality ideas entering the ‘market’: o Boosting capability to develop RTDI policy, strengthening project

and programme level intelligence; novel approaches to funding; a stronger involvement of users in evaluation and funding.

Policy as a discovery process:

• Promoting innovative, risky, flexible, “bottom-up” approaches; project pipeline building.

• Empowering people to innovate (‘bottom-up’), and demand side: procurement, regulation, clusters along the value chain, networks around societal problems.

RECOMMENDATIONS FOR INNOVATIVE POLICY

STRENGTH AND LIMITS OF APPROACH

• Strengths: Focus on the NIS bottlenecks as opposed to the mechanical transfer of policy models that may not be the most relevant for the NIS. Allows for internal coherence and looking beyond the quantitative input/output indicators. From macro to micro level analysis (focus on important details).

• Limitations of qualitative approach: lack of ‘hard’ data and evidence (e.g. as opposed to counterfactual analysis) for tracing the real change and explaining obtained effects. Object for the following evaluations.

• Recommendation for following evaluations: look for behavioural additionality (knowledge spillovers, changes in innovation process related behavioural patterns, interaction additionality, etc.), quantifying impact of networks

THANK YOU FOR ATTENTION!

2023 metų balandžio mėnesio 10 diena

Agnė Paliokaitė

Senior Policy Researcher

Public Policy and Management Institute+37061690469

agne@vpvi.lt

www.vpvi.lt