World Bank Document · 2016. 10. 8. · The national innovation system and composition of research and development ( R&D) funding in 2012 did not support enterprise innovation at

Document of

The World Bank

Report No: ICR00003911

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(TF071742)

ON A

GRANT

IN THE AMOUNT OF €8.4 MILLION

(US$11.9 MILLION EQUIVALENT)

TO THE

REPUBLIC OF SERBIA

FOR THE

INNOVATION SERBIA PROJECT

October 8, 2016

Trade and Competitiveness Global Practice

Europe and Central Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective July 6, 2016)

Currency Unit = Euro (€)

€1.00 = US$1.09

US$1.00 = €0.917

FISCAL YEAR

July 1 – June 30

ABBREVIATIONS AND ACRONYMS

C&JP Competitiveness and Jobs Project

CFP Call for Proposals

EPO European Patent Office

ENIF Enterprise Innovation Fund

EU European Union

EUD Delegation of the European Union

EU IPA European Union Instrument for Pre-Accession

FINS Institute of Food Technology Novi Sad

FM Financial Management

GDP Gross Domestic Product

GM Grant Manual

GoS Government of Serbia

GRA Global Research Alliance

IC Investment Committee

ICR Implementation Completion and Results Report

ICT Information and Communication Technology

IF Innovation Fund

IFVC Institute of Field and Vegetable Crops

IMGGE Institute of Molecular Genetics and Genetic Engineering

IMPR Institute of Medicinal Plants Research

IP Intellectual Property

IPA Instrument for Pre-Accession

IPB Institute of Physics, Belgrade

MOERD Ministry of Economy and Regional Development

MOSTD Ministry of Science and Technological Development

MOESTD Ministry of Education, Science and Technological Development

MPI Mihajlo Pupin Institute

MRI Maize Research Institute

PAD Project Appraisal Document

PDO Project Development Objective

R&D Research and Development

RDI Research and Development Institution

SIP

SME

Serbia Innovation Project

Small and Medium Enterprise

SRITTP Serbia Research, Innovation and Technology Transfer Project

TA Technical Assistance

TTF Technology Transfer Facility

VC Venture Capital

WB EDIF Western Balkans Enterprise Development and Innovation Facility

Senior Global Practice Director: Anabel Gonzalez

Sector Manager: Paulo Guilherme Correa

Project Team Leader: Natasha Kapil

ICR Team Leader: Natasha Kapil

SERBIA

Innovation Serbia Project

CONTENTS

Data Sheet

A. Basic Information………………………………………………………………….i

B. Key Dates………………………………………………………………………….i

C. Ratings Summary……………………………………………………………….....i

D. Sector and Theme Codes………………………………………………………......ii

E. Bank Staff………………………………………………………………………….ii

F. Results Framework Analysis……………………………………………………...iii

G. Ratings of Project Performance in ISRs………………………………………......vi

H. Restructuring ……………………………………………………………………..vi

I. Disbursement Graph……………………………………………………………...vii

1. Project Context, Development Objectives and Design ............................................... 1

2. Key Factors Affecting Implementation and Outcomes ............................................ 10

3. Assessment of Outcomes .......................................................................................... 16

4. Assessment of Risk to Development Outcome ......................................................... 27

5. Assessment of Bank and Recipient Performance ..................................................... 28

6. Lessons Learned ....................................................................................................... 30

7. Comments on Issues Raised by Recipient/Implementing Agencies/Partners .......... 31

Annex 1. Project Costs and Financing .......................................................................... 32

Annex 2. Outputs by Component ................................................................................. 33

Annex 3. Economic and Financial Analysis ................................................................. 40

Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 41

Annex 5. Beneficiary Survey Results ........................................................................... 43

Annex 6. Stakeholder Workshop Report and Results ................................................... 47

Annex 7. Summary of Recipient's ICR and/or Comments on Draft ICR ..................... 48

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 50

Annex 9. List of Supporting Documents ...................................................................... 51

MAP …………………………………………………………………………………..52

i

Datasheet

A. Basic Information

Country: Serbia Project Name: Innovation Serbia

Project

Project ID: P126229 L/C/TF Number(s): TF11257

ICR Date: 10/08/2016 ICR Type: Core ICR

Lending Instrument: TAL Borrower/Recipient: Government of Serbia

Original Total

Commitment: US$9.52 million Disbursed Amount: US$8.35 million

1

Revised Amount: US$7.15 million

Environmental Category: B

Implementing Agencies: Ministry of Education, Science and Technological Development

(MOESTD); Serbia Innovation Fund (IF)

Co-financiers and Other External Partners: The Donor is the EU Delegation in Serbia

B. Key Dates

Process Date Process Original Date Revised/Actual

Date(s)

Concept Review: 05/05/2011 Effectiveness: 12/08/2011

Appraisal: 06/01/2011 Restructuring(s): 09/05/2014

Approval: 11/23/2011 Mid-term Review: 08/25/2014 08/25/2014

Closing: 11/30/2014 01/10/2016

C. Ratings Summary

C.1 Performance Rating by ICR

Outcomes: Satisfactory

Risk to Development Outcome: Low or Negligible

Bank Performance: Satisfactory

Recipient Performance: Satisfactory

1 This system generated datasheet only reflects Recipient Executed disbursement amounts and

does not reflect Bank Executed components or fees associated with this hybrid trust fund. This

table would suggest significant undisbursed and cancelled resources at the end. However, only

Euro 655,000 were cancelled at project closing. Differences between the Revised amount and the

Disbursed Amount are most likely explained by the Euro vs. US$ exchange rate differential

between appraisal and closing.

ii

C.2 Detailed Ratings of Bank and Recipient Performance (by ICR)

Bank Ratings Recipient Ratings

Quality at Entry: Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Highly Satisfactory Implementing

Agency/Agencies: Highly Satisfactory

Overall Bank

Performance: Satisfactory

Overall Recipient

Performance: Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation

Performance Indicators

QAG Assessments

(if any) Rating

Potential Problem Project

at any time (Yes/No): No

Quality at Entry

(QEA): None

Problem Project at any

time (Yes/No): No

Quality at Entry

(QEA): None

DO rating before

Closing/Inactive status: Satisfactory

Quality of

Supervision (QSA): None

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Public administration - Financial Sector 40 40

Public administration - Industry and Trade 10 10

Tertiary Education 50 50

Theme Code (as % of total Bank financing)

Technology diffusion 100 100

E. Bank Staff

Positions At ICR At Approval

Vice President: Cyril E. Muller Philippe H. Le Houerou

Country Director: Ellen Goldstein Jane Armitage

Sector Manager: Paulo Guilherme Correa Lalit Raina

Project Team Leader: Natasha Kapil Natasha Kapil

ICR Team Leader: Natasha Kapil N/A

ICR Primary Author: Hiran Herat N/A

iii

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The development objective of the project is to assist in building the institutional capacity to

stimulate innovative activities in the enterprise sector by:

(a) Supporting the operationalization of the Serbia Innovation Fund (IF);

(b) Piloting financial instruments for technological development and innovation in

enterprises; and

(c) Encouraging selected research and development institutes (RDIs) to engage in

technology transfer and commercialization, and assisting in formulating RDI sector

reform policy.

Revised Project Development Objectives (No changes, as approved by original approving

authority

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : Number of financial instruments designed and introduced

Value

(quantitative

or Qualitative)

0 3 4

Date achieved 12/05/2011 08/01/2014 08/01/2014

Comments

(incl. %

achievement)

Target exceeded by 33%

Indicator 2 : Number of IF managers trained and applying newly learnt skills

Value

(quantitative

or Qualitative)

0 4 10

Date achieved 12/05/2011 08/01/2014 12/11/2015

Comments

(incl. %

achievement)


Indicator 3 : Number of start-ups mentored

Value

(quantitative

or Qualitative)

0 10 51

Date achieved 12/05/2011 08/01/2014 12/11/2015

Comments

(incl. %

achievement)


Indicator 4 : Mechanism to support training/mentoring of innovative start-ups established

Value

(quantitative No Yes Yes

iv

or Qualitative)

Date achieved 12/05/2011 04/30/2013 04/30/2013

Comments

(incl. %

achievement)

Target met.

Indicator 5 : Number of networking and educational events organized for enterprises, academia and

research

Value

(quantitative

or Qualitative)

0 8 22

Date achieved 12/05/2011 08/01/2014 12/11/2015

Comments

(incl. %

achievement)


Indicator 6 : Number of funded projects

Value

(quantitative

or Qualitative)

0 20 52

Date achieved 12/05/2011 04/30/2013 04/30/2013

Comments

(incl. %

achievement)


Indicator 7 : Number of funded enterprises

Value

(quantitative

or Qualitative)

0 15 48

Date achieved 12/05/2011 08/01/2014 08/01/2014

Comments

(incl. %

achievement)


Indicator 8 : Number of RDIs identified for technology transfer and commercialization support

Value

(quantitative

or Qualitative)

0 4 10

Date achieved 12/05/2015 08/01/2014 08/01/2014

Comments

(incl. %

achievement)


Indicator 9 : Number of RDIs assessed jointly by international experts & RDI management

Value

(quantitative

or Qualitative)

0 2 4

Date achieved 12/05/2011 08/01/2014 08/01/2014

Comments

(incl. %

achievement)


v

No. Date ISR

Archived DO IP

Actual

Disbursements

G. Ratings of Project Performance in ISRs

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised

Target Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : Amount of innovation financing mobilized by IF in addition to this EU IPA project

Value

quantitative or

Qualitative)

0 EUR 20 million

WB EDIF EUR 28

million

SRITTP Project EUR

6.9 million

Date achieved 12/05/2011 01/08/2014 12/11/2015

Comments

(incl. %

achievement)


Indicator 2 : Number of active technology start-ups funded

Value

quantitative or

Qualitative)

0 10 38

Date achieved 12/05/2011 04/08/2015 12/11/2015

Comments

(incl. %

achievement)


Indicator 3 : Number of new products and processes launched by beneficiary enterprises

Value

quantitative or

Qualitative)

0 8 19

Date achieved 12/05/2011 08/01/2014 11/30/2014

Comments

(incl. %

achievement)


Indicator 4 : Number of technologies transferred by RDIs

Value

quantitative or

Qualitative)

0 3 3

Date achieved 12/05/2011 08/01/2014 01/04/2016

Comments

(incl. % achievement) Target met.

Indicator 5 : Dissemination of policy recommendations for RDI sector reform

Value

quantitative or

Qualitative)

No strategy/Policy

exists

Disseminated to

stakeholders

Disseminated to

stakeholders

Date achieved 12/05/2011 08/01/2014 01/04/2016

Comments

(incl. % achievement) Target met.

vi

(US$, millions)

1 05/28/2012 Satisfactory Satisfactory 1.03



4 04/05/2014 Moderately Satisfactory Moderately Satisfactory 5.59




H. Restructuring (if any)

Restructuring

Date(s)

Board

Approved

PDO Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

in US$,

millions

Reason for Restructuring &

Key Changes Made DO IP

September 5,

2014 No S S 6.36

Closing date extended from

November 30, 2014 to January

10, 2016. The extension

allowed IF to complete core

program activities, including

contracting of the new awardees

from the fourth call for

proposals as well as disbursing

earlier approved funds to the

existing awardees.

I. Disbursement Profile2

2 This table would suggest significant undisbursed and cancelled resources at the end. In fact,

there was a small cancellation of remaining funds at Closing, equaling Euro 655,000. This chart

is explained by the exchange rate differential at appraisal and closing of the Euro vs. US$.

1

1. Project Context, Development Objectives and Design3

1.1 Context at Appraisal

1. The national innovation system and composition of research and development

(R&D) funding in 2012 did not support enterprise innovation at any significant level, and

the research sector modernization agenda was in its infancy. The Serbian R&D sector was

dominated by the public sector, largely inefficient, and most importantly delinked from industry

needs, particularly those of local small and medium enterprises (SMEs). The Serbian industry’s

capacity, both in terms of human capital and financial resources for R&D and innovation, had

been severely weakened post-transition and exacerbated by the lingering effects of the financial

crisis. Brain drain was a major concern, in the private sector and even more so in the scientific

community.

2. Serbia spent roughly €100 million (0.3 percent of gross domestic product [GDP] in

2010) on R&D, with a goal to reach 1 percent by 2014. While R&D spending had increased

from €28 million in 2001, Serbia still lagged significantly behind its neighbors. Slovenia, the

Czech Republic, and Croatia spend more than 1 percent of GDP, and the European average was

1.8 percent. In Serbia, the majority of R&D spending went to basic research, which accounted

for 50 percent of all R&D funding. A significant shift from basic to applied research was needed

for research commercialization and development of an innovative SME sector. This, along with

developing stronger connections between science and industry, was to be key in raising R&D

expenditures to 2 percent of GDP by 2020, with a target of leveraging half of this from the

private sector.

3. Outputs from the R&D sector were not commensurate with the public resources

being invested and did not support modernization of the Serbian economy. While the

number of scientific publications had increased during the previous few years, quality remained

poor. Intellectual property (IP) was either not being created or not being protected. Merely 21

patents were registered by public research and development institutes (RDIs) from 2003 to 2008,

with only 36 patent applications, mostly in Serbia. Figures for the private sector were similar.

4. In an attempt to modernize the national innovation system, the authorities had

implemented a reform program over the previous two years to revamp the regulatory and

institutional framework governing science, technology, and innovation. It had enacted a

number of laws to promote science and innovation and formulated the Serbian Scientific and

Technological Development Strategy 2009–2014. In addition, the Ministry of Education and

Science was created as the central body responsible for science and innovation in Serbia.4

3The PDO was not revised over the course of the Project.

3 This section is based on the Project Appraisal Document (PAD).

4 The March 2011 restructuring of the Government of Serbia (GoS) led to a merger of the Ministry of Science and

Technological Development (MOSTD) with the Ministry of Education, thus creating a new entity, the Ministry of

Education, Science and Technological Development (MOESTD). The innovation and technology development

priorities have remained unchanged as a result of restructuring. The review of programs described herein was

prepared by MOSTD.

2

5. Science infrastructure improvement had been the main focus to that point, while

stimulation of private sector-led R&D and innovation had been negligible. The European

Investment Bank provided a loan of €200 million to upgrade infrastructure at universities and

RDIs and to set up science parks as well as Centers of Excellence in priority areas including

biotechnology, nanotechnology, and advanced computing. In 2010, the Government disbursed

approximately €70 million for basic research, technological development, integral and

interdisciplinary research, and innovation activities. However, almost 80 percent of the funds

intended for science projects actually went to salaries for researchers.

6. RDIs were the primary recipients of financing, with private firms receiving funding

only under the technological development component. The 471 technological development

projects supported by the MOSTD were led by RDIs and not by industry. Public RDIs depended

on these projects as they did not receive regular institutional budget allocations. Projects

generally had a 90 percent approval rate, creating disincentives for researchers to innovate.

Except in a few cases, technology transfer and diffusion from RDIs was quite low with only a

few spin-offs and real technology-based start-ups, indicating the need to place greater emphasis

on supporting the various stages of technology development and innovation within enterprises—

for example, technology transfer, commercialization, diffusion, absorption, adaption, and

application.

7. Few mechanisms to incentivize private sector R&D and innovation had been

implemented to that point in time. The Ministry of Economy and Regional Development

(MOERD) administered the Program to Strengthen Innovation in SMEs, that is, matching grants,

to support SME investment in innovation through co-financing. However, the total budget for

2010 was RSD 40 million (€380,000). All legal entities were eligible for co-financing of up to 50

percent of justified expenses for innovation activity. Matching funds had to be provided from the

SME’s own resources and could not include any other public support. The awarded amounts

could not exceed RSD 800,000 (€7,500) for the first group of eligible activities, and RSD 1.5

million (€14,000) for the second group. In addition, under the Instrument for Pre-Accession

(IPA) 2010 program, €3 million was allocated for the Integrated Innovation Support Program

with the objective to ‘increase the competitiveness and economic growth in Serbia, through

strengthening of innovation in SMEs in accordance to National Strategy for Development of

Competitive and Innovative SMEs 2008–2013.’ However, implementation of this program had

not yet started.

8. In the post-crisis environment, the Government was expected to play a critical role

in stimulating economic recovery through policies that target firm-based R&D and

innovation. While the Government and other international donors had demonstrated efforts to

support R&D and innovation, these were focused on renewing infrastructure investments in

public RDIs with negligible support for private sector innovation. Given the World Bank’s

experience garnered in supporting private sector development in post-transition economies,

including in neighboring Croatia, the GoS was keen on a World Bank engagement on the

innovation agenda. Hence, the World Bank’s technical assistance (TA) to the GoS and to the

future Innovation Fund (IF), being established under the project as the implementing agency,

focused on filling these gaps and developing mechanisms to stimulate enterprise-led technology

development and innovation.

3

9. The objective of the FY12-15 CPS was to support Serbia’s EU accession and help

the Government strengthen competitiveness and improve the efficiency and outcomes of

social spending. Under the Strengthening Competitiveness pillar Improved innovation capacity

was identified as one of objectives. The CPS noted that the system and composition of R&D

funding did not support Serbia’s agenda to modernize and enhance its competitiveness. The CPS

sought to change this situation by supporting the establishment of institutional capacity to

stimulate activities in the enterprise sector. The capacity would arise from creating a Serbia

Innovation Fund, using financial instruments for innovation and technological development in

enterprises, and having R&D institutes engaged in transferring and commercializing technology

and in helping formulate RDI reform policy.

1.2 Original Project Development Objectives (PDO) and Key Indicators

10. The project development objective (PDO) was to assist in building the institutional

capacity to stimulate innovative activities in the enterprise sector by:

(a) Supporting the operationalization of the Serbia Innovation Fund (IF);

(b) Piloting financial instruments for technological development and innovation in

enterprises; and

(c) Encouraging selected research and development institutes (RDIs) to engage in

technology transfer and commercialization, and assisting in formulating RDI sector

reform policy.

11. The key performance indicators for this project included:

(a) Amount of innovation financing mobilized in addition to this European Union

Instrument for Pre-Accession (EU IPA) project;

(b) Number of active technology start-ups funded;

(c) Number of new products and processes launched by beneficiary enterprises;

(d) Number of technologies transferred by RDIs; and

(e) Dissemination of policy recommendations for RDI sector reform.

1.3 Revised PDO and Key Indicators

12. The PDO was not revised. PDO-level indicators 1 and 2 were refined for clarity’s sake

during negotiations with the donor and recipient, and are fundamentally the same as in the PAD.

1.4 Main Beneficiaries

13. The primary beneficiaries of this pilot project were the staff of the IF whose capacity was

built under Component 1; entrepreneurs, start-ups, and firms that received financing and

mentoring/training under Component 2; researchers and leadership of select RDIs that benefited

4

from exposure to modern practices in IP management, technology transfer, and

commercialization, as well as the Ministry of Education, Science and Technological

Development (MOESTD) that also benefited from the policy recommendations developed based

on the RDI TA program under Component 3. Important secondary beneficiaries included

stakeholders engaged during project launch and implementation through numerous open

houses/workshops/conferences on such topics as entrepreneurship, research commercialization,

or innovation finance. These stakeholders ranged from across Serbia’s national innovation

system, including leading universities, RDIs, incubators, technology transfer offices, nascent

early stage investors, and private sector actors.

1.5 Original Components

14. In order to build institutional capacity to stimulate innovative activities in the enterprise

sector, the project design intended to leverage three complementary components that would

enable piloting of institutional, financial, and commercialization support mechanisms absent

from the national innovation ecosystem with the view to support and ideally demonstrate

Serbia’s enterprise innovation potential.

Component 1: Capacity Building of the Serbia Innovation Fund - Recipient-Executed (€1.1

million)

15. The capacity of the IF was being established to encourage entrepreneurship, including by

financing enterprise innovation, as well as participating in long-term programming efforts by

national authorities, international organizations, financial institutions, and the private sector. In

this context, this component provided support for:

(a) operations of the Investment Committee and due diligence of grant applications;

(b) engagement of strategic and operations advisors to assist the IF’s current and future

programs, strategy, operations, and procedures;

(c) operational project management in relation to the European Union (EU) and World

Bank;

(d) IF staff training;

(e) IF beneficiary enterprise mentoring and training;

(f) project visibility events and workshops; and

(g) IF operational infrastructure, including information and communication technology

(ICT) and financial management (FM) development.

Component 2: Implementation of Financial Instruments Supporting Enterprise Innovation

- Recipient-Executed (€6.0 million)

16. This component financed (a) Mini Grants targeting the proof of concept and prototyping

stages, including but not limited to IP protection and business plan preparation for initial capital

5

mobilization; and (b) Matching Grants targeting R&D in technology development projects, for

new or improved technologies, products and processes. Incorporated entrepreneurs, innovative

start-ups, spin-offs, micro and small enterprises, with majority Serbian private sector ownership

were eligible for Mini and Matching grants. The Mini and Matching Grants applications were

evaluated by the IF’s independent Investment Committee (IC), with input from international peer

reviewers.

17. The resources under this component were earmarked notionally per instrument with the

objective of flexible reallocation among instruments based on demand. The terms and conditions

of the grant programs were designed with the IF advisors with sufficient flexibility to

accommodate changing market conditions and the quality and volume of the project pipeline. A

guiding principle was to decrease the matching contribution from the IF over time, particularly

for the Matching Grants instrument.

Table 1. Design Features of the Mini Grant Programs at Appraisal

Mini Grants

Baby Seed €1,500,000–€3,000,000

Objective and stage Proof of concept, prototyping stage, IP protection, business plan preparation for

mobilization of initial capital

Recipient Incorporated entrepreneurs, innovative startups, spin offs, micro and SMEs, all

with majority Serbian private sector ownership

Grant size Up to €80,000 for projects that will be completed within 12 months

Features Up to 85% of approved project costs

Table 2. Design Features of the Mini Grant Programs at Appraisal

Matching Grants €3,000,000–€4,500,000

Objective and stage R&D (technology development) and commercialization projects for new or

improved technologies, products, and processes

Recipient Incorporated entrepreneurs, innovative startups, spin offs, micro and SMEs, all

with majority Serbian private sector ownership

Grant size Up to €300,000 for projects that will be completed within 2 years.

Features Up to 75% Matching Grant with an optional 3–6% royalty component based on

sales revenue, up to 120% of the original grant within a predetermined time

frame

Component 3: Provision of Technical Assistance to Research and Development Institutes

(RDIs) - Bank-Executed (€0.7 million)

18. Component 3 was to provide (a) customized TA to up to two RDIs based on a detailed

needs assessment; and (b) limited TA to up to four RDIs based on a general needs assessment;

and (c) technical input to the Government’s future RDI sector reform program based on lessons

learned from the aforementioned TA program.

19. A customized TA program for up to two select RDIs entailed conducting a detailed

assessment to be carried out by international experts in partnership with the RDIs’ management.

The assessment was to cover issues, including but not limited to, the organization’s management

structure, human and budgetary resources, researcher promotion and incentive system,

infrastructure and laboratory facilities, research capabilities and outputs, IP protection,

challenges and potential for applied R&D, and technology transfer and commercialization. Based

6

on this diagnosis, technical support was to be provided for the design and implementation of

RDI-specific upgrading programs. Upgrading of labs and other infrastructure would not be

supported. The RDIs were identified jointly by MOESTD and the World Bank based on several

key factors, including the level of commitment to reform from the RDI management team,

potential for improvement, likelihood of success, sectoral coverage, and availability of resources.

The lessons derived from this program were expected to inform future strategic planning and

reforms in the RDI sector. Specifically, the recommendations were expected to be focused and

actionable, while identifying quick wins. The World Bank team was to organize a workshop with

the ministry in charge to disseminate the experience, lessons learned, and recommendations for a

future RDI system reform program.

1.6 Revised Components

20. There were no revisions to the components.

1.7 Other Significant Changes

21. There were no significant changes. A simple project restructuring was conducted with

approval from the management of the Europe and Central Asia Region and the donor to extend

the closing date from November 30, 2014 to January 10, 2016. The extension allowed the IF to

complete core activities, such as contracting of awardees from the fourth call and disbursing

previously approved funds to awardees.

22. The US$ appreciated almost 30 percent against the EURO during project implementation.

The US dollar amount at project approval was $11.9 million. The final disbursement amount –

both Bank and Recipient Executed - was approximately US$ 9.7 million, with disbursement via

the Recipient amounting to US$8.35 million. Of the total EURO 8.4 million, EURO 655,000

was cancelled. Fortunately, the exchange rate differential in dollars did not affect

implementation as the grant and expenditures were in Euros.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

(a) Soundness of the Background Analysis

23. The background analysis for this project was sound, given its pilot nature and little

government experience in supporting enterprise innovation or RDI sector reforms. The project

was aligned with the FY08–FY11 Country Partnership Strategy, wherein a key priority included

encouraging dynamic private sector-led growth to promote convergence with European levels. In

FY09 and FY10, the World Bank team delivered a TA program, supporting MOESTD to update

legislative frameworks that encouraged firm uptake of innovation support. Specifically, the

World Bank team worked on identifying challenges with existing innovation programs,

reviewing the legal framework governing R&D and innovation, and a new innovation strategy.

Further, at entry, the focus was on (a) appraising institutional arrangements supporting research

and innovation; (b) surveying and assessing firm perception of existing innovation and

technology absorption support programs and appetite for proposed instruments; and (c)

discerning level of opposition for support to firms and for research sector reform. The analysis

and consultations were instrumental in sharing regional and global experience, building

7

stakeholder confidence on the need to deploy a pilot project that would initiate a steady transition

to an enterprise-led model.

(b) Assessment of the Project Design

24. Pilot. The project was designed carefully as a modest pilot initiative that aimed at

building the missing institutional capacity in the Serbian innovation system, testing financial

instruments and mentoring mechanisms to reveal the emerging innovative entrepreneurial sector,

and identifying viable approaches to the challenges of Serbia’s ailing research sector. Given high

expectations from this pilot, the risky nature of both demand- and supply-side interventions,

accompanied by significant latent opposition from researchers in the public RDIs, the split

between World Bank and recipient-executed components was prudent. Equally judicious was the

attraction of top global expertise to build credibility for this pilot, and demonstration of key

principles in innovation policy management, including horizontality, independence, good

governance, efficiency, transparency, monitoring and evaluation (M&E), and open

communication with stakeholders.

25. Selection Procedures. Noteworthy design features included the IF’s efficient and

independent grant selection processes that engaged international peer reviewers and an IC

featuring international and diaspora experts with a background in scientific research,

entrepreneurship, technology commercialization, market trends, and venture investments.

26. Visibility. The donor placed significant emphasis on organizing project visibility

activities and these ended up being instrumental given the risky and novel nature of the pilot.

Most visibility activities were high level in nature and co-hosted by the recipient, the donor, and

the World Bank, which helped reinforce the Government’s commitment in spite of multiple

ministerial reshufflings within the lifetime of the project.

27. Ownership. The World Bank-executed TA component was designed to support

knowledge transfer and research commercialization within the lifetime of the project. Only those

RDIs where management demonstrated (a) willingness to be assessed by international expert

teams convened for their institution and (b) commitment to act on key recommendations from

such assessments, were included in the RDI TA program. This allowed the World Bank team to

narrow down the list of RDIs it chose to provide TA support. In retrospect, this important design

feature afforded flexibility, as both time and financial resources were limited.

(c) Adequacy of Government’s Commitment

28. The Government’s commitment toward the project—and especially for the enterprise

innovation objectives—was high at entry. The Government demonstrated this by pursuing the

Delegation of the European Union (EUD) to earmark EU IPA funding for enterprise innovation

for the very first time, approving a notional operational budget for the IF from 2011 to 2015, and

attracting a senior diaspora member to lead the IF. The World Bank was nominated by the GoS

to enter into a trust fund arrangement and execute Component 3, which was controversial at the

time.

8

(d) Assessment of Risks

29. Several risks were identified during project preparation and adequate risk mitigation

measures were incorporated into the project design as follows:

A pioneer on several fronts, there were important coordination and

reputational risks stemming from this pilot. This was the first ever EU-financed

World Bank-administered trust fund in Serbia, addressing unchartered policy

territory, facing skepticism by entrenched interests and significant political

instability. Through meticulous supervision and dialogue with the EUD, MOESTD,

MOERD, IF, and other stakeholders during missions and steering committee

meetings, the World Bank fostered shared awareness on project progress and

ownership of challenges as they arose.

Significant resistance from entrenched interests to RDI sector reforms. The

World Bank team managed the RDI TA program closely, expanding the scope of

assistance only to volunteer RDIs where management demonstrated firm

commitment to optimization and knowledge transfer and research commercialization

outcomes.

The IF had no demonstrable capacity to transparently select projects, manage

FM, procurement, environmental screenings, or support the private sector. The

World Bank’s no objection to the selection of IC members, and to procedures

followed during the first call, supported the credibility of the selection processes.

The international advisors provided continuous capacity building for IF staff on

program design couched in global practice, whereas the IC safeguarded the IF’s

independence, together tempering undue influence from the research community or

political interference.

2.2 Implementation

30. The following factors had an impact on project implementation:

Factors having a positive impact

o The establishment of the IF as an efficient independent institution with strong

and transparent governance practices was instrumental in smooth project

implementation.

o The independent and professional manner in which beneficiaries were selected

and monitored during implementation ensured that the selected subprojects

were successfully implemented.

o Continuous access to the IF Advisory team, oversight from the World Bank,

and access to MOESTD, EUD, and the IF’s Board, was important in allowing

the IF to overcome teething and political challenges over the course of the pilot,

focus on creating a positive environment for its beneficiaries, and introduce

new innovation and technology transfer programs for Serbia.

9

o In line with the concept of the pilot, both recipient- and World Bank-executed

components were monitored closely and pivoted incrementally to enhance

program outcomes and inform longer-term instrument design. A few examples

include design of the mentoring program to accommodate mandatory and

customized training elements for IF staff and beneficiary entrepreneurs;

deploying the Mini and Matching Grant programs sequentially so the IF could

incorporate lessons from the first call of the Mini Grant Program; increasing the

private sector contribution (from 25 percent to 30 percent ) under the Matching

Grant program; converging on design details for the royalty scheme; and

engaging commercialization brokers to work with RDIs to identify

commercialization wins within the lifetime of the project.

o Under Component 3, the growing acceptance of knowledge transfer and

research commercialization was an important cultural shift among researchers

and managers in the RDI community, enabling the recipient to consider sector

wide reforms in the future.

o Continuous access to the IF Advisory team, oversight from the World Bank,

and access to MOESTD, EUD, and the IF’s Board, was important in allowing

the IF to overcome teething and political challenges over the course of the pilot,

focus on creating a positive environment for its beneficiaries, and introduce

new innovation and technology transfer programs for Serbia.

Factors having a negative impact

o 2012–2013. During the early stages of project implementation, disbursements

were lower than planned at entry. This was due to GoS insistence at entry to

plan as many as two Mini and two Matching Grant Calls for Proposals (CFPs)

in 2012. The IF had just been established and did not have the capacity at the

time to handle the proposed workload. However, based on guidance the IF

received from its advisors, it was eventually agreed to phase the rollout of the

programs and to pilot the first CFP for the Mini Grant Program separately, and

incorporate any lessons from this call into future ones. In retrospect, this was an

important adjustment, enabling IF staff to be better prepared for subsequent

calls.

o 2014. Serbia faced a turbulent year in 2014 and so did the IF. This included

catastrophic flooding, multiple changes in the Government, delayed funding,

and management restructuring within the IF. Implementation progress slowed

due to the lack of budget funds for the IF’s operations and given the

uncertainty, the IF experienced significant staff departure. Additionally,

MOESTD’s interest in engaging the World Bank on the public research sector

reform agenda diminished in the absence of a larger World Bank operation. The

project was downgraded from Satisfactory to Moderately Satisfactory.

Nonetheless, with guidance from its advisory team, the IF was able to rebuild

staff capacity by conducting additional trainings to bring the new cohort to

speed. During this period, project implementation continued, albeit at a slower

10

speed. By July 2014, the IF was fully staffed, the budget fully funded, and

operations commenced satisfactorily. After the mid-term review in August

2014, the project was again rated Satisfactory, and this status was maintained

through closing in January 10, 2016.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

Rating: Substantial

31. M&E design. There was no systematic effort to capture firm-level innovation or research

commercialization data in Serbia at entry. Publicly available indicators were limited to typical

inputs (for example, public R&D spend) or research outputs (for example, publications and

patents). Given minimal baseline information, skepticism around the pilot in Serbia, the growing

debate on effectiveness of Matching Grants in the World Bank, and high donor and recipient

expectations, the M&E activities envisaged under this project were deliberate. Beyond the set of

input, output, and outcome indicators in the results matrix, the project focused on building the

IF’s long-term capacity to identify additional outcome indicators, build its internal systems to

capture and analyze firm-level innovation data to support future evidence-based policy making in

Serbia beyond this modest pilot. At project inception, M&E activities were managed and

supported by two IF staff members, a program manager, and a senior associate, whose tasks

entailed assisting in data collection, overseeing deliverables, and payment schedules. However,

as implementation progressed, the IF realized that it lacked the M&E expertise to establish and

operate an in–house system. As a result, a reputable external think tank with a track record in

conducting innovation program impact evaluations was engaged by the IF from May 2013 to

November 2015 to ensure the quality of methodology and implementation, and to design the

most relevant approach for the small sample size of beneficiaries expected under the pilot.

32. M&E implementation. The consulting firm developed a framework and procedures for

M&E activities under the project which was incorporated in an M&E manual for the IF, and in

addition, trained the relevant staff. Staff training provided by the external consultant focused on

identifying relevant indicators, monitoring processes, specifically deploying IF data collection

systems. As part of their assignment, the firm and IF gathered extensive data on both

beneficiaries and applicants. M&E functions were to monitor and analyze the performance of the

Mini and Matching Grant programs’ subprojects. IF staff were also trained to monitor financing

programs to be managed by the IF in the future. The IF produced semiannual M&E reports.

These reports were shared with the Government, World Bank, and EUD, and served as a useful

tool for project monitoring and progress reporting. The independent evaluation was led by an

independent firm. For the World Bank-executed component, M&E data was collected and

analysed by the World Bank team.

33. M&E utilization. The IF and the World Bank monitored the interim outputs of the

project systematically to assess progress toward end-of-project target outcomes, identifying

issues and risks, and reporting on time-bound action plans to mitigate them. In August 2015,

with participation of IF staff, the consulting firm presented its findings to key stakeholders,

including senior representatives from MOESTD, MOERD, EUD, Ministry of Finance, and IP

office, among others. The focus of the presentation was on the selection process, demonstrating

the additionality of IF programs and recommendations for scaling of future programing. The

11

evaluations of the Mini and Matching Grant programs justified the EUD’s decision to provide

additional funding for these programs under its future programming and also justified inclusion

of these programs under the World Bank-funded Competitiveness and Jobs Project (C&JP), a

results-based operation (2015). On Component 3, the monitoring indicators proved useful in

concentrating efforts of RDI management and researchers on knowledge transfer and

commercialization activities. Importantly, it allowed the World Bank team to estimate the

financing gap for knowledge transfer and research commercialization in Serbia, enabling the

GoS to advocate for the sequel EU IPA-financed program with the EUD, which supported the

establishment of the national Technology Transfer Facility (TTF) and the Collaborative Grant

Scheme under the ongoing Serbia Research, Innovation and Technology Transfer Project

(SRITTP).

2.4 Safeguard and Fiduciary Compliance

34. The project did not entail any major environmental risk, include construction or

land acquisition. The project was rated category ‘B’. There were no issues in compliance with

environmental safeguards. Due to the limited size and scope of the grants, there was no

significant environmental impact associated with the project’s activities. The overall innovation

process under this component did not support environmentally unfriendly technologies and

practices. The Environmental Management Framework was implemented successfully. The

project did not entail any social risk or risk of adverse social impact.

35. FM was assessed as Satisfactory. There were no outstanding audit reports for the

project and all audit reports were found satisfactory to the World Bank. Quarterly interim

unaudited financial reports were submitted to the World Bank on time. The project audits

received ‘unqualified’ opinions of the audited financial statements and no major systems and

control issues were identified by the auditors.

36. Procurement was rated Satisfactory, based on review of prior review documents

and post review missions. The IF and beneficiaries followed the World Bank's procurement

guidelines and no major issues were encountered. For procurement under the grants, World

Bank-approved simplified commercial practices were used.

2.5 Post-completion Operation/Next Phase

37. In 2014, the Government was unable to finance post-pilot calls for the IF grant programs

as budget resources were diverted to post-flood remediation efforts. However, key elements for a

post-pilot transition are now in place with the approval of the EU IPA SRITTP and the C&JP

operations in 2015. SRITTP provides support for national technology transfer activities,

collaborative industry/academia research activities, and support for development of the new

strategy, the infrastructure road map, and their related action plans. C&JP supports among other

activities, the integration of the IF’s operational budget into the GoS’s annual budget,

continuation of the Mini and Matching Grant Programs, scaling of technology transfer initiatives,

and RDI sector reforms through a results-based approach.

3. Assessment of Outcomes

12

38. There were no major shortcomings in achievement of project objectives, efficiency, or

relevance.

3.1 Relevance of Objectives, Design, and Implementation

(a) Relevance of Objectives

39. The relevance of the objective is rated High. The objectives and related outcomes of

this project are as relevant at the time of the Implementation Completion and Results Report

(ICR) as they were at entry and remain strongly aligned with the World Bank’s engagement in

Serbia and the Government’s development agenda. This project remains highly relevant for the

overarching objectives laid out in the FY16-20 CPF which is to support Serbia in creating a

competitive and inclusive economy and, through this, to promote integration into the EU. One of

two broad CPF focus areas - Private sector growth and economic inclusion should address

significant constraints to private sector development and economic inclusion: financing,

investment, connectivity, and labor market constraints. This Project was both timely and relevant

as it supported the creation of institutional capacity and programming for private sector

innovation that can be scaled with future IPA and eventually EU funding.

40. The GoS is undertaking measures to rebalance the economy, enhance the quality and

efficiency of public expenditures and service delivery (including state-owned enterprise reform),

and stimulate private sector initiatives to enhance Serbia’s competitiveness. In March 2016, the

Government adopted the Strategy for Science and Technology Development: Research for

Innovation 2016–2020, specifically calling for reforms in the public RDI sector and reinforcing

the importance of enterprise innovation and technology transfer for the economy, including

support for the activities carried out by the IF. This pilot furnishes evidence to engage on

innovation policy and is demonstrated by GoS engagement in the SRITTP and C&JP.

(b) Relevance of Design

41. The relevance of the design is rated Substantial. Project design was essential in

supporting the delivery of PDO outcomes and enabled successful piloting of missing elements of

the Serbian innovation system. The core design features remain relevant and are therefore being

continued or expanded under SRITTP and C&JP. They would warrant being supplemented by

increased research infrastructure investments, coupled with reforms in the public research sector,

and direct support instruments for enterprise innovation and market access in the future. Under

Component 1, the IF leveraged funding and partnerships with international donor organizations

to strategically build entrepreneurship and innovation programming from traditional early-stage

matching grant funding (SRITTP, C&JP) to early stage equity funding (that is, Western Balkans

Enterprise Development and Innovation Facility [WB EDIF]). Through Component 2, the IF

demonstrated long-term demand for early-stage enterprise innovation funding and the ability of

Serbia’s private sector to lead innovation projects (often in partnership with the research sector)

and co-finance these with own matching resources. Under Component 3, the World Bank

managed to provide in-depth TA to four RDIs and deliver technology transfer targets by being

selective and staying focused. This built the case to establish a national TTF to stimulate

collaborative research under the SRITTP and support RDI sector reform activities under the

C&JP. At entry, there were no visible private ecosystem players present in Serbia; hence, the

13

pilot could not be designed to engage them. In future iterations of innovation programming and

financial instrument design (including of the Matching Grant programs), the IF could explore

leveraging local private ecosystem actors—for example, business angels, other early stage

investors, accelerators, and so on—in its selection, co-funding, and/or mentoring processes.

(c) Relevance of Implementation

42. Relevance of implementation is rated Substantial. Implementation outcomes remain

relevant, although implementation arrangements can be further strengthened in the future. The

emphasis placed on good international practices, efficiency, horizontality, independence, and

transparency is deemed relevant for the implementation outcomes of the pilot. So much so, that

the donors and the World Bank continue to leverage the IF and MOESTD for additional

innovation activities under the SRITTP and C&JP. Although currently manageable, if the IF and

MOESTD become solely responsible for innovation, without commensurate staffing and

capacity building in the new areas of responsibility (for example, technology transfer), there

could be trade-offs in terms of quality, delivery, and overall direction of the agenda. Further

expansion in the IF’s implementation responsibilities ought to be accompanied by wider

ownership and endorsement by key line ministries of the cross-cutting research and innovation

agenda, such as ministries in charge of the finance, economy, agriculture, and so on.

3.2 Achievement of Project Development Objectives

43. Achievement of PDO is rated High. Overall, the project achieved or exceeded its stated

objectives of building institutional capacity to stimulate innovative activities in the enterprise

sector as explained below. This was achieved through the operationalization of the IF, piloting of

financial instruments for technological development and innovation in enterprises; and

encouraging selected RDIs to engage in technology transfer and commercialization, and assisting

in formulating RDI sector reform policy.

44. Operationalization of the IF. As the main implementing agency, the IF was responsible

for (a) successfully designing, piloting, and managing enterprise innovation programs; (b)

supporting proof of concept and prototyping activities and demonstrating a strong pipeline of

entrepreneurs and innovative firms in Serbia; and (c) participating in co-financing of programs

and leveraging other activities organized by international organizations, financial institutions,

and the private sector, all to further the innovation agenda on behalf of the GoS. The IF engaged

its international strategic, operations, and regional advisors to develop its own capacity and

operational procedures and provide guidance to enterprises and the GoS on enterprise innovation

policy and programs.

45. Building Institutional Capacity within the IF and Beyond. An essential service

provided by the IF was the mentoring provided through Enterprise Training, in addition to the

specialized training programs that entrepreneurs would elect themselves. This was conducted by

international advisors covering key innovation management issues, such as business formation

and expansion, IP protection and commercialization, investment readiness, and early stage

technology and business development. Entrepreneurs reported these trainings to be very useful,

valuable, and otherwise inaccessible. This training also prepared IF staff for the advanced

capacity building led by the advisory team, and especially for the international trainings in Israel,

14

Finland, and Croatia that focused on ecosystem development, institutional, policy, early-stage

entrepreneurship and innovation program design, sequencing, and evaluation. The IF today is

well-regarded by its peers, and newer innovation agencies, including Macedonia and Georgia,

have requested the IF to conduct training programs for their staff.

46. Supporting Evolution of the Ecosystem. The IF played a vital awareness-building role

for the nascent innovation ecosystem by conducting around 22 educational and networking

events for its beneficiaries and a broad set of institutional stakeholders. These activities included

project launch activities across six cities in Serbia, successive launches of the Mini and Matching

Grants Programs, IP and Technology Transfer Workshop, WB EDIF workshop and four info-

days for the awardees (addressing project implementation and training held by the Serbian IP

office), four Enterprise Training Sessions for awardees, presentation of the IF Annual Report,

Finland-Israel Knowledge Economy Workshop, consultations with four major technology

transfer offices on the design of the centralized TTF, industry-RDI consultations on the design of

the Collaborative Grant Scheme, and project closing.

47. Leveraging of Resources by the IF. During design, the intention was for the IF be

established as an entity that is both experienced at managing and capable of raising additional

funds beyond this pilot. The aforementioned factors and IF’s strong delivery positioned it as an

institution with significant credibility to manage the EU and other donor funds in a professional

independent and efficient manner, far beyond what was originally planned at the onset of the

pilot, including for the following activities:

(a) WB EDIF (€140 million). The IF led the preparation of this regional program and

mobilized €28 million from the GoS.

(b) SRITTP (€6.9 million). A sequel to the current pilot, the SRITTP is financed

through the 2013 EU IPA.

(c) Serbia C&JP (€89.5 million). Under C&JP, the IF can access €12 million for its

operational budget, financing of new calls for the Mini and Matching Grant

Programs, and technology transfer activities.

44. Piloting financial instruments. The responsible governance mechanisms established by

the IF—through its 60-plus independent international peer reviewers, 5-member professional

diaspora-led independent IC, efficient under-90-day selection procedures codified in its Mini and

Matching Grant Manuals (GMs)—underscored the transparent merit-based selection processes it

developed. The IF completed four CFPs for the Mini and Matching Grants programs by October

2013, announcing the last batch of awardees on February 14, 2014. Funding was awarded for 41

Mini Grants and 11 Matching Grants to start-ups and firms with innovative projects and high

potential for commercialization. Nineteen new products and processes were launched by the

beneficiary companies, over 300 high-value jobs were supported, and 51 companies were

mentored through Enterprise Training.5 By closing, €5.4 million had been disbursed to awardees

who expressed their gratitude to the IF and authorities for designing and managing the grant

5 Detailed description of each sub-grant and the amounts disbursed can be found in project files. Annex 2 provides a

few examples of Mini and Matching Grant Projects.

15

facility in a transparent manner. Several of these early-stage projects are beginning to

demonstrate successes, with some receiving noteworthy coverage of their achievements in the

local and international press, including in the Financial Times, the BBC, and the New York

Times. At the time of ICR completion, 36 of the 38 startups funded under the Mini Grant

program and all 10 of the small and medium size firms financed under the Matching Grant

program were still active, demonstrating a 95 percent survival rate. In line with international

good practice, this pilot project was subject to a rigorous independent evaluation and annex 5

captures both key findings on additionality of programs as well as critical recommendations for

future scale up by GoS.

Box 1. Case Study of a Successful Mini Grant Program: Start-up Strawberry Energy

Strawberry Energy was a Mini Grant recipient. This Serbian startup made renewable energy sources more

accessible by developing, designing, and implementing clean technologies. It developed a solar charger and then

applied the expertise gained to create the ‘Strawberry Tree Mini’—a flexible and transportable mini solar charger

for cell phones—with IF financing. Besides improved characteristics of power production, consumption

management, and functionality of solar panels and batteries, the mini charger includes an interactive component

that shares renewable energy information with end users. A finalist of the 2013 World Technology Award,

Strawberry Energy has been recognized as an innovation leader in environmental and energy efficiency.

Strawberry Tree was featured on Mashable among 25 top technologies every smart city should have. Following

completion of the IF project, Strawberry Energy has continued development of its technology and has created the

‘Strawberry Mini Rural’, a solar charger suitable for areas with low access to electrical energy. The Accelerator

Venture Fund Eleven from Bulgaria made a €100,000 investment in Strawberry Energy in 2014. In October 2014,

Strawberry Energy was the only participating company from outside the United States to win the international

start-up contest organized by the GreenBiz Group, held in San Francisco, California.

Table 3. Additional Achievements of Mini and Matching Grant Awardees

Activity Number of Firms

Revenue generated as a result of the grants 19

International cooperation with firms 9

Foreign market distribution secured (United States) 2

New business collaboration formed 18

Applications for Patent Cooperation Treaty 13

Applications to the European Patent Office (EPO) 4

Patent applications: United States, Australia, Euro-Asia, and Japan 5

National patent applications 33

Trademarks 10

Copyrights 6

Source: IF.

45. Technology transfer and commercialization. Another activity that this pilot tackled

was to encourage RDIs to engage in institutional assessments and knowledge transfer and

research commercialization activities to glean insights for sector wide policy reforms. The World

Bank team conducted detailed diagnostic assessments of four RDIs (classified as Category I);

that is, two more RDIs than planned at entry: Institute of Physics, Belgrade (IPB), Institute of

Molecular Genetics and Genetic Engineering (IMGGE), Institute of Food Technology (FINS),

Novi Sad, and Institute of Medicinal Plants Research (IMPR). These assessments provided

detailed recommendations (and specific action plans) for improved institutional capacity,

performance management, and knowledge transfer and research commercialization practices that

were the basis for customized technical support provided to these RDIs. As part of the TA, the

World Bank organized trainings for a broader group of 10 RDIs (classified as Category II) to

16

identify opportunities for institutional improvements. These institutions participated in

discussions to identify cross-cutting topics of interest to RDIs, including: Program/Project

Management, IP Management, R&D Marketing and Sales, Performance Evaluation and Career

Planning/Development, and Managing/Developing R&D Capabilities. Based on expressed

interest, the World Bank delivered three high-level training workshops on topics of interest to the

RDI sector community.

46. Research Sector Reform. Given its modest means, the TA program made important

contributions to the voluntary institutional adjustments and technology transfer efforts at IPB and

IMGGE. These two RDIs were responsible for three of the early-stage technology transfer

targets met under this project. However, supporting technology transfer and commercialization

was very challenging owing to basic research orientation of most RDIs and the weak institutional

capabilities, low level of technological and even lower market readiness of the proposed R&D

projects. With the establishment of an Innovation Center at the IPB, technology transfer

activities progress picked up during the project. IPB leadership as well as many department

heads and researchers demonstrated strong interest and willingness to undertake some difficult

institutional and mind set changes. The IMGGE transformed itself from a university-like

organization to a research institute comparable with its international counterparts and at closing

was on the verge of exporting a new product supported under the TA (see annex 3 for details).

Both RDIs reported a significant change in attitudes among its management teams and

researchers in engaging the private sector on potential knowledge transfer and commercialization

projects. Both RDIs reported the culture as more favorable toward conducting applied research

with commercialization potential by partnering with the private sector, internal reforms in

performance and institutional management, and the need to look for non-budgetary sources of

revenues for the future. Both RDIs were able to obtain several knowledge transfer contracts from

the private sector firms and European organizations, produce patent filings and high quality

publications.

47. The management of the FINS and the IMPR—the two other RDIs supported under the

detailed TA—appreciated the in-depth assessments, but found it difficult to motivate staff to

proceed with the recommended institutional reforms without an explicit mandate push or support

from MOESTD.

48. Overall, this World Bank-led activity provided extensive insight into the market

relevance of the RDI projects and revealed the financing gap to support commercialization and

conduct RDI sector reforms nationally. In October 2013, the World Bank delivered a Policy Note

on RDI Sector Reform to MOESTD. In 2015, subsequent MOESTD leadership took up

recommendations from the Policy Note to initiate systematic reforms in the R&D sector. The

World Bank and IF teams also provided specific inputs to the ministry officials for the design of

two projects: (a) the EU IPA-funded SRITTP and (b) World Bank’s C&JP. For instance, under

the C&JP, the GoS is adopting a strategy that commits to undertaking reforms in the public

research sector by conducting RDI-level assessments with global experts and establishing

market-aligned key performance indicators that are for both individual researchers and RDIs.

Both projects were approved and currently are under implementation.

3.3 Efficiency

17

Rating: Substantial

49. Overall, this pilot achieved its objectives efficiently and is rated Substantial. It is not

useful to conduct a cost-benefit analysis for a project (a) that was attempting to pilot multiple

missing elements in an innovation ecosystem; (b) that was going to trigger systemic changes; (c)

that required significant scaling of financing to demonstrate impact; and (d) where the full effect

of the outcomes would be seen only several years post pilot. Comparing costs of similar

programs elsewhere can lead to erroneous inferences as initial circumstances vary by country.

The PDO was successfully achieved with the operationalization of the IF (€1.1 million) and the

project achieved better outcomes than originally envisaged: (a) leverage of at least €35 million

through three donor projects versus €20 million target; (b) grant decision making in 60–90 days

versus regional average of 150 days; (c) design of four financial instruments during the project

versus target of three; (d) Funding of 38 active startup that launched 19 new products versus 10

and 8, respectively; (e) mentoring of 51 startups versus initial target of 10; (f) rebuilding staff

capacity in spite of two waves of staff departures; and (g) being recognized by regional peers.

50. The independent quantitative evaluation (see annex 3 for analysis) delivered by the

external consulting firm in October 2015, titled ‘Estimation of the Effect of Participating in the

IF Financing Programs on Firm Performance,’ concluded that (a) the IF was a credible and

transparent platform, and accessible to many Serbian businesses, not just a small group of

entrepreneurs; (b) Mini and Matching Grant Programs, developed and piloted within the project,

were found to be clear, accessible, and effective, with the application process being similar to

other international programs and the evaluation and decision process being transparent and

trustworthy; (c) the IF succeeded in expanding the opportunities for tech-based entrepreneurs by

creating a number of innovative start-ups/spin-offs. Almost half (156 out of 326) of the firms that

submitted the application for IF financing were created for them to apply for the IF operation. In

addition, the evaluation found a high survival rate among supported firms. The IF succeeded in

expanding the network of tech-based entrepreneurs by creating a relatively high amount of

collaborations with academia and industry (also internationally); and (d) under the Mini Grants

Program (€2.9 million), 40 firms were approved, of which 37 were active at the end of the

project period (January 2016). Simple differences in sales changes between firms that received a

Mini Grant and firms that applied but were rejected revealed that the former increased their sales

by an additional €27,000 and their employment by an additional five employees, on average.

Based on these numbers, on average, the additional sales generated as a result of the Mini Grants

was approximately €1 million. Based on which, the investment in this component would be

recouped in a period of three years.

51. The RDI TA (€0.7 million), conducted four instead of two in-depth RDI assessments

planned, identified 10 RDIs for potential institutional improvements support instead of five,

delivered on three technologies transferred, and brought a new commercialization culture to

these RDIs. Most importantly, perhaps, learnings under this component informed the design of

the SRITTP and C&JP.

3.4 Justification of Overall Outcome Rating

Rating: Satisfactory

18

Table 4.Summary of Ratings Supporting Overall Outcome Rating

Relevance of Achievement of PDO

Efficacy Efficiency Overall

Outcome Objective Design Implementation Objective

High Substantial Substantial High Substantial Satisfactory

52. The overall outcome rating is based on ratings for Relevance, Efficacy, and Efficiency as

broken down in Table 4. Project Objective remains relevant and aligned with Serbia’s overall

development priorities and is rated as High. Project Design and Implementation are both

deemed to be relevant in the EU pre-accession context and are therefore rated as Substantial.

Some delays were experienced during early stages of implementation and there was also a need

to extend the closing date of the project. The delays were justified (and agreed to by the World

Bank, donor, and recipient) given strategic guidance provided to the IF by its advisory team to

stagger the launch of the Mini and Matching Grant Programs early during implementation to

allow for learning to be incorporated in subsequent rounds. The extension facilitated the

completion of all awardees’ projects, achievement of technology transfer targets with RDIs, and,

most importantly, rebuilt the ministry’s commitment on innovation policy and RDI reforms. In

this context, given that the Project mostly exceeded the PDO outcome targets, its Efficacy is

rated as High and Efficiency was rated as Substantial. All of the aforementioned ratings lead to

the Overall Outcome Rating of Satisfactory.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development

53. This was a small pilot project aimed at stimulating innovative activities in the enterprise

sector. The 51 projects financed through the pilot in the total amount of €6 million over four

years were too small to have an impact on poverty, gender aspects, or social development.

However, the increased income and employment generated by the pilot program have given an

indication that, if scaled up substantially, the program could have a positive impact on poverty

reduction and economic development (see annex 3).

54. Women entrepreneurs. The IF programs supported a few remarkable women researcher

become entrepreneurs who were even featured in the international media. They have become an

inspiration for younger women entrepreneurs. It would be important to strengthen gender

emphasis in future programming given the very specific challenges Serbian women

entrepreneurs face in entering the market, accessing both debt and equity finance, and extensive

harassment while running a business.

(b) Institutional Change/Strengthening

55. The project has had a significant impact with respect to institutional changes, especially

with respect to the IF, but also the management teams and researchers at RDIs supported under

the TA program, as well as MOESTD, as discussed above.

19

(c) Other Unintended Outcomes and Impacts (positive or negative)

56. At entry, there were few players of any significance in Serbia’s innovation and

entrepreneurship ecosystem, with the exception of incubators in Belgrade and Novi Sad. By

project closing multiple public and private actors were active in Serbia, including business

angels, accelerators, incubators, technology transfer offices, science parks, and so on. As the

project did not actively finance ecosystem development, causality cannot be inferred between the

project and its evolution. However, stakeholders and beneficiaries have credited the IF and its

programming for shining a spotlight on Serbia’s early-stage entrepreneurial pipeline, attracting

regional investors, highlighting its policy efforts to promote research and enterprise innovation.

20

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

57. Based on the interviews carried out with the beneficiaries as part of the M&E process, the

M&E consultants and World Bank staff concluded that (a) overall project objectives were

adequately specified; (b) the Mini and Matching Grants Programs were clear, accessible, and

effective; (c) program evaluation and decision process was transparent and trustworthy; (d)

application process was similar to other programs (internationally); (e) the IF staff ranked high

on professionalism and willingness to help, and were flexible; (f) implementation was

effective—most of the projects achieved their objectives; (g) RDI management valued the

changes in researcher behavior toward technology transfer; and (h) the Government takes

ownership of R&D sector reform agenda.

4. Assessment of Risk to Development Outcome

Rating: Negligible

58. The risk to development outcome is considered Negligible. All activities are either

completed or sustainable under follow-on projects. Although there was a change of Government

during project implementation, the risk that the program would be derailed did not materialize.

Sustainability of the pilot’s achievements has been further enhanced by the Government and the

World Bank’s commitment to further support the PDO through the SRITTP and C&JP.

5. Assessment of Bank and Recipient Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory

59. The World Bank’s performance for quality at entry was rated Satisfactory. The World

Bank heavily leveraged top-notch global expertise and lessons learned from multiple post-

transition economies to recast the innovation policy discourse to focus on the role of the

Government and entrepreneurship in building that national innovation system. Additionally, it

emphasized the importance of piloting and stronger than usual M&E practices for this

institutional capacity-building project.

(b) Quality of Supervision

Rating: Highly Satisfactory

60. Quality of supervision was far more intense than a regular World Bank project and very

meticulous given the pilot nature of the project and first time relationship with the donor.

Supervision missions were carried out at least twice a year and up to five times in the early

stages of the project owing to the flexibility afforded by the World Bank-executed TA

component. The World Bank team ensured continuity in project management and a strong

thematic and operational skills mix with strong support from staff in the local office. The World

Bank-executed component enabled mobilization of leading global practitioners in their

respective fields, thus earning the trust and respect of the research community. The team was

proactive and responsive to the multiple GoS requests for change during project implementation

and demonstrated flexibility and understanding in the use of project resources. The World Bank

led project restructuring in consultation with the donor and recipient to maximize achievement of

21

the PDO. On fiduciary requirements, the performance was Highly Satisfactory with FM,

procurement, and environment activities under the project implemented effectively with

significant capacity building and no major issues.

(c) Justification of Rating for Overall Bank Performance


61. Overall, the World Bank was very proactive and responsive to the client’s needs from the

design stage through to project closing, with close supervision and excellent support from the

country office. The World Bank team also played a very important role in establishing the IF’s

capacity by offering its own experience and guidance. Based on the above, the overall World

Bank performance in ensuring quality at entry and quality of supervision is rated Satisfactory.

5.2 Recipient Performance

(a) Government Performance

Rating: Moderately Satisfactory

62. The recipient worked closely with the World Bank during project preparation in a

satisfactory manner. During implementation, the recipient could have been more proactive in

providing support to the IF (especially its operating budget), and introducing some quick win

policy recommendations resulting from the RDI TA program. Although the Recipient complied

with all covenants under the Grant Agreement and development outcomes were met by the

project, the shortcomings in budget support to the IF slowed down implementation progress in

early 2014 and destabilized the IF’s operations. Therefore, the overall performance of the

Recipient is rated as Moderately Satisfactory.

(b) Implementing Agency or Agencies Performance

Rating: Highly Satisfactory

63. The IF as the implementing arm of the Recipient was very effective in providing efficient

implementation, monitoring, and reporting, allowing for better management and implementation

of the project and facilitating the World Bank’s supervision activities. The IF was staffed with

qualified experts, selected through a competitive process and adequately compensated for their

work. Although there were teething problems such as the absence of a permanent managing

director and lack of timely GoS budget support on multiple occasions, the IF staff have done a

remarkable job in implementing the project and achieving the PDOs. The IF staff should be

commended for the professionalism exhibited in the discharge of their duties and the

performance is rated Highly Satisfactory.

(c) Justification of Rating for Overall Recipient Performance


64. The overall Recipient performance is rated Satisfactory. The success of the project was to

a large extent due to the efficiency and professionalism demonstrated by IF staff in the

implementation of the program and they should be highly commended. The Recipient, although

22

committed, could have been more proactive in providing budget and policy support to the IF,

RDIs, and the sector overall.

6. Lessons Learned

65. Governance arrangements are key for institutional integrity and program delivery. Governance mechanisms established at the IF were underscored by the transparent selection

process that placed a premium on merit and commercialization, and integrated feedback from

international peer reviewers, decision making by an independent IC and continuous guidance

from a professional global advisory team. This positioned the IF as an institution with significant

credibility and capacity to absorb and distribute EU funds in a thorough and independent

manner. This aspect was confirmed by beneficiaries, who commended the transparent and

independent process and the mentoring received from peer reviewers.

66. Institutional capacity matters for delivery of innovation programming. A well-

functioning national innovation agency, staffed with competent and committed people and fairly

remunerated is essential for implementation of a complex project, such as this, and is an

important element of the national innovation system. The IF is an institution that is well regarded

by its beneficiaries and its peers, and should be nurtured to continue to serve the country and

remain agile in an ever changing global innovation landscape. This will mean additional rounds

of capacity building and staff renewal to meet new client and market demands. This should

remain part of the IF’s strategic planning efforts.

67. Horizontality of innovation policy and programming matters in nascent ecosystems.

One of the principles of design considered important in nascent post-transition innovation

ecosystems—dominated by entrenched state enterprise and RDIs—is horizontality of the

innovation policy and interventions, that is, sector agnosticism. At entry, many assumed that the

deal flow would be limited to innovative projects in ICT and digital technology from Novi Sad

and Belgrade. The pilot—by adhering to horizontal selection criteria that emphasized merit and

commercialization potential—revealed a diverse pipeline in terms of sectors, cities, and gender.

68. Size of innovation effort matters and so do rhythm and scale. The 51 projects the IF

financed over four years was too small a number to have an economy-wide impact, but

manageable for capacity building. However, to move the needle at an economy-wide level, the

independent evaluation estimates that the IF would need to provide funding to 200–300 projects

annually. Having established the IF and piloted instruments that demonstrate additionality for

enterprise innovation, the GoS should ensure continuous uninterrupted support to retain IF

capacity, generate innovative entrepreneurs, rebalance the enterprise innovation system, and

continue to position Serbia as a budding knowledge and IP creator in the global innovation

landscape. Losing the momentum generated would be ill advised and create significant loss of

trust in public institutions and could undermine future efforts.

69. Visibility of innovation activities really matters. World Bank operations could take a

serious lesson from the EU’s approach and benefit from a more systematic approach to providing

visibility to project activities with stakeholders, sharing lessons learned, and celebrating

successes. The Project Opening Ceremony engaged the IF’s advisory and management teams,

and its IC to showcase the capacity engaged and discuss intentions transparently, thus building

credibility among the sceptical research and entrepreneurial communities. The Closing

23

Ceremony—a celebration of the journey the Serbian ecosystem had made over the course of the

project—displayed the institutional capacity of the IF and RDIs, the remarkable 50 plus

entrepreneurs who demonstrated their firms’ prototypes and viable products, as well as the

investor groups and entrepreneurship support institutions that did not exist at the onset of the

project. The unrehearsed testimonials provided by these ecosystem actors during visibility events

were picked up by policy makers and donors and are informing future decisions on financing.

70. Support for the evolution of an innovation ecosystem is critical. In the early days of

the project, the IF was a lonely advocate for the ecosystem and would often come under undue

scrutiny and criticism. Eventually the strong entrepreneurial pipeline attracted regional partners

and global investors, making it easier to advocate for support to other actors and promote long-

term symbiotic relationships in the ecosystem.

71. The policy world is changing fast. The innovation arena is witnessing an increase in

well-networked private sector investors and support institutions globally, including in incubation,

acceleration, and financing. Policy makers are beginning to review policies and adopt legislation

to enable crowd funding, which could further democratize early-stage innovation finance. While

not an immediate concern for any of the post-pilot operations, the GoS and IF should familiarize

themselves with these developments and policy implications to ensure relevance of future design

and implementation.

72. Flexibility during design and implementation are key for pilots. During design and

especially during implementation of this pilot, the World Bank team worked closely with the IF

to monitor project progress. When issues were encountered, the teams worked quickly to find

solutions. This flexibility allowed for smoother and efficient implementation. A few examples

include (a) design of the mentoring program to accommodate mandatory and customized training

elements for IF staff and beneficiary entrepreneurs to be more useful; (b) deploying the Mini and

Matching Grant programs sequentially so the IF could incorporate lessons from the first call of

the Mini Grant Program for subsequent programs; (c) increasing the private sector contribution

(from 25 percent to 30 percent) under the Matching Grant program; (d) converging on design

details for the royalty scheme; and (e) engaging commercialization brokers to work with RDIs to

identify commercialization wins within the lifetime of the project.

73. Incentives Matters: Entrepreneurs must have “Skin in the Game”. The requirement

for the beneficiary entrepreneurs to provide a match in case only (not in kind) and the success

royalty component weeded out several serial ‘grant writers’ during the project. This scenario is

known from international practice and hence influenced project design and played out in Serbia.

74. Building relationships with development partners takes times but is worthwhile. This was the first time in Serbia that a Trust Fund Agreement was forged between the Bank and

the EU Delegation. As both organizations were unfamiliar with policies and procedures of their

counterparts, this led to some delays during project preparation. However, once both

organizations were comfortable with each other’s policies and procedures, project

implementation progressed smoothly. It would be prudent that for future joint collaborations to

allow some lead time for project preparation so that any procedural issues could be sorted out

and not delay project preparation. Additionally, the Bank team ensured that it met with the EUD

24

counterparts during each mission to allow maximum information sharing on project progress and

any challenges that could be jointly addressed in bilateral conversations with the GoS.

7. Comments on Issues Raised by Recipient/Implementing Agencies/Partners

(a) Recipient/implementing agencies: please see Annex 7 for report and statement from IF.

(b) Co-financiers: please see Annex 8 for statement from EUD.

(c) Other partners and stakeholders: Please see Annex 8 from key partner: Belgrade Incubator

25

Annex 1. Project Costs and Financing

(a) Project Costs by Component (in Euro)

Appraisal Estimate

Components Project Cost

(€, millions)

Grant

Financing (€,

millions)

Government

Financing*

% of EU

Financing

Component 1. Capacity Building of the

Serbia Innovation Fund 3.21 1.11 2.10 35

Component 2. Implementation of Financial

Instruments Supporting Enterprise Innovation 6.00 6.00 0.0 100

Component 3. Provision of Technical

Assistance to Research and Development

Institutes (RDIs)

0.70 0.70 0.0 100

Total base cost 9.91 7.81 2.10

Contingency 0.12 0 0.12

Total project cost 10.03 7.81 2.22 78

World Bank administrative fees (7%) 0.59

Total EU Grant 8.40

Final Project Cost

Components Project Cost

(€, millions)

Grant

Financing (€,

millions)

Government

Financing*

% of EU

Financing

Component 1. Capacity Building of the

Serbia Innovation Fund 3.82 1.06 2.76 28

Component 2. Implementation of Financial

Instruments Supporting Enterprise Innovation 6.51 5.41 1.1** 83

Component 3. Provision of Technical

Assistance to Research and Development

Institutes (RDIs)

0.80 0.80 0 100

Total Project Cost 11.13 7.27 3.86 65

World Bank Administrative Fees (7%) 0.59

Total EU Grant 7.86***

Note: *Government financing covering period from 2011–2015 (funds are expressed in € using middle exchange

rate of National Bank of Serbia on the last day of each year). Initially, project closing date was set on November 30,

2014. However, it was extended to January 10, 2016, which caused increased Government financing (Government

financing in 2015 equals around €534,000).

**Contributions from the Mini and Matching Grant private sector enterprises were at least €1.1 million.

*** EURO 0.655 million was cancelled.

(b) Financing

Source of Funds

Appraisal

Estimate (€,

millions)

Actual/Latest

Estimate (€,

millions)

Percentage of

Appraisal

EU Grant 8.40 7.86 94

Government Funds 2.22 3.86 175

26

Annex 2. Outputs by Component

Component 1: Capacity Building of the Serbia Innovation Fund - Recipient Executed

1. The IF was to be established with the objective of encouraging entrepreneurship and

managing financing for innovation, as well as participating in co-financing of programs, projects,

and other activities organized by international organizations, financial institutions, and the

private sector. This has been fully achieved. Under this component, the IF developed the

requisite capacity for designing and managing grant instruments geared toward technology-based

firms. Importantly, the IF has built and retained an impressive international team, including

strategic, operations, and regional advisors, in addition to strong governance practices espoused

by its international diaspora-based independent IC.

2. The major networking and educational events held by the IF were project launch and

launch of the Mini Grants Program, followed by the launch of the Matching Grants Program, the

IP workshop (for RDIs and key stakeholders), WB EDIF workshop (for RDIs and private sector

and key government stakeholders), and four info-days for the awardees (addressing project

implementation and training held by Serbian IP office), presentation of the IF Annual Report,

knowledge economy workshop, industry and RDI consultation regarding the new technology

transfer project design, consultations with local Technology Transfer Offices on the cooperation

and functioning of centralized TTF.

3. The IF also provided several training programs for its staff and beneficiary enterprises.

One of the essential services the IF provided to its grant awardees was Enterprise Training,

which was conducted by international advisors covering topics of innovation, business formation

and expansion, IP protection and commercialization, investment readiness, and other valuable

themes associated with early-stage technology and business development. As part of its program,

the IF also conducted several project visibility events and workshops. Overall, this component is

considered highly satisfactory for the establishment of the IF, with its remarkable governance

and the institutional capacity.

4. Staff training for the IF. The following training courses were attended by IF staff.

Preparation and management of the World Bank-funded projects (World Bank

office, Belgrade). IF staff received training on all aspects on the preparation and

implementation of World Bank-funded projects.

Management of innovation and technology support programs (at BICRO,

Croatia). Training provided at BICRO was expected to introduce IF staff to BICRO

programs and organization, management of calls for financing, due diligence, M&E.

Introduction to programs for supporting operation of incubators and technology

transfer programs was also provided.

FM (Turin, Italy). Training was provided to the financial manager to design and

operate the FM systems and execute corresponding payments/disbursements in

accordance with sound professional standards and the harmonized requirements of

donors and their respective governments.

27

Venture capital (VC, Tel Aviv, Israel). Training was provided to acquire critical

skills in five key areas: (a) interpreting governance diagnostics and political

economy analyses, (b) creating multi-stakeholder coalition building strategies and

tactics to support reform, (c) providing communication skills that support the

implementation on governance reforms, (d) leveraging social/digital media tools and

analytics effectively, and (e) developing communication metrics and applying M&E

frameworks. Training was attended by the IF’s interim managing director.

VC (Tel Aviv, Israel - two times for staff). Training consisted of introduction to

organization, operations, and principles of private sector VC funds, transaction, and

due diligence process for early-stage financing. Training was useful for the IF

managing director and program managers to structure and implement privately

managed/publically funded equity instruments and to efficiently connect Serbian

companies with private sector VC funds.

World Bank procurement and FM procedures (Sarajevo, Bosnia and

Herzegovina). Introduction to the World Bank procurement and FM procedures and

guidelines was provided. Training was attended by the procurement associate and

financial manager.

Innovation funding programs (Helsinki, Finland). Tailor-made training for the IF

managing director and program managers at TEKES and SITRA in Finland was

provided to get an overview of financing mechanisms for private sector companies

and enterprise R&D as well as fund management in Finland; applications evaluation;

monitoring, funds management; and programs for large companies.

Program management (Turin, Italy). Two-week training on project and program

cycle management based on the logical framework approach, including project

identification, stakeholder analysis, project design, and the development of M&E

systems was provided. Training was attended by the IF’s senior associate.

Represent. The IF’s public relations (PR) person and a person in charge of

managing the social network pages (LinkedIn and Facebook) have received training

to provide knowledge on the most important aspects of presenting the institution

through social network profiles/pages.

Training on EU PRAG procedures. IF employees have attended the tailor-made

training designed to provide better understanding of the PRAG procedures that was

required in the implementation of the projects financed by IPA funds.

PR training. IF employees received PR training from the World Bank

representative (Belgrade office) in Belgrade.

5. A critical ingredient for the success of the project was that a responsible governance

mechanism was established at the IF, underscored by the transparent selection process that

placed a premium on merit and commercialization potential of projects, and included inputs from

international peer reviewers and decision making by an independent IC. This has positioned the

28

IF as an institution with significant credibility and capacity to absorb and distribute EU funds in

a professional and independent manner.


- Recipient Executed

6. The IF started its operation on December 5, 2011, as Serbia’s main agency for innovation

support through two main programs: the Mini Grants Program and the Matching Grants

Program.

7. The purpose of the Mini Grants Program was to expand opportunities for tech-based

entrepreneurs by stimulating commercialization of R&D and the creation of innovative

enterprises based on knowledge through start-ups and/or spin-offs. The Matching Grants

Program aimed to expand collaboration opportunities for innovative micro and small companies

with strategic partners (for example, private sector industry, R&D organizations, and VC/private

equity funds) with the goal of increasing private sector investment in R&D and the

commercialization of projects promoting new and improved products and services.

8. The financing offered by the Mini Grants Program covered a maximum of 85 percent of

the total approved project budget and up to €80,000 for a one-year project. The reminder of the

total approved project budget was to be secured by the applicant from other, preferably private

sector sources, independent of the IF.

9. The financing offered by the Matching Grants Program covered a maximum of 70

percent of the total approved project budget and up to €300,000 for a two-year project. The

remainder of the total approved project Budget was to be secured by the applicant from other

sources independent of the IF such as private sector industry, private investors/VC/private equity

funds or the applicant’s own internal resources.6

10. Financing decisions were made on a competitive basis by the IF’s independent IC. The

number of awards was determined by the quality of the proposals and subject to the total funds

available (allocated) to the programs. In addition, these programs provided support for FM, IP

protection, and business development training, as well as mentoring of the awardees to enable

them to prepare for the next stage of development.

11. Between January 31, 2012 and October 28, 2013, the IF managed four financing calls

during which 326 Serbian enterprises submitted applications for either the Mini or Matching

Grants Programs. The IF held about 20 visibility events per call to promote the CFPs and to

explain the Mini and Matching Grants Programs. Some 91 firms applied to the Matching Grants

Program, while 247 firms applied to the Mini Grants Program (12 firms applied to both the

6 Upon successful technology commercialization, the revenue (and not just the profits) derived from the sales of the

product/service and any or subsequent products/service based on the technology developed in the Matching project

financed by the IF, become the basis for a royalty payment in compensation for the IF financing, at a rate of 5

percent of annual revenue derived from sales of product or service or 15 percent of licensing revenue derived from

such product/service up to 120 percent of financing received from the IF for a period of up to five years after project

completion, whichever is achieved first.

29

Matching and Mini Grant Programs). Overall, 471 projects were submitted by the 326 firms. A

total of 55 projects were approved for IF support, out of which 52 projects were financed (out of

the 471 projects); a 12 percent approval rate. Details of the selected awardees from the last CFPs

and their achievements can be found in project files. Some samples of Mini and Matching Grants

Awardees are stated below:

Mini Grants

12. 365ads. The project ‘Brides2bride’ (B2B) was developed by a Belgrade ICT start-up

‘365ads’. ‘Brides2bride’ was a new platform through which ex brides—sharing their own

experience—help future brides organize their weddings. B2B connects different actors in the

process—former brides enter the world of wedding organization by recommending companies

they were satisfied with, future brides get free help in wedding organization and first-hand

information, and companies get the best possible advertisement, direct recommendation. IF

financing, €76,460.

13. COFA Games. The Belgrade-based company, COFA Games, operating in the gaming

industry, developed a new multiplayer, action real-time strategy video game titled ‘Awakening of

Heroes. The game will be deployed to different operating systems and will also enable players to

access the game through different devices, such as tablets, desktop computers, or Oculus Rift VR

Headset. The game was built using the Unity 3D platform, which supports browser games, so

that it can be played anywhere and does not require additional installation. IF financing, €80,000.

14. Coprix Media. The company has profiled itself into a specialized ICT-based education

aid company, with their latest project (following ‘Interactive spelling book’ and ‘Interactive

mathematics’) being ‘Interactive the world around us’. Through it, the company created a new

application for touch-sensitive devices (tablets, smart phones), with the result being a new

gamification-driven learning tool for preschool and primary school children. The project’s main

aim is to help keep education interesting and technologically updated for the generations to

come. IF financing, €80,000.

15. Eipix Entertainment. A young, but rapidly growing company from Novi Sad developed

a new free-to-play (F2P) video game that combines hidden object puzzle adventure play style

with rich interactive content, set in a story-driven environment. F2P is a very popular approach to

modern gaming, represented in digital hybrid form and usually consisting of freemium (full

product for free for nonpaying users, and a premium version for paying users), micro

transactions, and a virtual economy. IF financing, €70,880.

16. Ekofungi. The new mushroom-growing technology that was developed by the Belgrade-

based company, Ekofungi, yields an increase in quality of the mushroom’s fruit body, and also

results in a significant reduction or even complete elimination of harmful compounds used

during the growing of mushrooms. Due to the absorption of various minerals from the substrate

and a substantially greater presence of microelements (for example, selenium, magnesium), as

well as use of other natural compounds, this new type of mushrooms will have a much higher

nutritional value and will be devoid of mycotoxins that can harm humans. IF financing, €72,580.

30

17. Euro Cal. From the city of Kragujevac, Euro Cal developed a new type of ultrasonic heat

meter that finds application in chemical, processing, food, pharmaceutical, and various other

industries. The company’s new product will be manufactured in Serbia and is expected to offer

several competitive characteristics: energy efficiency, increased functionality, high

manufacturing quality, and an affordable price compared to other ultrasonic heat meters already

offered on the market. IF financing, €80,000.

18. Novel IC. The young Belgrade ICT company, Novel IC produced a new solution to help

people with visual impairments. The system, which is embedded in a spectacle-shaped frame,

will help users to detect the distance and speed of objects that are in nearby surroundings by

using the integrated radar sensors (60 GHz RF ASIC). This new solution will also find its

applications among individuals functioning in dark and fog-like environments. IF financing,

€79,710.

19. NS Web Development. NS Web Development’s ICT project named ‘Clinic in Cloud’

represents a new business framework for patient and appointment management systems. The

most important modules of the project are patient data, appointment and insurance policy

management, as well as data security and advanced reports. This software-as-a-service solution

is designed for various clinics and the approach is to be piloted in Serbia before moving into the

international market. IF financing, €61,900.

20. Projektinženjering Tim. Throughout their new project, the construction engineering

company ‘Projektinženjering Tim’ developed an innovative system for seismic protection of

buildings during an earthquake. The system is based on new GOSEB technology that will enable

efficient multilevel and multidirectional protection of the entire buildings. Using special

technical solutions, the system is designed to optimally disperse and reduce the shock energy

(forces) in all directions, as well as to adapt its own vibration features depending on the intensity

of the earthquake. The company is planning to export the solution to various international

markets. IF financing, €78,700.

21. Unitehna Alati from Kruševac developed a new, cost-efficient device for green pruning

with improved characteristics in optimal field of visibility, pruning accuracy, and responsiveness

to commands. The improvement will be achieved by employing an innovative pneumatic

controller system for the device, instead of relying on traditional hydraulic solutions. The device

is easily adaptable to different models of tractors and is expected to be 50 percent lighter and

cheaper than the products of the competition. IF financing, €80,000.

Matching Grants Program

22. Techno Naiss Group. Hailing from the city of Niš, ‘Techno Naiss Group’ developed a

new model of X-ray collimator with a durable, affordable, and modular mechanism for automatic

filter replacement. The effective electromechanical solution behind the new filter replacement

mechanism eliminates the need to have manual operators handling filter changing. Furthermore,

it will be possible to apply the filter replacement system to other existing types of collimators,

offering greater versatility to the project. Relying on their globally present partner ‘X-Alliance’,

Techno Naiss Group plans to sell the new collimator model in various international markets. IF

financing, €259,959.

31

23. BSK. The main activities of the company are design, manufacturing, and sales of

innovative products based on technologies in the field of hydrodynamics. One of the innovations

of BSK is the Air Berry Harvester KOKAN 500S, intended for picking of raspberry, blackberry,

blueberry, and currants. It uses the air current’s force to pick the fruit off the plant, then purifies

it from the dried leaves and twigs and places the fruit into crates for storage. The project is

completed. After setting up a new time frame following the setback caused by last year floods,

the company has completed the building of the Air Berry Harvester KOKAN 500S, for picking

fruits (raspberry, blackberry, blueberry, and so on). This new prototype will be used as a model

for commercial production. First unit sale has been reported. IF financing, €282,669.

24. Vizus. Vizus operates from Niš and it is a producer of aluminum and aluminum-wood

windows and doors. The company offers an innovative solution for manufacturing of high

efficiency windows and doorframes for passive construction with low coefficient of thermal

conductivity, increased durability, and higher resistance to penetration of water and air, and

increased stability. The company successfully developed innovative solution for manufacturing

of high efficiency windows and doorframes. The serial production was set up during the

beginning of 2015. Interest was generated from potential partners from Germany and the United

Kingdom. First sale of the new windows stared in June 2015. IF financing, €246,365.

25. Techno Foam. The company is operating in the machine industry, developing

specialized machines for the needs of companies dealing in sponge production. Techno Foam is

designing and developing a new machine for cutting, profiling, and packaging of sponges. The

company has completed prototyping and establishing the production of the new automatic

machine for cutting, profiling, and packaging of sponges. Development of this automatic

machine is a continuation of the Mini Grants Program that was completed in 2013. Interests for

purchase of the machine have been expressed from companies abroad: United States, Argentina,

and the United Arab Emirates. IF financing, €296,740.

26. Milinković Company. The company is improving a new integrated system of

construction, using prefabricated building blocks (‘big bricks’) to construct multipurpose halls of

various dimensions. Their solution represents a quick and simple way of construction of long

lasting, durable objects, and results in energy-efficient objects of high quality, which are

simultaneously resistant to storm winds, fire, and earthquakes. The company has successfully

completed its project of improving a new integrated system of construction, using a prefabricated

building block. Final certification was also completed within the project implementation period.

It has secured local contracts for the new product. IF financing, €183,070.

Component 3: Provision of Technical Assistance to Research and Development Institutes -

Bank Executed

27. The work on Component 3 started with the first visit of the World Bank team to Serbia in

December 2011. The team discussed key issues, challenges and opportunities in the Serbian RDI

sector, and the proposed approach and gauged interest of RDI management to participate in the

TA program. This was to be followed by building support for the action plans, dissemination

workshop, and assisting RDIs’ management during implementation of the respective action

plans. The work on a policy note outlining systematic RDI sector reform issues started around

32

mid-2012 and was completed in October 2013. Highly qualified international technical experts

were engaged as needed and when specific needs of RDIs were identified or requested by RDIs.

28. Prevailing R&D environment. This work was launched under prevailing difficult

conditions in the Serbian R&D sector. There were serious budget constraints nationally affecting

the R&D sector. This was compounded by the frequent changes in ministry leadership (during

2011–2014 there were four ministers), and a lack of policy stability and commitment to

implement any reforms. Further, there was no reliable information available on the RDIs quality

and volume of R&D outputs and their overall performance. Further, it was expected that the

ministry would undertake systematic R&D sector reforms and request a US$100 million World

Bank project. Both these critical initiatives were not pursued by previous ministry leadership, but

were followed up by the subsequent administration in 2015.

29. Key R&D sectoral issues. Initially, RDI management teams identified a number of

issues in the Serbian RDI sector. These included brain drain of scientists, aging of the scientific

community, difficulty in hiring of young researchers, limited size of the Serbian market for their

research services, difficulties in penetrating external markets and funding, weak managerial and

business-related skills, a lack of culture and skills related to the protection of IP, technology

transfer and commercialization, spin-offs, shortage of financial resources, inadequate laboratory

infrastructure, perverse incentives to conduct applied R&D, and so on. While some of these

issues were common to most RDIs, some RDIs seem to be doing quite well in several areas. For

example, institutes such as the Mihajlo Pupin Institute (MPI), Institute of Field and Vegetable

Crops (IFVC), Maize Research Institute (MRI), FINS, and IMPR were mostly self-sufficient,

earning revenues from the market and receiving only a fraction of support from the ministry in

the form of project grants for specific purposes.

Selection of RDIs

30. The RDIs to be assisted were identified jointly by the ministry and the World Bank team

keeping in mind (a) interest of RDIs management to participate in the TA program, (b)

commitment of RDI management for reform and potential for improvement, (c) diversified

sectoral coverage, and (d) resources under the project. The demand from RDIs to participate in

the TA program was strong, as many RDIs wanted to benefit from World Bank assistance. At the

request of RDIs’ management, the World Bank team agreed with the ministry to include the

following RDIs (more than originally anticipated) to benefit from the TA program.

Category 1 RDIs (detailed TA)

(a) IPB, Belgrade

(b) FINS, Novi Sad

(c) IMPR, ‘Josif Pančić’, Belgrade

(d) IMGGE, Belgrade

33

Category 2 RDIs (limited TA focused on institutional improvements)

(a) MPI, Belgrade

(b) Institute of Chemistry, Technology and Metallurgy (ICTM), Belgrade

(c) Institute of Biological Research (IBR), Belgrade

(d) VINCA Institute of Nuclear Science (VINCA), Belgrade

(e) IFVC, Novi Sad

(f) MRI, Zemun Polje

31. Category 1 RDIs (detailed TA). Scope of work for Category 1 RDIs included

conducting a detailed assessment, carried out jointly by international experts and RDI

management. This involved an assessment of the RDI’s organizational structure, human and

budgetary resources, researcher promotion and incentive system, infrastructure and laboratory

facilities, research capabilities and outputs, IP protection, challenges and potential for applied

R&D, technology transfer and commercialization, and so on. Based on this diagnosis, technical

support was provided for the design and implementation of a RDI-specific upgrading program

(action plan). This included technical expertise, training workshops, and advice, but no support

for the upgrading of labs and other infrastructure. The four selected institutes reflected a

diversified portfolio. Key differences between them include the share of basic versus applied

research conducted at a given RDI (IPB is almost entirely basic research, while IMPR is

predominantly applied); IPB is almost fully state-financed, while IMPR is predominantly

commercial income-based by selling its own products. The IMGGE is a mix of basic and applied

R&D as is the FINS.

32. Category 2 RDI (limited TA). This work included design and delivery of training

workshops on multiple cross-cutting topics of interest to RDIs. The topics included IP;

technology transfer and commercialization; training of research managers; and market

development, collaboration, and EU-Framework Program funding.

The Approach

33. The World Bank’s approach was based on strong participation of all key stakeholders,

both during the assessment and design of RDI sector reforms. These included lead ministry

officials, EUD, Serbian Academy of Science and Arts, other ministries and public sector

agencies, RDIs management and researchers, private sector, nongovernmental organization, and

relevant international organizations such as the Global Research Alliance (GRA).7 The work

7 The GRA, a virtual alliance, is a network of nine of the world’s most prestigious knowledge-intensive R&D and

innovation organizations with a goal of creating ‘A Global Knowledge Pool for Global Good’. The GRA has a

capacity to undertake projects with a magnitude and complexity that transcend the capabilities of any single

organization. It includes organizations of the northern and southern hemisphere, with more than 60,000 scientists

and engineers. The alliance partners perform basic and applied research, technology transfer and commercialization,

and specialize in the implementation of innovative commercially and socially viable solutions. The GRA has within

34

involved analyses of available data supplemented by interviews with senior management and

focus groups, and discussions with the ministry and industry officials.

34. The World Bank team engaged an extensive roster of reputable global experts and

practitioners, most of whom had managed R&D organizations and/or designed and implemented

systematic organizational reforms.

Deliverables

35. Component 3 includes the following deliverables:

(a) Assessment reports including action plans for two Category 1 RDIs

(b) Brief diagnosis of four Category 2 RDIs

(c) Design and delivery of in-depth TA to two RDIs

(d) Technical inputs to the Serbian RDI sector reform program

(e) Dissemination and consultation workshop

(f) Input to research sector reforms

36. The project has either met or exceeded expectations delivering a higher level of quality

outputs in comparison with what was expected in the project agreements. Key outputs include

the following:

(a) Assessment reports including action plans for two Category 1 RDIs. The team

has delivered comprehensive assessment reports including actions plans for four

RDIs: (a) IPB, Belgrade, (b) IMGGE, Belgrade, (c) IMPR, Belgrade, and (d) FINS,

Novi Sad. These RDIs are provided with TA by high quality international experts

from relevant fields, who conducted in-depth analyses of each RDI, identifying its

position in the Serbian R&D sector, and Serbian and international business society,

and outlining specific reform programs. Each RDI was provided a comprehensive

but separate, tailored document entitled ‘Challenges and Opportunities’, presenting

the experts findings and advice for further transformation. Customized TA is

provided to each RDI to implement the recommended action plans, tailored to meet

their specific needs and willingness to undertake reforms. Summary information on

four Category 1 RDIs is provided in the attachment.

(b) Brief diagnosis of Four Category 2 RDIs. This work provided input to the design

of training workshops on cross-cutting topics of interest to most RDIs. The topics

its ranks highly innovative members who among other breakthroughs have developed an MP3 player, Xerox

copying technology, carbon fiber composite wings for light combat aircraft, a space satellite, drought resistant millet,

mesh networking for rural internet provision, the world’s first flu drug, low-cost HIV and tuberculosis drugs, an

open-source drug discovery platform, and so on.

35

included IP; technology transfer and commercialization; training of research

managers; and market development, collaboration, and EU program funding.

(c) Design and delivery of TA to selected RDIs. The expert team designed training

workshops on a number of cross-cutting topics of interest to RDIs, including

Governance and Institutional Management, Program/Project Management, IP

Management, R&D Marketing and Sales, Performance Evaluation and Career

Planning, and Managing/Developing R&D Capabilities. Three training workshops

on these issues were delivered to the participating RDIs and ministry officials. All

ten RDIs and some ministry and industry officials participated in the workshops.

(i) Two-day Workshop on RDI Institute Management, September 2012

(ii) High-level Workshop on Strategic RDI Sector Reform, December 2012

(iii) Workshop on Commercialization and Technology Transfer, December 2013

(d) Technical inputs to the Serbian RDI sector reform program. The Bank team

prepared a comprehensive policy note outlining key RDI systematic sectoral issues

and recommendations, ‘Enhancing the Productivity of Serbian Research and

Development Institutes: Initial Observations and Recommendations’, delivered to

MOESTD in August 2012. At the request of the new minister, an updated policy

note was prepared and delivered to MOESTD in October 2013. These policy notes

analyzed the Serbian RDI environment and identified issues and critical factors and

outlined key recommendations for successful reforms of the RDI sector. These

recommendations have provided a strong platform for the new ministry leadership to

undertake systematic reforms in the R&D sector. Many of the recommendations for

these policy notes have been incorporated in the design of recently adopted Serbian

Strategy for Research and Innovation 2016–2020.

(e) Dissemination and consultation workshop. A dissemination and consultation

workshop on systematic RDI sector wide reforms was organized in December 2012,

with the participation of the MOESTD minister and officials, Serbian Academy of

Science and Arts, RDIs, and industry. Presentations were made by MOESTD, RDI

and academy officials, and technical experts. The World Bank team provided

specific and detailed inputs to the ministry officials for the design of two projects

currently under implementation: (a) the EU IPA SRITTP, and (b) the World Bank’s

C&JP.

Key Performance Indicators and Outcomes

37. The result indicators related to Component 3 included the following:

(a) Number of technologies transferred by RDIs

(b) Dissemination of policy recommendations for RDI sector reform

36

38. The project fully achieved both indicators, as explained in the following paragraphs. The

project outcomes were especially noteworthy, considering frequent changes in the ministry

leadership, their weak capacity and unwillingness to undertake any meaningful reforms, the huge

funding crunch, and postponement of the request for World Bank project support.

39. Overall, the project has a huge positive impact on the general attitude toward RDI reform

including applied R&D, commercialization, and innovative opportunities in general, the area

with great potential, which was (and still is) not properly utilized in the Serbian R&D system.

The new ministry leadership is highly committed to reform the sector and has asked the World

Bank team to assist in designing a comprehensive reform agenda for the R&D sector. They have

already started implementing several policy and regulatory changes.

40. Management of all four Category 1 RDIs expressed deep appreciation for the TA and

opined that owing to the TA they are experiencing positive changes in their institutions, with the

extent of positive changes varying from highest at IPB and lowest at IMPR. RDI management

and a significant number of researchers are energized and motivated to pursue applied R&D and

technology commercialization. They are actively pursuing new nongovernment sources of

revenues and liaising with industry by exploring technology transfer and service opportunities,

both within Serbia and in Europe. They have also increased their efforts to participate in the

European R&D projects. In some cases, RDIs have secured R&D/service project funding from

external sources, including from a large global company and European Patent Office.

Summary of Category 1 RDIs

Institute of Physics, Belgrade

41. IPB is among the largest RDIs in Serbia, employing over 120 PhDs, with an annual

budget of about €4 million, where over 90 percent comes from the Government, through research

projects. IPB was affiliated with about 10 percent of all scientific publications published by

Serbian scientists. Being an almost exclusively basic research-oriented RDI, it surprisingly

expressed the largest enthusiasm toward applied R&D and commercialization, as well as the

institutional changes in general. In accordance with the World Bank team’s recommendations,

IPB management formed an internal team (strategic growth unit), which consists of five people,

and actively supports the project at IPB and continuously acts within the space temporarily

allowed by the legal framework. Through the process, IPB customized the technology readiness

level scale for IPB needs, and established an innovation center. It also developed the long-term

strategy, in which IPB aims to switch from almost exclusively government financing, to one-

third from the Government, one-third from commercial operations, and one-third from externally

financed scientific research, such as EU’s Framework Program or Horizon, and so on.

42. Commercialization. IPB initially identified 14 R&D commercialization proposals,

which reduced to 8 through the review process, and further down to 4 actively participating, after

critical assessment. IPB researchers compiled four new, technologically interesting proposals.

One IPB proposal was rather mature, with the prototype successfully tested in a working

environment. IPB proposals are technologically more demanding, and therefore in most cases

required commercial gap funding to reach technology readiness levels, where the project may be

presentable to an investor. IPB has been successful in achieving a mind-set change of its

37

managerial team and majority of researchers, a very difficult but fundamental element to pursue

any meaningful applied R&D and commercialization program. It has executed knowledge

transfer contracts with private sector firms, including a large multinational company and

increased participation in EU FP programs and outreach to industry has increased its visibility.

The institute has filed European and national patents for its breakthrough currency security

technology (called butterfly). It is also advising smaller RDIs in their transformation process,

including supplying a customized institutional management and monitoring system. IPB aims to

build the capacity (institutional and TRL of projects proposed) to have a solid base to compete in

coming IF CFPs.

Institute of Molecular Genetics and Genetic Engineering

43. The IMGGE was acknowledged as one of the premier R&D institutions in Serbia. It has a

long history, excellent talent, and reasonable laboratory facilities. Its research output (mainly

publications) has been growing, as has the quantity and quality of its collaborations within the

international R&D community. However, in recent times, IMGGE has focused mostly on basic

research, with very limited technology transfer and R&D commercialization outcomes, thereby

limiting its contributions to Serbian enterprises and the overall economy. Further, like many

other RDIs, IMGGE operates under a government policy regime that does not provide stable

base funding to support R&D operations and has limited authority to establish efficient and cost-

effective operations. As a consequence, its governance and management systems are not yet

aligned with international RDI best practices. The World Bank TA to IMGGE although slow at

the start because of difficulties in deploying reputed global experts has been instrumental in

achieving significant institutional-level improvements and mind-set change among its researcher

community.

44. Commercialization. IMGGE researchers submitted nine projects with commercial

innovative potential. After initial assessment, seven proposals were labeled as in the research

phase, as the disclosures resembled applications for further research rather than

commercialization. One proposal was marked as potentially interesting, for further market

analysis. Another proposal identified held potential as a service. IMGGE has entered into a

contract with a private sector company to transfer its probiotic technology and filed European

and national patents. In the knowledge transfer area also, IMGGE has been quite successful, for

example, securing an advisory services contract (over €1.5 million) from the European Patent

Office. IMGGE expects to pursue further knowledge and R&D transfer opportunities.

Institute for Medicinal Plant Research

45. IMPR was identified as a small-size research organization (out of 147 employees, 12 PhD

staff, 69 employees in production plant), established in 1948 to conduct scientific research on

herbal-based medicine. Later on, IMPR started operating its own production facility, which, at

present, provides about 85 percent of IMPR annual income (€3.7 million). Through the years,

IMPR built a reputation with customers in former Yugoslavia, predominantly selling teas and

herbal extracts of good quality at affordable prices. The World Bank team identified a number of

areas for improvement in management systems, as well as in the existing business model

(exploring new areas of research, opening up for private funding, exploring innovative

potentials, optimizing production, reaching out to the international market, and so on). IMPR

38

leadership did not show serious commitment for internal reforms (despite agreeing with the

World Bank team’s recommendations), citing lack of overall GoS commitment to sector wide

reforms. As a result, in early 2013, the World Bank team stopped providing further TA to IMPR.

46. Commercialization. IMPR has over 400 commercial products. Its dominant source of

income comes from the market, so it is successful in commercializing its R&D outputs. It has a

strong reputation in the market. However, the institute is not exploring its full potential or

increasing its product output by focusing on higher value added products.

Institute of Food Technology, Novi Sad

47. The World Bank team delivered a comprehensive report titled ‘Challenges and

Opportunities’, to FINS’ management outlining the current state and advice on short-, mid-, and

long-term actions to be taken to foster innovation to contribute more actively to the Serbian food

industry. FINS was identified as a small-size research organization, counting about 100 staff

members, with 55 researchers (25 hold a PhD degree). The annual income of FINS is about €2

million, with 25 percent coming from commercial operations (mostly services), while the rest

comes from the government-financed research projects. In the past, FINS had active

collaboration with the food processing industry in Serbia, which decayed as production vanished.

The staff and management attitude toward commercialization of research results was supportive,

although they were lacking resources and know-how in technology marketing. As FINS was

highly dependent on government financing, which is exclusively project based, FINS falls in the

category of RDIs whose commercialization potentials are heavily hindered by an inappropriate

financing mechanism. Although having a strategically good place in the Serbian food industry,

well-equipped laboratories, and staff motivated to compete on the market, FINS, as all other

RDIs in Serbia, is financially stimulated only to publish scientific articles. Unfortunately, FINS

management has not shown any serious interest in moving forward with institutional reforms or

supporting significant commercialization efforts. Therefore, the World Bank team scaled down

its TA support to FINS.

48. Commercialization. Following several visits by World Bank experts, FINS researchers

proposed six innovative projects for commercialization, from the fields of food supplements,

functional food, animal feed, and animal feed supplements. Although FINS did not have a

technology readiness level model adopted yet, according to their internal grading, their products

are rather advanced in technology readiness terms, although only one of the projects was in

negotiation with a potential client, which did not materialize owing to typical mismatched

expectations on IP value between the researcher and the prospective buyer.

39

Annex 3. Economic and Financial Analysis

1. Economic analysis was not carried out during preparation or recorded in the PAD, as it

was not required for Track 1 projects. However, below are important extracts from the report

titled ‘Estimation of the Effect of Participating in the IF Financing Program on Firm

Performance’, conducted by an independent consultant. It should be noted that the report was

concluded a few months before project closing in January 2016 and the final data was not

available.

2. Most funding (€7.12 million) under this pilot was executed by the IF and focused on the

‘Implementation of Financial Instruments Supporting Enterprise Innovation’, which consisted of

the Mini and Matching Grants Programs. The purpose and aim of the Mini Grants Program was

to expand opportunities for tech-based entrepreneurs by stimulating commercialization of R&D

and the creation of innovative enterprises based on knowledge through start-ups and/or spin-offs.

The Matching Grants Program aimed to expand collaboration opportunities for innovative micro

and small companies with strategic partners (for example,, private sector industry, R&D

organizations, and VC/private equity funds) with the goal of increasing private sector investment

in R&D and the commercialization of projects promoting new and improved products and

services. Overall, 469 projects were submitted by 326 firms. A total of 55 projects were

approved for IF support (out of the 469 projects); a 12 percent approval rate. These projects were

the subject of the impact evaluation analysis.

Table 3.1. Number of Applications, Program and Call

Applications Projects

Call # Call Date Matching Mini Total Matching Mini Total

1 January 31,

2012 0 57 57 0 11 11

2 August 31,

2012 40 106 146 5 9 14

3 April 15,

2013 39 85 124 5 13 18

4 October 28,

2013 40 102 142 2 10 12

Total 119 350 469 12 43 55

3. Under the Mini Grants Program, the financing was approved to 38 firms, out of which 36

were active at the end of the project period (January 2016). Over 80 percent of firms approved

for Mini Grants were established between 2012 and 2013 and 90 percent were start-ups or spin-

off firms established as a special purpose company to participate under the project. The amount

of funds disbursed for this program was €2.9 million. The contribution made by the participating

firms was at least €0.4 million. The quantitative evaluation for the Mini Grants was based on the

difference between the observed performance outcome of the supported firm (firms that were

awarded grants) and its hypothetical or counterfactual outcome if it would not have participated

under the program. Simple differences in sales changes between firms that received a Mini Grant

and firms that applied but were rejected revealed that the former increased their sales by an

additional €27,000 and employment by an additional five employees, on average. These are very

large effects because the pretreatment sales and employment were zero and zero to one

employees, respectively, for the vast majority of the supported firms. About 75 percent of Mini

40

Grant firms finished their projects as planned. About 16 out of the 38 (i.e. 42 percent) Mini Grant

firms had products that reached the marketing stage and 89 products were introduced by those

firms. Based on these numbers, the additional sales generated as a result of the Mini Grants were

approximately €1 million and it is expected that the investment in this component would be

recouped in a period of three years.

4. Under the Matching Grants Program only 11 firms could receive support as the amounts

awarded were much larger than under the Mini Grant Program. This number is inherently too

low to perform any reliable statistical analysis. Given the small sample, the analysis generated

very imprecise estimates and the independent evaluation report could not formally reject the null

hypothesis that the Matching Grants Program did not have a statistically significant effect on

sales and employment. The report reiterated that this result was likely because of the fact that

there were not enough observations to generate meaningful estimates of the Matching Grants

effect. The amount of funds disbursed for this program was €2.6 million. The contribution made

by the participating firms was approximately €0.63 million. Of the 11 firms, 4 submitted patent

applications, there were 13 collaborations with academia (including foreign universities), and 13

cooperative efforts with industry (23 percent of which were foreign partners). Data was

unavailable to quantify the value of these collaborations.

5. The report concluded that the budget of €6 million for the early-stage grant programs was

too low to have an economy-wide impact. The report estimates that the GoS should resume calls

and allocate at least €30 million per year to support 200–300 R&D projects for economy-wide

impacts. The firm contributions to match the government contributions (based on the above

examples) would be approximately €5.1 million. The €35 million injection into the system could

prove a great catalyst for the sector. The report suggested that the GoS address the ‘Valley of

Death’ and create government support for a second round of investment for IF graduate firms,

(which should at least partially be addressed by WB EDIF going forward).

6. Knowledge spillovers. The GoS should not expect that R&D knowledge, especially from

high-tech industries or services (most of the grants awardees), will drill down and assimilate

quickly in the traditional SME industries. Theory and experience suggests that traditional SMEs

require special attention and resources for spillovers to occur. To avoid polarization of the

Serbian economic structure, the GoS could consider special funding to increase innovative

activities to increase productivity of firms in the traditional industry.

41

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team Members

Names Title Unit Responsibility/

Specialty

Lending

Natasha Kapil Private Sector Development Specialist GTCIE Task team leader

Andrej Popovic Private Sector Development Specialist GFM01

Private sector

development

Hiran Herat Senior Operations Specialist GFM01 Implementation

Vinod K. Goel Science, Technology and Innovation

Advisor GTCDR

Science, technology,

and innovation

Itzhak Goldberg Innovation Policy Advisor Innovation

Kashmira Daruwalla Senior Procurement Specialist GGO03 Procurement

Aleksandar Crnomarkovic Financial Management Specialist GGO21 FM

Esma Kreso Environmental Safeguards Specialist GEN03

Environmental

safeguards

Amy Evans Environmental Safeguards Specialist GEN03

Environmental

safeguards

Lola Ibragimova Social Safeguards Specialist GSU03 Social safeguards

Nicholay Chistyakov Senior Finance Officer Disbursements

Ignacio Jauregui Legal Counsel LEGLE Legal

Supervision/ICR

Natasha Kapil Senior Private Sector Specialist GTCIE Task team leader

Vesna Kostic Senior Communications Officer ECAEC Project Visibility

Jasna Vukoje Senior Program Assistant ECCYU Liaison

Hiran Herat Senior Operations Specialist GFM01 Implementation

Vinod K. Goel Science, Technology, Innovation Advisor GTCDR STI

Kashmira Daruwalla Senior Procurement Specialist GGO03 Procurement

Aleksandar Crnomarkovic Senior Financial Management Specialist GGO21 FM

Nina Rinnerberger Environmental Safeguards Specialist GEN03

Environmental

Safeguards

(b) Staff Time and Cost

Stage of Project Cycle

Staff Time and Cost (Bank Budget Only)

No. of Staff Weeks US$, thousands (Including Travel

and Consultant Costs)

Lending 12.46 77,137.92

Total: — —

Supervision/ICR 53.53 266,563.43

Total: 65.99 343,701.35

Note: Includes budget for the additional financing.

42

Annex 5. Beneficiary Survey Results: Excerpts from an Impact Evaluation

1. Between January 31, 2012 and October 28, 2013, the IF managed four financing calls

during which 326 Serbian firms submitted applications for either the Mini or Matching Grants

Programs. The IF held about 20 visibility events per call to promote the CFPs and to explain the

Mini and Matching Grants Programs. Of the firms, 91 applied to the Matching Grants Program,

while 247 applied to the Mini Grants Program (12 firms applied to both the Matching and Mini

Grant Programs). Overall, 469 projects were submitted by the 326 firms. Of the 469 projects, 55

were approved for IF support; a 12 percent approval rate. These projects are the subject of the

IF’s impact evaluation analysis.

2. The qualitative evaluation of the IF consisted of personal interviews with the IF staff, the

Investment Committee members, several peer reviews, and many firms (both accepted and

rejected from Mini and Matching Grants Programs) as well as on-site visits to several firms and

to the local university.

3. Based on the quantitative evaluation, the main findings are as follows:

The IF is a credible and transparent platform, accessible to many Serbian businesses,

not just to a small group of entrepreneurs.

The IF succeeded in expanding the opportunities for tech-based entrepreneurs by

creating a lot of innovative start-ups/spin-offs. Almost half (156 out of 326) of the

firms that submitted the application for IF financing were created because of the IF

operation. In addition, there is a high survival rate among supported firms.

The IF succeeded in expanding the network of tech-based entrepreneurs by creating

a relatively high amount of collaborations with academia and industry (also

internationally).

Overall project objectives are adequately specified.

The Mini and Matching Grants Programs are clear, accessible, and effective.

Program evaluation and decision process is transparent and trustworthy.

Application process is similar to other programs (internationally).

The IF staff ranks high in professionalism and willingness to help and in flexibility.

The IF has overcome organizational difficulties.

The program’s implementation is effective—most of the projects achieved their

objectives.

4. The quantitative evaluation of the IF’s performance is based on a rich data set that was

gathered using specially designed questionnaires to firms, which had applied for the IF’s support,

as well as supplementary information from the Serbian Business Registry. At this stage,

43

information on Serbian firms from the Amadeus database had proven to add very little, if at all,

to the analysis.

5. The IF effect is defined as the difference between the observed performance outcome of

the supported firm and its hypothetical or counterfactual outcome in the case it would not have

participated in the IF program. The counterfactual outcome was estimated using statistical

techniques. This is done using a propensity score-matching algorithm to match a ‘twin’ firm

from among the non-supported firms to each supported (treated) firm. The matching ensures that

a twin firm is ex ante and is as likely to receive an IF grant as the firm that actually received the

grant. Then a matched difference-in-difference estimator was computed, which is based on the

differences in outcome differences (for example, growth in sales or employment) between the

treated firm and its twin.

6. Simple differences in sales changes between firms that received a Mini Grant and firms

that applied but were rejected reveal that the former increased their sales by an additional

€27,000 and employment by an additional five employees, on average. The evaluation analysis

generates similar results to the raw data: receiving a Mini Grant increases sales by €20,000 and

employment by five employees, on average, relative to what these firms would have experienced

had they not received the Mini Grant. The effect on employment is also statistically

significant, but the effect on sales is not. These are very large effects because the pre-treatment

sales and employment were 0 and 0–1 employees, respectively, for the vast majority of the

supported firms.

7. Only 11 firms received a Matching Grant. This is too low a number to perform any

reliable statistical analysis. Indeed, the evaluation analysis generates very imprecise estimates

and the null hypothesis that the Matching Grants Program did not have a statistically significant

effect on sales and employment cannot be formally rejected. It is reiterated that this result is

likely because there are not enough observations to generate meaningful estimates of the

Matching Grants effect. The results cannot therefore be interpreted as meaning that the

Matching Grants Program had little or no effect.

8. To fully measure the impact of IF grant, there is need for a longer time-horizon. The

evaluation analysis takes place at a point in time which is very close to the completion of the

project (or, more precisely, the part of the project being financed by the IF) and even closer to

the time new products and services were introduced to the market and the beginning of sales.

This may give a relative disadvantage, in the short run, to firms that actually did invest in R&D

to create new products and entering new markets, thereby, underestimating the effect of IF

support.

9. The finding of a positive IF effect on employment in the Mini Grants program is

consistent with increased R&D activity by these firms (as most of the R&D expenditures are due

to wages). The modal number of employees in 2011 among firms receiving a Mini Grant (before

the IF program was established) was zero. These firms were created to benefit from IF support

and are taking their first steps in establishing an R&D program. Simply put, these firms are still

at the inventive stage. Hence, increases in employment have been observed (but not a

corresponding increase in sales), which is interpreted to indicate that the IF has had a positive

effect on firms R&D expenditures (positive additionality of R&D).

44

Our policy recommendations are as follows:

(a) 52 projects in 4 years is too small a number to have an impact on the economy.

Unless the IF reaches a critical mass of 200–300 R&D projects at work annually,

efforts will have little impact on the Serbian economy. The GoS should resume

calls for application submission as soon as possible and allocate a budget of €30

million per year to meet the need for 300 R&D projects at work.

(b) The GoS should consider dealing with the ‘Valley of Death’ to create Government

support for a second round of investment for the IF graduate firms.

(c) The GoS should consider the creation of a special fund to increase the productivity

of firms in the traditional industry.

45

Annex 6. Stakeholder Workshop Report and Results

Press Releases and summaries from Stakeholder Workshops are available on the Innovation

Fund’s website at www.inovacionifond.rs under the Serbia Innovation Project section.

http://www.inovacionifond.rs/

46

Annex 7. Summary of Recipient's ICR and/or Comments on Draft ICR

Overview

1. From December 2011 to January 2016, the IF implemented the SIP the first EU IPA

Grant (€8.4 million) allocated to promote innovation in Serbia and administered by the World

Bank. The main objectives of the SIP are to encourage innovative entrepreneurship, boost

awareness regarding technological development and innovation in the economy, and address

missing elements of the Serbian innovation system, which is likely to be vital for improving the

competitiveness of the enterprise sector and Serbia’s long-term growth prospects.

2. The SIP consists of three core components:

Component 1: Supporting the operationalization and institutional capacity building

of the recently established Serbia IF

Component 2: Piloting financial instruments for technological development and

innovation by enterprises

Component 3: Encouraging selected public RDIs to engage in technology transfer

and assisting in formulating a RDI sector reform policy

3. The IF was responsible for implementation of Components 1 and 2 under the SIP, while

the third component was World Bank-executed.

SIP Implementation

Component 1: Capacity Building of the Serbia Innovation Fund

4. The IF was established in 2005 by the Innovation Law to encourage entrepreneurship and

manage financing for innovation, including participating in co-financing of programs, projects,

and other activities organized by international organizations, financial institutions, and the

private sector. As the SIP was the first pilot project implemented by the IF, one of the initial and

primary goals was operationalization and institutional capacity building of the IF.

5. Capacity building during the SIP focused on IF operational management and

infrastructure, as well as staff trainings on management of innovation and R&D financing

programs. The IF has engaged competent staff (15 permanent employees) and has established an

organizational structure that works effectively on the development of the financial instruments,

as well as the implementation and the monitoring of the awarded projects. Trainings cover

various aspects of the IF operations, both the administrative and business development, such as

management of innovation and technology support programs, management of innovations, FM,

and due diligence in early-stage technology development.

6. Additionally, a strong governance structure and processes in line with international good

practice have been established, based on trainings the IF staff has received during visits to similar

institutions in Israel, Finland, and neighboring Croatia. The IF has established a fully transparent

47

evaluation process that includes international peer review and an independent body IC,

responsible for evaluation of the project’s financing decisions.

7. The IF has also engaged a team of prominent advisors, ranging from strategic, operations,

and regional levels, who assisted and mentored the IF during the SIP implementation and

advised on future programs, strategy, operations, and procedures, in turn strengthening the IF

institutional capacity.

8. As of the SIP closing, the IF capacity is stable and operational even though it is still not

operating at full capacity due to the Government-imposed hiring restrictions and budgetary

measures for public sector organizations. The IF’s organizational structure has continued to

evolve with the introduction of the operations manager and senior legal advisor positions to

ensure expanded program operations (SRITTP, WB EDIF, Enterprise Innovation Fund [ENIF])

go uninterrupted and efficiently. Continued organizational improvements are expected on the

performance-based employee evaluation system front.


9. With the fundamental TA of the World Bank, the IF has established and piloted two

R&D grant programs: (a) the Mini Grants Program, targeting proof of concept and prototyping

stages among start-ups and (b) Matching Grants Program, targeting R&D in more mature

companies for development of technologies, products, and processes.

10. During the SIP, four CFPs were successfully conducted and the entire €6 million were

committed for the financing of 43 Mini Grants (approximately, €3 million) and 12 Matching

Grant (approximately, €3 million) projects.

11. By introducing the Mini Grants Program, the IF aimed to support early-stage, private, and

micro and small enterprises, which possess a technological innovation that have a potential for

creation of a new IP and clear market need. Following the initial success of the Mini Grants

Program, the IF introduced the second financial instrument called the Matching Grants Program,

designed to help companies struggling to address the significant financial investment associated

with the development cycle and the high cost of translating research into a commercially viable

product. The main goal of the Matching Grants Program is to expand collaboration opportunities

for Serbian innovative micro, small, and medium companies with strategic partners and increase

private sector investment in technology development and commercialization projects for new

and improved products/services.

12. After the initial completion of the first CFP for financing under Mini Grants Program, the

IF ran the CFP for both Mini and Matching Grants Programs simultaneously, by setting the same

deadline for project submission:

January 31, 2012 - CFP 1 Mini Grants

August 31, 2012 - CFP 2 for Mini Grants and CFP 1 for Matching Grants Programs

April 15, 2013 - CFP 3 for Mini Grants and CFP 2 for Matching Grants Programs

48

October 28, 2013 - CFP 4 for Mini Grants and CFP 3 for Matching Grants Programs

13. With respect to the implementation of the awarded project, the IF has been continuously

working on improvement of the GMs, procedures, and program documentation to make the

processes more fluid and easier for the companies. The IF team put significant effort to maintain

the evaluation process, not only fully transparent, but also within the set deadline of 8–10 weeks,

which complies with the some of the best examples of international practice such as Tekes

(Finland) and Matimop (Israel).

14. To complete its core program activities, including contracting of the new awardees and

implementation of the projects from the last CFP as well as disbursing funds to the existing

awardees, the SIP closing date was extended from November 30, 2014 to January 10, 2016. By

this date, all awarded projects were completed.

15. Overall, the implementation of the SIP was successfully conducted and issues that came

up during the implementation were resolved in collaboration with key IF stakeholders

(MOESTD, the EUD, and the World Bank) who also contributed to the IF’s organizational

capacity building. Internal monitoring of the set indicators have shown that all PDOs were

reached and/or exceeded (for an overview of the indicators, please see appendix 1).

Project Outcome (Achievements)

16. Addressing missing elements in the development of the innovation ecosystem.

Aiming to provide necessary funding to the innovative micro and small enterprises, the IF has

addressed some of the missing elements of the Serbian innovation system that are vital for

improving the competitiveness of the enterprise sector and Serbia’s long-term growth prospects.

Even though the financing under the SIP is mainly dedicated to the R&D activities, the programs

provide much needed support in the initial phases and give the companies a chance to focus on

the product/service development. Being the first financial instruments of its kind in Serbia, the

Mini and Matching Grants Programs initiated changes in the local innovation ecosystem and

contributed not only to the expending of complementary elements such as business incubators

and hubs, but also to the emerging of VC funds. Scaling-up of activities established under the

SIP is a step toward a more competitive start-up community, leading to establishing the

principles of the knowledge-based economy.

17. Encouraging innovative entrepreneurship. By providing programs that offer essential

seed financing for innovative projects, the IF has encouraged a certain portion of entrepreneurs

to think about further implication of their technologies. Furthermore, the IF has received

numerous project proposals coming from the university spin-offs, which have shown researchers’

potential and willingness to shift toward market-implicated research. With a total number of 471

submitted applications, of which 421 were eligible for evaluation, Mini and Matching Grants

Programs have shown that there is a clear demand for financing of the innovative SMEs’ R&D

activities. Most of the applications are from the ICT sector, combined with other high-tech

industries, but some are also from traditional industries such as mechanical engineering and food

and agriculture. Besides diversity in the industry area, the IF has also received project proposals

from all across Serbia, showing demand for the Mini and Matching Grants Program financing to

be present across various cities and university centers.

49

18. Enhancing awareness of the importance of innovation. By showcasing the results of

the financed projects and the SIP itself, the IF has been working on boosting the awareness of

technological development and innovation in the economy as a whole. Together with its

stakeholders, the IF has engaged in a number of events and promotion activities aimed to

increase knowledge on the significance of an innovation-driven economy, including recent

events such as SRITTP Launch (February 2015, Belgrade); visit to the Fund for Innovations and

Technology Development of Macedonia (March 2015, Skopje); Start-up Sauna at the Royal

Palace (March 2015, Belgrade); Europe Day Assembly of the City of Belgrade (May 2015,

Belgrade); 59th International Fair of Technique and Technical Achievements (May 2015,

Belgrade); Project Fair ‘The EU and Serbia in Action’ (May 2015, Niš); Local Slush

Competition (August 27, 2015 in Belgrade); Georgia's Innovation and Technology Agency visit

to the IF (October, 2015 Belgrade); and the SIP Closing Event (December 2015, Belgrade), to

name a few.

Project Outputs (Deliverables)

19. Since the start of the SIP, the IF has conducted four CFPs under the Mini Grants Program

and three under the Matching Grants Program, through which it provided financial support for 41

projects of young innovative companies under the Mini Grants Program, together with the 11

projects under the Matching Grants Program.

20. Overall, the awarded projects have shown a high percentage of success in technology

development, with most companies reaching their expected milestones of a working prototype, a

fully developed technology, and/or even a market-ready product.

21. Through four CFPs under the Mini Grants Program and three CFPs under the Matching

Grants program, the IF has supported over 300 jobs within innovative Serbian SMEs.

Additionally, the IF has also supported the engagement of around 90 highly qualified R&D

advisors on these projects. More than 60 persons engaged in the financed projects hold a PhD

degree from various technological backgrounds. The IF’s programs have also proven to foster

the collaboration between the private sector companies and the academia, with more than 20

different RDIs and faculties engaged on the financed projects.

22. Although, it is still early to determine the full impact of the IF’s financing on companies’

results, the awardees from Mini and Matching Grants Programs have already achieved some

notable results:

19 awardees have generated revenue from the sale or lease of their innovative

products.

9 awardees have established international cooperation.

2 awardees have secured distributors for the U.S. market.

On a national level, 33 patent applications have been submitted, together with 10

trademark applications, and 6 applications for authorship protection.

50

To protect their IP abroad, awardees have submitted 13 Patent Cooperation Treaty

applications, 4 EPO applications, 5 U.S. patent applications, along with patent

applications in Australia, Euro-Asia, and Japan.

18 awardees have established new business cooperation or partnerships.

Bank Performance

23. The World Bank team has been very active in supporting the IF team, pertaining to both

the building of overall organizational capacity and designing and managing of the financial

instruments. The World Bank encouraged the IF to take a more decisive role in the

implementation of the SIP and be more proactive in strategic planning, which contributed highly

to establishing the IF’s capacity. Key areas in which the World Bank has provided TA are the

following:

Establishing IF capacity through organizing various in-house training for the IF staff,

as well as working with the IF team on the training plan

Designing of the financing programs, including program documentation

Organizing and managing of the visibility events

Continuation of financing and scaling up of the existing grant schemes

Development of new financing programs

Strategic planning

24. The results achieved during project implementation were in large part due to the

accessibility and support of the World Bank’s task team leader. The World Bank team and their

advisors were instrumental during the drafting of program documentation and highly valuable

during project implementation and improvement of the project processes.

25. The World Bank team was also very determined in securing funds for the next rounds of

financing under Mini and Matching Grants, as well as expending the portfolio of IF programs

through the SRITTP that includes a new collaborative grant scheme and the establishment of the

TTF.

Conclusions and Final Remarks

26. During the period of the SIP implementation, the IF has managed to build capacity and

has established itself as a trustworthy institution, fully capable of absorbing funds and creating

and implementing new projects. M&E analysis of IF’s Mini and Matching Grants Programs

based on indicators defined by the SIP was completed in November 2015 by the consultant

company, Applied Economics (Israel). Key findings of this analysis include that the IF is a

credible and transparent institution, accessible to many Serbian businesses. In addition, the report

identified relatively high levels of collaboration between academia and the private sector

industry (both domestically and internationally) involvement in the beneficiaries’ projects. Mini

51

and Matching Grant Programs were found to be clear, accessible, and effective, with the

application process being similar to other international programs and the evaluation and decision

process being transparent and trustworthy.

27. Following the achievements of the SIP, the IF has been actively working on securing

additional funds to scale up activities under the existing programs to achieve higher impact on

the Serbian economy. This effort resulted in €8.1 million being provisionally allocated to the IF’s

Mini and Matching Grant Programs over the next three years as part of the Loan Agreement

between the World Bank and the Republic of Serbia under the Serbia C&JP. Furthermore,

additional funds are expected from the IPA 2014 pot to be available to the IF in late 2016. With

the prospect of securing funds for resuming the Mini and Matching Grants Programs in 2016 and

with adequate Government support, the expectations are for the IF to remain an integral catalyst

in the Serbian innovation ecosystem.

1

Appendix 1. Performance and Monitoring Indicators

(as per Attachment to Supplemental Letter No.2 of the Project Agreement)

Indicators Number and Definition Cumulative Target Values Actual Values

Frequency Data Source/Methodology YR1 YR2 YR3 YR4 June 30, 2015

Indicator 1: Amount of innovation financing

mobilized by IF in addition to this EU IPA

project

— — €10

million €20 million

WB EDIF €28

million1

SRITTP €6.9

million2

6 months Internal reports of IF

Indicator 2: Number of active technology

start-ups funded 2 5 8 10 38

3 6 months Internal reports of IF

Indicator 3: Number of new products and

processes launched by beneficiary

enterprises

0 2 6 8 194 6 months Internal reports of IF

Indicator 4: Number of technologies

transferred by RDI’s 0 1 2 3 3

5 6 months

Consultants’ reports to IF

and IBRD

Indicator 5: Dissemination of policy

recommendations for RDI sector reform — — —

Disseminated to

stakeholders

Disseminated to

stakeholders

Upon

completion of

TA

Consultants reports,

MOESTD and IBRD

Note:

1. WB EDIF: Support approved by the Western Balkan Investment Framework on December 7, 2011 for ENIF, a regional VC fund. The GoS passed the

conclusion 05 no. 337-195212012 of 22 March 2012, appointing the IF as an organization for implementing the WB EDIF on behalf of the Republic of Serbia

and approved the provision of financial contribution of the Republic of Serbia in ENIF in 2012 by approving the transfer and payment of RSD 21,528,000.00 in

2012 through IF.

2. Budget support for SRITTP, to be financed through IPA 2013 is launched in February 2015 (total project budget is €6.9 million, out of which €4.9 million is

funds of IPA, €1 million budget of the Republic of Serbia, and €1 million private co-financing ).

3. IC decision to finance project proposed by Stamar was revoked by the IC; Project Fastro was terminated; number of Mini Grants projects financed in each of

four CFPs, 10, 8, 13 (excluding Stamar and Fastro), and 8 respectively. Companies DNET Inovacioni Centar, Ekofungi and Coprix Media were awarded a

second Mini Grant and one company that finished a Mini Grant project was subsequently awarded a Matching Grant in one of the later CFPs (Techno Foam).

4. Techno Foam, Strawberry Energy, Angros, Adecom, iStreetLight, Euroheat, Ekofungi, Coprix Media, mBraintrain, Trk inovacije, Spik Briket, Brainstorm,

Novel IC, Eipix Entertainment, Intens Net, Vizus, BSK, TajfunHil, and Morena Inzenjering reported product launch.

5. Spin-off from IMGGE; IMGGE - service licensed/contract research; IPB - product sale (sale started before the implementation of the SIP, but the TA from the

project resulted in the increase).

Intermediate Results (Part A): Built IF’s capacity for designing and managing financial instruments for enterprise innovation, technology transfer and

mobilizing funds

Intermediate Result Indicator 1: Number of

financial instruments designed and 2 2 3 3 4


2

Indicators Number and Definition Cumulative Target Values Actual Values Frequency Data Source/Methodology

introduced


IF managers trained and applying newly

learnt skills



start-ups mentored 2 5 8 10 51


Intermediate Result Indicator 4: Mechanism

to support training/mentoring of innovative

start-ups established

No Yes Yes Yes Yes 6 months Internal reports of IF


networking and educational events

organized for enterprises, academia and

research


Intermediate Result (Part B): Piloted financial schemes for technology development and innovation in enterprises


funded projects 5 10 15 20 52



funded enterprises 5 10 10 15 48


Intermediate Result (Part C): Assessed a selected number of RDIs and their innovation potential with recommendations for technology

commercialization

Intermediate Result indicator 1: No. of

RDIs identified for technology transfer and

commercialization

3 4 4 4 10 6 months Consultant reports,

MOESTD and IBRD


RDIs assessed jointly by international

experts and RDI management

0 2 2 2 4 6 months Consultant reports,

MOESTD and IBRD

Note:

1. The IF has designed the Collaborative R&D Grant Scheme under the SRITTP, while TTF was established under the same program. It is expected that both

programs will officially commence in the first part of 2016.

2. Total number of IF managers trained from the beginning of the IF operations.

3. All Mini Grants awardee companies have received Enterprise Training together with three companies that were preselected in CFP1. Besides that, the IF has

organized a local ‘Slush’ competition that also included mentoring sessions for participating companies.

4. The major networking and educational events held were project launch and launch of the Mini Grants Program, launch of the Matching Grants Program, the IP

workshop (for RDI’s and key stakeholders), EDIF workshop (for RDIs and private sector and key government stakeholders), and four info days for the 53

selected projects (addressing project implementation and training held by Serbian IP office), four Enterprise Training sessions for awardees, presentation of the

IF Annual Report, knowledge economy workshop, industry and RDI consultation regarding the new technology transfer project design, consultations with local

3

Indicators Number and Definition Cumulative Target Values Actual Values Frequency Data Source/Methodology

technology transfer offices on the cooperation and functioning of centralized TTF.

5. IF terminated the Financing Agreement with one of the CFP4 Matching Grants awardee, the Stan Technologies company, because it was unable to secure its

part of project co-financing, and subsequently, the company requested to be withdrawn from project implementation. On December 18, 2014, the IC made a

decision to terminate the project.

1

Annex 8. Comments of Co-financiers and Other Partners/Stakeholders

Comments from Co-financier: Delegation of the European Union to the Republic of Serbia

As a general comment, this is a very specific project – the first one of this kind in the EU Delegation to

Serbia – that provided significant resources in the form of grants (to the amount of EUR 6 million) to the

innovative private sector companies. Furthermore and as far as I am aware, it was for the first time in

Serbia that start-up companies were eligible for grants as well. Both of these initiatives proved to be very

successful and turned this icebreaker into one of the flagships of the EU funded projects in the area of

Research, Development (R&D) and Innovation.

The piloting of financial instruments for technological development and innovation in enterprise, in the

most transparent way, was rather revolutionary for a Serbian innovation market. At the same time, the

capacity building of the Innovation Fund was not less challenging. Although the Innovation Fund was

established by the Innovation Law in 2006, it became operational only in 2011, with resources provided

from the European Union through the Instrument for Pre-Accession Assistance (IPA) and under the

umbrella of the Serbia Innovation Project.

Finally, the customized technical assistance was successfully provided to selected Research and

Development Institutes in Serbia.

All of the above was achieved in a very difficult and challenging political and economic environment.

During the project life cycle, three Ministers changed in the Ministry of Education, Science and

Technological Development. These changes included significant fluctuations at the level of State

Secretaries and Assistant Ministers as well. The political instability was followed with budgetary

constraints and delayed funding. At one point, the employees of the Fund were without salary for three

months. Lack of general understanding of importance of innovation and R&D for the economic recovery

was always present at various levels. On the other side, three Acting Directors of the Innovation Fund

have also changed and the Fund was understaffed most of the time.

Piloting this intensive project, with two different grant schemes and EUR 8.5 million budget, under the

above mentioned circumstances was rather difficult, to say the least. It was even more demanding if we

have in mind the specific reputational risk of the EU as the biggest donor in the region and the World

Bank.

Apart from the excellent results achieved during the project implementation – and most probably because

of them, the Serbia Innovation Project laid ground for further interventions in this area. In that regard, we

are already implementing the Serbia Research, Innovation and Technology Transfer Project with the

Collaborative grant scheme for public sector R&Ds and SMEs. Furthermore, replication of the Mini and

Matching grant scheme is in the pipeline for IPA 2014. The project also demonstrated that Serbia has

many brilliant, young and innovative entrepreneurs that just need a more conducive environment for their

success.

An excellent visibility of the project (including the one-page story in the Financial Times) proved to be

relevant for the change of perspective and demonstrated that costs of innovation (and R&D) should not be

regarded as expenditure but as an investment. Of course, much more will have to be done to make this

perspective live.

As one of the lessons learned, it would be good to underline the necessity for having the project contact

point in Belgrade, dealing mostly with this or any other specific project. This would improve daily

communication and contribute to resolution of urgent and technical issues.

2

I would also like to use this opportunity to commend the very good World Bank team and specifically the

Team Leader, Ms Natasha Kapil. Their knowledge, experience, professionalism, efficiency and

persistence contributed significantly to the huge success of this project.

The ICR report provides a realistic and correct picture of the project and offer an excellent analysis of all

relevant project elements: from context and design to implementation and assessment of the outcome.

Notwithstanding this, please note that the above comments should not be regarded as an approval of the

final report for this project. The final narrative and financial report will be commented once submitted for

approval, as per the administration agreement with the World Bank.

Dejan M. Šuvakov Programme Manager for SMEs, Competitiveness, Innovations - Operations II

Delegation of the European Union to the Republic of Serbia Vladimira Popovića 40, 11070 Belgrade, Serbia

Comments from Stakeholder: Business Technology Incubator of Technical Faculties Belgrade

The Business and Technology Incubator of Technical Faculties Belgrade (BITF) was the first technology

incubator launched in Serbia in 2008. Since the early days of BITF, we have faced the lack of funding

sources for startups and innovation programs. In addition, there was no systematic approach and funding

for incubators in general, which made this period of time extremely difficult. At that time, there was the

Innovation Fund (IF), established under the Innovation Law in 2006, however it was not operational until

2011, when it was activated with the World Bank support through the Serbia Innovation Project (SIP),

financed by the European Union through the Instrument for Pre-Accession Assistance (IPA 2011) funds

for Serbia.

The beginning of the Serbia Innovation Project implementation was marked by distrust of the government

in the potential for innovation and fear of not having enough projects on the one hand, and by distrust of

startups that the main criterion in selecting projects for funding would be innovation on the other hand. At

the same time, it was more than difficult to raise the capacity of the Innovation Fund due to lack of local

experts in the field of innovation, especially in the field of financing innovation.

As for startups, it was noticeable that the selection of the first group of projects funded from the

Innovation Fund (beginning of 2012) encouraged other startups, so that every subsequent call for

proposals for small grants (Mini Grants) attracted more and more project proposals, which resulted in the

greater visibility of the innovation potential of Serbia at all levels. There is no doubt that through the

entire implementation period of the Serbia Innovation Project, the level of support to innovation in Serbia

was constantly increasing. At the same time, all stakeholders learned a lot at the macro level from the

example of impressive development in Israel, and at the micro level, with the preparation of project

proposals according to the demanding World Bank methodology which was a learning process for small

companies in Serbia.

Now it is necessary to multiply the positive effects of this project through the future work of the IF in

order to advance the technological development of the country. For that purpose, the Innovation Fond,

being an organization that received a significant capacity building support through the Serbia Innovation

Project, must secure funding and continuously invest in innovation development and startups by

providing small grants or similar types of support.

Gordana Danilovic Grkovic,

BITF, Manager

3

Annex 9. List of Supporting Documents Available Upon Request and in Project Files in

WB Docs

(a) Project Appraisal Document

(b) Data and statistics on grant beneficiaries

(c) Impact Evaluation Report - Estimation of the Effect of Participating in the IF

Financing Programs on Firm Performance (October 2015)

(d) Project Aide Memoires

(e) Project Progress Reports

(f) Project Implementation Status Reports

(g) IF Annual Reports

(h) RDI reports, workshop materials, manuals, and so on

4