Embed Size (px)
Citation preview
SEMINARPromotion, development, support and evaluation of
Innovation
STI policies and impact evaluation
FINEP / ABC
Rio, 28th August 2017
SERGIO SALLES-FILHO
summary
• Initiatives and trends around the world
• FAPESP evaluation policy
• Cases
• Concluding remarks : challenges and questions
Initiatives around the world
+International trends
• ex-ante – interim – ex-post – ex-post facto
Complete Cycle
• Internal efforts
• External ad hocSystemic
• Quantitative + Qualitative
• Secondary + PrimaryEvidence based
• Economic
• Social
• Environmental…Multidimensional
The Inno-Appraisal Project, MIoIR - UK
• Project InnoAppraisal
• Overview of purposes, methods, topics, impact, usefulness
• Includes 242 evaluation reports linked to 158 unique policy measures
5Slide borrowed from Jakob Edler, Manchester Institute of Innovation Research
Budget and Planning
6
No
53%
Yes
47%
Dedicated Budget for
Appraisal
No/D
on't
Know
33%Yes
67%
Appraisal foreseen
and planned for
Slide borrowed from Jakob Edler, Manchester Institute of Innovation Research
The Inno-Appraisal Project, MIoIR - UK
Multidimensionality AND Stakeholders
7
25%
6%
5%
15%
24%
48%
38%
22%
22%
32%
27%
56%
72%
63%
44%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Economic Impact
Social Impact
Environmental Impact
Scientific Impact
Technological Impact
Only participant Participants and beyond No
Slide borrowed from Jakob Edler, Manchester Institute of Innovation Research
The Inno-Appraisal Project, MIoIR - UK
Sponsors of the evaluation
8
92%
21%
7%
8%
79%
93%
0% 20% 40% 60% 80% 100%
Programme Owner / Manager
Other Governement Department
Other Public Bodies
Yes No
Slide borrowed from Jakob Edler, Manchester Institute of Innovation Research
The Inno-Appraisal Project, MIoIR - UK
Data Sources and Collection Methods
9
83%
72%
71%
62%
60%
47%
22%
14%
2%
17%
28%
29%
38%
40%
53%
78%
86%
98%
0% 20% 40% 60% 80% 100%
Monitoring Data
Interviews
Existing Surveys / Databases
Document Search
Participant Surveys
Focus Groups / Workshops / Meetings
Non-Participant Surveys
Peer Review
Technometrics / Bibliometrics Search
Yes No
Slide borrowed from Jakob Edler, Manchester Institute of Innovation Research
The Inno-Appraisal Project, MIoIR - UK
SIPER: Science & Innovation Policy Evaluations Repository
+ SIPER: Components
• Repository of existing evaluation reports (pdf) (focused on EU MS and OECD countries)
• Coordinated with
OECD-World Bank innovation policy platform (IPP, www.innovationpolicyplatform.org)
EU Research & Innovation Observatory (RIO)
11
+ SIPER: participants
• Developed by Manchester Institute of Innovation Research, The University of Manchester
• International partners:• Latin America (GEOPI/UNICAMP) • France (IFRIS)• More international linkages in discussion…
12
The experience of FAPESP
Fapesp Evaluation Policy
OBJECTIVES
1- Planning – on-going and new programs/instruments
2- Institutional learning
3- Accountability – differentcategories of stakeholders
4- Advocacy
Fapesp Evaluation Policy main Axes
Systematic / permanent evaluation
Secondary and Primary data + Qualitative and Quantitative
Counterfactuals
Quality control
Complementary external evaluation
FAPESP web site on evaluation
• http://www.fapesp.br/en/evaluation/
Some cases of impact evaluation
+Some cases
Who really takes advantage of fiscal incentives to innovate in ICT sector?
What is the profile of most innovative SMEs?
Does peer review system select more productive recipients for scholarships?
Does STI programs perform better than isolated projects in the same field?
How to improve impacts of innovation programs over small holders?
Does matching public and private funds engage companies and ROs?
Case 1Who really takes advantage
of fiscal incentives to innovate in ICT sector? Impact study on the ICT law in Brazil (1998-2009)
Compared to the sector
Value Chain – variation per segment(added value – 1998 a 2008)
Segments of value chain
Small and
micro firms
Medium size
firms Large firms
Supply chain Var. % 2.2 10.1 6.5
Conception Var. % 3.3 12.6 10.0
Hardware Var. % 5.3 11.1 10.9
Software Var. % 5.5 8.6 16.3
Design Var. % 4.4 17.9 13.8
Marketing Var. % 1.6 11.3 8.5
Answer:1- Large MN companies that does not want/need to invest in R&D but become competitive over fiscal incentives
2- Medium size national companies that takes more advantage of value creation and appropriation through R&D and fiscal incentives
Case 2What is the profile of most innovative SMEs?Small Business Innovation Program at FAPESP - PIPER$ 52,9 millions(214 projects)Compared to SBIR in the USA
+ PIPE – most innovative profile
Mostinnovative
projects
Companies are spin offs of other
companies
No incubation
Partnership withResearch
OrganizationsProject
coordinator ispart of the board
of directors
Company was in operation
+ PIPE – Principais Conclusões
-10
0
10
20
30
40
50
60
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Milh
õe
s
Fluxo de Caixa - PIPEvalores de 2007 - com contrapartida
Beneficios Custos Fluxo Líquido
TIR = 45,7Análise Benefício-custo
5,98
Answer:Potentially all may be innovative,
but
Spin offs + affiliation of researcher to the company + relationship with Ros
and
Concentrate in 10% of projects
Case 3Does peer review system select more productive
recipients for scholarships?FAPESP Scholarships
Undergraduate; MSc and PhD
Treatment group: FAPESP ex-scholarship holders
Control group: other ex-scholarship holders (granted without peer review process)
+Publications and the peer-review effect :
quasi-experimental evaluation awarded vs rejected in Fapesp but had grant from other funding agencies
Ratio Effects p-value CI 95%
ScIPubl. Average 0,99 0,880 0,85 1,15
JIF Publ. Average 1,22 <0.001 1,09 1,36
MScPubl. Average 1,04 0,310 0,97 1,11
JIF Publ. Average 1,13 <0.001 1,07 1,20
PhDPubl. Average 1,37 <0.001 1,24 1,51
JIF Publ. Average 1,07 0,022 1,01 1,14
Effect of FAPESP scholarship on scientific production related to ScI, MSc and PhD with PS
Answer:Yes, at least in FAPESP case compared to non-FAPESP ex-holders
It’s also correlated to other impacts as for income level and job positions
Case 4 – Does STI programs perform better than isolated projects in the same field?
BIOTA CASE
Treatment: Biota projects
Control group: similar biodiversity projects out of Biota
Publications
The effect was weaker when all output cited in researchers’ CVs was considered, falling to
about 30–40 % and displaying less statistical significance. No differences were observed
with regard to supervision of theses and dissertations.
Another important approach to the evaluation of output is analysis of discontinuity in
the time series of average annual production per researcher. Analyzing whether this series
changes after the start of the project, and if so how, helps corroborate the findings of PS
estimation, as shown in Table 3. In this case, total production by the researchers concerned
was analyzed, since focusing only on articles originating in the projects would have been
incompatible with the time series. A negative binomial regression model was used for this
analysis. Asnoted above, the adjusted model included all interactions among the following
three variables: centered time, research group, and project inception (Table 3). Interactions
were estimated up to the third order. The parameters thus represent the average slope for
production before project inception, changes in the average representing discontinuity due
to the project, and the slope after project inception, enabling them to be tested against the
differences between the groups.
Figure 3 presents the discontinuity plots for (a) average production of articles per
researcher and (b) average number of distinct co-authors per researcher. In the case of
articles, the increase is smoother for the BIOTA group, with no discontinuity in average
production (δ2 = 0.14, p = 0.42), but the slope of the growth curve after year 0 (project
inception) is significantly steeper than before year 0 ([δ2 + δ4] = 0.13, p = 0.03). In other
words, the average number of articles produced per year rises faster. The Control group
displays a different effect: production falls shortly after project inception (β3 = −0.27,
p = 0.008) and then returns to a rate of growth similar to that seen before project inception
but at a lower average level. The effect for co-authors is the same in terms of statistical
significance but visually clearer.
Discussion
Because the BIOTA Program is a research network with rigorous criteria for project
selection, peer recognition, and multiple but well-aligned programmatic objectives (bio-
diversity characterization, conservation and sustainable use), with scope for large-scale
Table 3 Impact of BIOTA on scientific production indicators, estimated by multivariate and weightedpropensity scores
Model 1—multivariate Model 2—weighted PS*
BIOTAeffect
p value CI 95 % BIOTAeffect
p value CI 95 %
Articles published as result of project 1.2 0.19 [0.9–1.8] 1.9 \ 0.0001 [1.2–3.1]
Total articles in Lattes CV 1.1 0.31 [0.9–1.4] 1.3 0.19 [0.9–2.0]
Co-authors of project articles 1.2 0.37 [0.8–1.8] 2.1 \ 0.0001 [1.3–3.3]
Co-authors in Lattes CV 1.2 0.28 [0.9–1.5] 1.4 0.28 [0.8–2.3]
Supervisions (project) 1.2 0.27 [0.8–1.7] 1.2 0.37 [0.7–2.2]
Supervisions (Lattes CV) 1.2 0.40 [0.9–1.6] 1 0.98 [0.6–1.8]
* Adjusted by variable “ funding category: Thematic Projects” and lagging variable
Scientometrics
123
Author's personal copy
Answer:- yes, particularly in scientific production, but also in other variables related to cooperation for R&D as well
Case 5 -How to improve impacts of innovation
programs over small holders?
The INCAGRO program in Peru
970 interviews
Answer: reorienting the program
Reorienting the
Program
Cluster 1
Instruments towards a more strucuturedinnovation system
Beyond thethreshold
Cluster 2
Promoting capabilities togive access to knowledge
On the threshold
Cluster 3
Institutional stability and capabilities to give accessto knowledge
Structuralconditions to reachthe threshold
Case 6 - Does matching public and private funds engage
companies and ROs in innovative projects?
EMBRAPI Case in Brazil
Pilot phase: 63 projects; 44 companies
How companies evaluate the success of projects
+ 80% within or above expectations
Very high
Low
+Behavioral changes: ROs
New prospecting processes
Formal competences in
R&D Management
Established project management
processes
Internal policy of valuation and negotiation
Organizational rearrangements
Answer- yes, when followed by organizational and managerial requirements for governance
Conclusions and challenges
The cultural challenge
• Evaluation as in-built component of programs and policies
• Evaluation budget and resources
• Stakeholders• Funding Agencies / Government Agencies / Policy
Makers
• Researchers and scientific community
• ROs
• Companies
• Accounting offices
• Society…
The Big Data challenge
Bibliometrics
Altmetrics
Data Science
Social Network Analysis
+
Year X-1 Year XYear
X+1
Year
X+2
Year
X+3
Year
X+4
Year
X+5
•Proposalselaboration and analysis
Ex-ante
• How the project/activity is doing?
Monitoring• What are the
resultsachieved
Ex-post
• What are theimpacts?
Ex-post facto
Project development
The Complete Cycle Challenge
Questions to address
Is the large set of policy instruments bringing results in terms of innovation and impacts over economic and social indicators in Brazil (without crowding out)?
Answer: We simply do not know
Is there systematic evaluation being implemented in Brazilian agencies?
Answer: Only for few exceptions
How to make evaluation part of the policy rationale?
Answer: Let’s discuss it seriously?
