Embed Size (px)
Citation preview
Peer review and knowledge dynamicsTerttu Luukkonen
Research Institute of the Finnish Economy
ERC WorkshopMonitoring the performance and quality of
peer review systems28-29 November, 2013, Brussels
Focus
• Evaluation of research proposals• Groundbreaking/pathbreaking/frontier/
transformative research
Content
• Defining groundbreaking research– potential differences between fields in the
understanding• Phases in the emergence and growth of new
research areas • Comparison of peer review with other
methods in its ability to select groundbreaking research
• Conclusions
Categories of groundbreaking
• Discovery of a novel phenomenon (serendipitous discoveries and others)
• New method or technique or their combination as an enabler
• Access to new data• General explanations (paradigms and other)– unsolved ‘big questions’ (Laudel and Gläser, 2012)
Impacts of groundbreaking researchon
• Own discipline or research field• Several other research fields• Opening up new research fields• Merging fields/interdisciplinary areas of
research
Different perspectives• Selection of proposals - Forward look– Promise of opening up new avenues of research– Enabling new research directions – New perspectives– Paradigm shifting, revolutionary – Great uncertainty
• Backward– Pin down what gave rise to the observed
development – often a longer process and several contributors
LS PE SH0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Difference between excellence and groundbreaking
To a great or some extentTerttu Luukkonen: interviews with 24 ERC peer reviewers, Luukkonen, 2012
Different definitions of groundbreaking and excellenceERC peer review panelists (Luukkonen, 2012)
• Groundbreaking– Synonyms: pathbreaking, cutting-edge, frontier– Originality, novelty , revolutionary, paradigm shifting
• Excellence– Originality, novelty, going beyond current state of the
art, making a difference for the development of science, but also
– Robustness of the research, methodological rigour, use of up-to-date methodology, coherent discussion of the research problem and purpose
Development of ideas and scholars over time
Early adaptors
Early Majorities
Late majorities Latecomers
IDEAS
SCHOLARS
TIME
NR OF IDEAS & SCHOLARS
Brown, 2012
Project selection methods and cycles of ideas
Early adaptors
Early Majorities
Late majorities Latecomers
TIME
NR OF IDEAS & SCHOLARS
ERC
HHMI
Bibl. methods
National RCs National RCs
National RCs
ERC peer review system Luukkonen, 2012
• Evaluation criteria • Quality of the peers • Panels have to consider feasibility and risks– Capabilities of the investigator– Instruments and equipment– Contingency plans– Avoid speculation and dilettantism– Put in context, tradition – Reasonable risk
Laudel and Gläser on ERC, 2012• “ERC grants have impact on research because, at least, some of
them fund scientific innovations, the exploitation of recent discoveries, or answers to ‘big questions’ across all discipline groups”
• The funded research has epistemic properties not usually met by grants from national funding agencies– Contradicting the mainstream– Addressing the community’s blindspots– Linking otherwise separate communities– Strategic & technical uncertainties– Complexities in equipment, approaches– Length of time it takes to conduct the research
Project selection methods and cycles of ideas
Early adaptors
Early Majorities
Late majorities Latecomers
TIME
NR OF IDEAS & SCHOLARS
ERC
HHMI
Bibl. methods
National RCs National RCs
National RCs
Bibliometric measures suggestedHörlesberger et al., 2013
• Novelty– Timeliness: how recent are the publications listed on the
reference list of application – Similarity of proposal with emerging topics
• Risk– Similarity of the proposed research to the investigator’s
previous research– Interdisciplinarity
• There has to be some body of publications in the field for the measures to be counted
Project selection methods and cycles of ideas
Early adaptors
Early Majorities
Late majorities Latecomers
TIME
NR OF IDEAS & SCHOLARS
ERC
HHMI
Bibl. methods
National RCs National RCs
National RCs
Varieties of peer review
• Robustness– Use of international vs national experts– Independence of the panels– Their evaluation instructions and criteria
• Further features:– Fine-grained vs. rough marking (Langfeldt, 2001)
– Remote reviews vs. or, in addition, panelists’ reviews– Degree of interdisciplinarity of panels– Panels rank or rate – Evaluation criteria: ground-breaking vs. excellence
Conclusions• Peer review conservative?• Not just one peer review, but many varieties in quality,
criteria, organisation• The way peer review is organised and applied makes a
difference• Peer review in combination with the conditions of the
scheme can make a difference for progress of science• Quality of peer review provides legitimacy to the scheme
and affects quality of the applicants• A risk that a thorough monitoring of peer review shifts the
system towards more conventional proposals – short term indicators
Literature– Braun, Dietmar. 2012. Why do scientists migrate? A diffusion model, Minerva, 50: 471-491. – Grant, Jonathan and Allen, Liz. 1999. Evaluating high risk research: an assessment of the Wellcome
Trust’s Sir Henry Wellcome Commemorative Awards for Innovative Research, Research Evaluation, 8: 201-204.
– Hörlesberger, Marianne, Roche, Ivana, Basagni, Dominique, Scherngell, Thomas, Francois, Claire, Cusax, Pascal, Schiebel, Edgar, Zitt, Michel, and Holste, Dirk. 2013. S concept for inferring ‘frontier research’ in grant proposals, Scientometrics, 97: 129-148.
– Langfeldt, Liv. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcomes, Social Studies of Science, 31/6: 820-841.
– Laudel, Grit and Gläser, Jochen. 2012. The ERC’s impact on the grantees’ research and their careers (EURECIA Work package 4 summary report). January 2012.
– Luukkonen, Terttu. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices, Research Evaluation, 21 (2012), No. 1, pp. 48–60.
– Nedeva, Maria. 2012. Peer review and path-breaking research: selection practices of research funding organisations. Unpublished.
– Wagner, Caroline S. and Alexander, Jeffrey. 2013. Evaluating transformative research programmes: A case study of NSF Grants for Exploratory Research Programme, Research Evaluation, 22: 187-197.
UNI project: Universities, funding systems, and the renewal of the industrial knowledge base – a project funded by Tekes, 2012-2014; coordinated by Terttu Luukkonen; empirical data gathering with research group leaders in universities in Finland and the UK.
