The Future of Peer Review - European Association of ...€¦ · Mark Ware: ‘far from being in crisis, peer review remains widely supported and diversely innovative’ Ware M (2011)

The Future of Peer Review

Irene Hames, PhD, FSB @irenehames Editorial and Publishing Consultant

Council Member, COPE (Committee on Publication Ethics)

ORCID: http://orcid.org/0000-0002-3806-8786

EASE/ISMTE Conference, Blankenberge, 23 September 2013

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

http://orcid.org/0000-0002-3806-8786

Not that long ago … .

Irene Hames, EASE/ISMTE, 23 Sept 2013 2


What is (editorial) peer review?

Peer review in scholarly publishing is the process by which

research output is subjected to scrutiny and critical

assessment by individuals who are experts in those areas.

(Hames, 2012, in Academic and Professional Publishing, Chandos Publishing, Eds

Campbell, Pentz and Borthwick, p.16)

and

…the critical assessment of manuscripts submitted to journals by

experts who are not usually part of the editorial staff

(ICMJE, International Committee of Medical Journal Editors, http://www.icmje.org/)

http://www.icmje.org/

Enormous scale

~28,100 active scholarly peer-reviewed journals

publishing ~1.7-1.8 million articles a year

The STM Report: an overview of scientific & scholarly journal publishing, Nov

2012, M Ware & M Mabe

http://www.stm-assoc.org/2012_12_11_STM_Report_2012.pdf





.

*Good practice and quality in peer review is

system and access- and business-model

independent*


What do people think of peer review?

Editors … value it

The research community … does too Peer review survey: Ware and Monkman, 2007 data; Sense

About Science, 2009 data

85% & 82% - peer review greatly helps scientific communication

83% & 84% - without peer review would be no control in scientific

communication

accuracy and quality of work not peer reviewed cannot be trusted

89% & 91% felt own last accepted paper improved by peer review

Open access survey: Taylor & Francis Group 2012-13 data

authors rated ‘rigorous peer review’ most important service expected

when pay to publish OA (above rapid peer review & publication)


“Peer review in scholarly publishing, in one form

or another, is crucial to the reputation and

reliability of scientific research” (Para 277)


… but is dissatisfaction

12% (Ware & Monkman) and 9% (SAS) in the

two surveys

Only about a third in both surveys think current

system of peer review is best that can be

achieved

Researchers want to improve peer review, not

replace it


Criticisms of peer review


“Peer review is in

crisis”

“Publish all, filter

later”

Unreliable and unfair

No clear standards, idiosyncratic

Open to abuse and bias

Stifles innovation

Slow, causes delays in publication

Expensive and labour intensive

Reviewers overloaded, working ‘for free’

Almost useless at detecting fraud and misconduct

Can ‘fail’ in even the best-run systems [Image, Gideon Burton, Utah, USA (CC BY-SA 2.0)]

Critical role of the Editor

“…[peer review] works as well as can be expected. The critical feature

that makes the system work is the skill and insight of the editor. Astute

editors can use the system well, the less able who follow reviewer

comments uncritically bring the system into disrepute.”

(a respondent, Ware & Monkman, 2008, PRC peer review survey)

“Unfortunately, all too often editors relinquish their responsibilities and

treat the peer review process as a vote … the real problem is editors …

increasingly, one sees editors who don’t use any judgement at all, but

just keep going back to reviewers until there is agreement.”

(Dorothy Bishop, Professor of Developmental Neuropsychology, Oxford University, ‘In

defence of peer review’, comment 4 Jan 2011, to R Smith (2010) Breast Cancer

Research, 12(Suppl 4):S13 )


Editors have to act as editors

Being an editor is:

not just moving manuscripts automatically

through the peer-review process

not just ‘counting votes’

not passing on editor responsibilities to

reviewers

making critical judgements (‘reviewers advise,

editors decide’)


Some problems due to

Variable quality of peer review

Inconsistency in decision making

Lack of training for new editors

Perceived gaming by journals

Unethical behaviour


‘Fake reviewer’ cases


Ethics and integrity

Important in research communication/publication

whatever the model

Lack of knowledge and training

COPE

Guidelines and resources


.

‘COPE’s new Ethical Guidelines for Peer Reviewers: background, issues, and evolution’,

ISMTE, EON May 2013, Vol6, issue4, http://www.ismte.org/Shared_Articles-

COPEs_new_Ethical_Guidelines_for_Peer_Reviewers_background_issues_and_evolution/


.

Innovations in peer review


Two functions of peer review separated

Publication based on ‘soundness’ - research methodology,

results and reporting - not novelty, interest or potential impact

Evaluation of interest/impact left for post-publication


Launched December 2006, now a ‘mega’ journal

Published 23,468 articles in 2012 (14,000 in 2011)

(~1.4% of world’s scholarly literature)

Used >60,000 reviewers in 2012 (38,400 in 2011) from

154 countries

Impact Factor ~4

‘PLOS ONE clones’ being launched (BMJ Open, Sage

Open, Scientific Reports, Biology Open, AIP Advances,

SpringerPLus)

18 Irene Hames, EASE/ISMTE, 23 Sept 2013

Eliminating ‘wastage’ of reviews

Rejected manuscripts can go from journal to

journal, fresh reviews at each

‘Cascading’ submissions and reviews

Within publishers and societies

Between publishers

• Neuroscience Peer Review Consortium

(http://nprc.incf.org/)

• eLife, BMC, PLOS and EMBO – ‘portable peer review’


http://nprc.incf.org/

More transparent approaches

Publishing reviews, ms versions and editorial

correspondence, reviewers’ names may or may not be

revealed

BMC series medical journals – ‘pre-publication history’

The EMBO journals – ‘peer review process file’

BMJ Open – ‘peer review history’

eLife – decision letter + author response (have doi’s)

Reviewer interaction: pre-decision at The EMBO Journal

(‘cross-peer review’) and eLife

Reviewer + author + editor interaction: Frontiers

‘Open’ peer review


In 1996 …


TPJ anonymous reviewer

“That you would consider making a reviewer’s identity

known to the submitting authors is alarming in the extreme.

How can such a practice not but undermine the peer review

process and lower standards? ... I would hold the journal,

not the reviewers, responsible for the evaporation of your

journal’s credibility, and for an inexcusable erosion of the

philosophical framework of modern plant biology.”


‘Peer review process goes meta via blogs and Twitter’

‘Ultra-open’ peer review at GigaScience July 2013

Reviewer blogged about a manuscript under peer-review

Shock, followed by, “This is fantastic!!”

“As we are promoting and encouraging more transparent

review and the use of pre-print servers, if the authors

and reviewers consent then we do allow open discussion

of the work prior to publication.”

http://blogs.biomedcentral.com/bmcblog/2013/07/23/ultra-open-peer-

review/


http://blogs.biomedcentral.com/bmcblog/2013/07/23/ultra-open-peer-review/







New initiatives/models


.

… and even newer


.

Peer review doesn’t stop at publication

When real peer review starts?

Post-publication review and evaluation

Increasing opportunities for innovation

Challenges and problems

Increasing importance of blogs, twitter and

other social media (#arseniclife)


.

“Online scientific interaction outside the traditional

journal space is becoming more and more

important to academic communication”

Mark Hahnel, founder, FigShare (http://figshare.com/)


http://figshare.com/

http://figshare.com/

A big challenge - data

Massive amounts being generated

Recognition for producing, making usable by

others and curating

Where to put?

Dryad http://datadryad.org/ - international

repository of data underlying peer-reviewed

articles in basic and applied sciences; can be

made securely available for peer review


http://datadryad.org/


A reviewer’s point of view

‘Reviewers peering from under a pile of ‘omics’ data’ J.K.

Nicholson, Nature (2006), 440, 992

“The scientific community needs to reassess the

way it addresses the peer-review problem,

taking into account that referees are only human

and are now being asked to do a superhuman

task on a near-daily basis.”

Will peer review survive?

Mark Ware: ‘far from being in crisis, peer review remains

widely supported and diversely innovative’ Ware M (2011) New Review of Information Networking, 16(1): 23-53

Fiona Godlee: (BMJ Editor): ‘At its best I think we would

all agree that it does improve the quality of scientific

reporting and that it can improve, through the pressure of

the journal, the quality of the science itself and how it is

performed.’ Godlee F (2011) Evidence given to the UK House of Commons Science and

Technology Inquiry into Peer Review, 11 May 2001. Transcript of oral evidence,

HC856, Q97.

http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf


http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf

In 2013 …


2007: Peer review … at its best a very powerful and sophisticated tool

… since so much hinges on it, it is essential that it is carried out well

and professionally, and that it is viewed with confidence and respect.

But …

… what will it look like?

… who will be running it?

… how will it cope with the increasing volume

and range of research output?



Thank you!

Dr Irene Hames

email: [email protected]

twitter: @irenehames

mailto:[email protected]

Documents

The Future of Peer Review - European Association of ...€¦ · Mark Ware: ‘far from being in crisis, peer review remains widely supported and diversely innovative’ Ware M (2011)