A referee's plea reviewed

Based on Mark Al lman’s A Referee 's Plea presentat ion - May 21 , 2001

A REFEREE'S PLEA

REVIEWED

Stenio Fernandes

CIn/UFPE

Nov 2012

FUNDAMENTALS OF THE

PEER REVIEW PROCESS

• The importance of understanding the peer

review (PR) process

In general

Avoid silly mistakes

Adjust the level of expectancy of your submission

To not be upset with some reviews

Targeted submissions

Know how your paper will be treated

UNDERSTANDING THE PEER REVIEW

PROCESS

Peer review is a very old process

Ishaq al-Rahwi (854-931) of Syria first described the PR process

The Philosophical Transactions of the Royal Society is the first journal

to formalize the peer review process (300 years ago)

Validation by peers and publication in a scientific journal

the method through which authors register, validate, disseminate, and

archive their discoveries and results.

process and the speed at which articles are peer reviewed and

published are key elements in the appropriate accreditation of

scientific findings.

Recent Survey

85% of scientists agree that PR greatly helps scientific

communication

93% of them believe peer review is necessary.

Single Blind Review

The names of the reviewers are hidden from the author.

Traditional method of reviewing (the most common type)

Advantage

Reviewer anonymity allows for impartial decisions free from influence by the

authors

Disadvantages:

Authors fear the risk that reviewers working in the same field may

withhold submission of the review in order to delay publication,

thereby giving the reviewer the opportunity to publish first

Reviewers may use their anonymity as justification for being

unnecessarily critical or harsh when commenting on the author’s

work.

Prejudice. Yes, it happens all the time!

TYPES OF PEER REVIEW (1/3)


Double Blind Review

Both the reviewer and the author remain anonymous

Advantages:

Author anonymity prevents any reviewer bias based on, for

example, an author’s country of origin or previous controversial

work.

Articles written by ‘prestigious’ or renowned authors are

considered on the basis of the content of their papers, rather than

on the author’s reputation .

Disadvantage:

It is uncertain whether a paper can ever truly be ‘blind’ –

especially in specialty ‘niche’ areas. Reviewers can often identify

the author through the paper’s style, subject matter or self -

citation.



Open Review

Reviewer and author are known to each other

Advantage

best way to prevent malicious comments, stop plagiarism,

prevent reviewers from drawing upon their own ‘agenda’ and

encourage open, honest reviewing.

Disadvantage

the opposite view:

a less honest process in which politeness or fear of retribution may

cause a reviewer to withhold or tone down criticism.

Junior reviewers may hesitate to criticize more esteemed authors for

fear of damaging their prospects



Again: Understand the Purpose of Peer Review

Peer review is a critical element of scholarly publication, and one of the

major cornerstones of the scientific process.

Peer Review serves two key functions

Acts as a filter: Ensures research is properly verified before being published

Improves the quality of the research: rigorous review by other experts helps to

improve key points and correct inadvertent errors

On Being Asked To Review

Does the article you are being asked to review truly match your

expertise?

Only accept an invitation if you are competent to review the article

Do you have time to review the paper?

an article will take about 5 hours to review properly

Are there any potential conflicts of interest?

A conflict of interest will not necessarily eliminate you from reviewing an

article

REVIEWER GUIDELINES – FIRST STEPS

THE VIEW FROM THE

REVIEWER

Referee

2007 – 2007 - IEEE Transactions on Multimedia

2007 – 2007 - Journal of Internet Engineering

2009 – Atual - Computer Communications

2009 – Atual - Peer-to-Peer Networking and

Applications Journal

2009 – Atual - Journal of Network and Systems

Management

2009 – Atual - Journal of Network and Computer

Applications

2008 – Atual - Ad Hoc Networks

2010 – Atual - Journal of The Brazilian Computer

Society

2011 – Atual - Wireless Communications and

Mobile Computing

2012 – Atual - Wireless Networks

2009 – Atual - Computer Networks

2012 – Atual - Journal of Communications and

Networks

Editor ia l Board

2012 – Atual - International Journal on Advances

in Intelligent Systems

2012 - Atual - Journal of Mobile and Ubiquitous

Computing Technology

Referee

IEEE ICC, GLOBECOM,

INFOCOM, WCNC, GHTCE,

ICNS, CCNC, HPSR, BCN

TPC Member

IEEE WCNC, CCNC,

INFOCOM TMA WORKSHOP

CHAIR

IEEE TRICANS, ACM HPN

CONNEPI 2010 (interesting

stuff, long history)

GENERAL GUIDELINES FROM

EDITORS AND TPC CHAIRS

the article should not be disclosed to a third party

you would be expected to evaluate the article according to the following

Originality

Is the article sufficiently novel and interesting to warrant publication ?

Is the research question an important one?

Structure

Are all the key elements present: abstract, introduction, methodology, results, related work, discussion (lessons learned), conclusions?

Title: Does it clearly describe the article?

Abstract: Does it reflect the content of the article?

Introduction: Does it describe what the author hoped to achieve accurately, and clearly state the problem being investigated?

Method: Is there sufficient information present for you to replicate the research?

Results: has appropriate analysis been conducted? Are the statistics correct?

Conclusion/Discussion: Are the claims in this section supported by the results, do they seem reasonable?

CONDUCTING THE REVIEW

Commentary should be courteous and constructive

You should indicate whether your comments are your own

opinion or are reflected by the data

If the review is not blinded (yes, it’s true!)

Do the authors have a "track record" of working in this area?

Are they from a reputable institution?

Think about the consequences of this? (my question)

Subjectively:

do you believe what the authors are telling you?

do you suspect some consistent error in the hypothesis, methods,

analysis of data, etc.?

Is there some chance that there is scientific fraud or plagiarism

involved in this manuscript?


Finally, on balance, when considering the whole article, do the

figures and tables inform the reader, are they an important

part of the story?

Previous Research:

If the article builds upon previous research does it reference that

work appropriately?

Are there any important works that have been omitted?

Are the references accurate?


RELEVANCE

Is the topic of the paper relevant to the journal?

Is the paper aimed at the readership?

Is the technical depth of the paper relevant to the journal?

CONTENT

Is the paper needed (does it fill a gap)?

Are the references up to date?

Is the content (theories, concepts, methodologies) sound?

Does the originality of the material warrant publication?

PRESENTATION

Is the paper well organised?

Do any parts require further explanation of clarification?

REVIEWER CONFIDENTIAL COMMENTS TO

EDITOR - ELSEVIER

Section I. Overview

A. Reader Interest

1. How relevant is this manuscript to the readers of this journal ?

B. Content

Is the manuscript technically sound?

What do you see as this manuscript's contribution to the literature in

this field?

What do you see as the strongest aspect of this manuscript?

What do you see as the weakest aspect of this manuscript?

C. Presentation

Are title, abstract, and/or keywords satisfactory?


EDITOR - SPRINGER

Please identify and discuss the contribution of this manuscript.

Please include in your discussion:

Does the paper have significant tutorial content?

Is there enough background provided so that the generalist in

communications can understand its main contributions? Elaborate .

Does the paper contain original contributions? What is the nature

of the contributions?

Is there a description of lessons learned that are given to the

reader to help the reader avoid pitfalls in his own work?

Is there a need for a paper such as this in the communications

community?

For example, are there articles that are already available which cover

more or less the same topic at about the same depth?

Please discuss the quality of the citations in this manuscript


EDITOR - IEEE

Please comment on the organization of the paper, and offer any

suggestions that you think will improve the paper and its

readability

Please comment on the technical correctness of the manuscript

in general

identify any specific technical inaccuracies that you find, and make

suggestions for correcting those

If the manuscript does not require major revision, please provide

a list of minor changes

Please provide a summary comment on the overall suitability of

the paper for publication in IEEE *


EDITOR - IEEE

Mark Allman

ACM SIGCOMM Referee

2001

A VIEW FROM A TOP

CONFERENCE REVIEWER

I now realize that the community generates a large amount of

“junk”

I don't mean bad ideas

Each paper I reviewed for SIGCOMM had at least a nugget of an interesting

idea.

Definition of Junk: "it took me twice as long to read this as it should

have"

In other words, the paper was sloppy

I had a very difficult time trying to figure out what the paper was

proposing

What the experiments were really showing.

The following is a short list of *easy* ways to improve your

research papers

Most comments are general

Last comment is directed at the internetworking research community

0 - LARGE AMOUNT OF “JUNK”

It seems to me that everyone has access to a good spell

checker these days.

Not using the spell checker just indicates that either you are

sloppy or you do not care.

If you're sloppy with your writing why should I believe you are

not sloppy with your experiments/analysis as well ?

If you don't care about your paper, why should I ?

1 - FIRE UP THE SPELL CHECKER,

PLEASE!

Proofread your paper

I see lots of mistakes that indicate the paper was not looked over

before it was submitted (e.g., "The foo algorithm works better

than the the bar algorithm.").

Again, if you do not take enough care in presenting your work,

why should I be compelled to recommend accepting it?

Sloppy papers give the indication of sloppy research

As well as reflecting poorly on your work, such papers are often very

difficult to read and understand

The point of writing a paper is to convey new information to the

world

Therefore, there is major value in clear writing.

If a referee can't get the point of the paper, or has a very difficult time

getting the main idea, there is a high likelihood that the paper will be

rejected

2 - SLOPPY PAPERS ARE LONG SHOTS

I print out all papers I review. I read them in my lazy boy. I

scribble notes on them.

Do not assume that I am looking at color plots!

Color plots come out of my black and white printer very badly in most

cases.

Besides, most conferences/journals publish black and white papers

Make it easy to tell the difference between the lines on your plots.

Do not plot on a grey background.

Make the font on the plots readable without a magnifying glass

Help me out! Please! Often the plots are key to understanding the

paper

If I cannot understand the paper I will recommend rejecting it

3 - I READ SUBMISSIONS IN MY LAZYBOY

You do not need to give an in-depth tutorial of all background material.

How much background material to include?

It depends on the conference to which you are submitting, the maturity of the background work, etc.

There are some topics (e.g., TCP congestion control) that are well understood by people attending any networking conference.

You do not need to describe the algorithms in painful detail.

A short paragraph re-capping the main ideas and providing references to the original works should suffice.

4 - DON'T WASTE MY TIME (I)

You do not need to provide me with an example of every

kind of plotting strategy you use

When you first show a plot with notation that I might

not understand, explain the notation

But, using half a page to show me a plot that adds nothing to

my understanding of the topic is a less than compelling use

of real estate

5 - DON'T WASTE MY TIME (II)

Roughly stick to the page limit and the requested format.

Don't make me try to guess how long your paper really is

because you 1.75-spaced the paper rather than double -

spacing it as requested in the call for papers.

Editors usually check this

Authors have been relying on technical reports to “extend” the paper

6 - DON'T WASTE MY TIME III

If you need a constant for the foo algorithm and you define

c=17, I want to know why you chose 17.

Also, some items in papers jump out at reviewers as odd.

For instance, I was recently reviewing a paper that simulated

a network with a bottleneck bandwidth of "about T1 rate". The

actual rate of the link turned out to be something like 1.3

Mbps.

why the authors used 1.3 Mbps rather than using T1 rate (~1.5

Mbps)?

Even if the explanation you end up with is rough, it is better than no

explanation at all in the vast majority of the cases.

7 - ANSWER ALL THE EASY QUESTIONS

8. Make My Life Easy

It is not a super-big deal, but

it is always nice to get a paper that is easy to read

This includes things like using a very clear font that is of readable

size and numbering the pages

8 - MAKE MY LIFE EASY

Your paper does not solve the entire world's networking

problems. Really.

Many papers rub reviewers the wrong way by making bold

claims about solving all sorts of problems without any sort of

evidence (or only weak evidence)

It is much better to show some restraint and perspective when

discussing your results.

It is alright to solve small problems, or to only show

preliminary results that *may indicate* a solution.

Just don't over claim what you have shown.

9 - A LITTLE RESTRAINT

10. Use Units

"The length of our data transfers is 1" means nothing. 1 what? 1

packet? 1 KB? 1 minute?

Sometimes it is obvious from the context -- sometimes it isn't. If you

use units you don't have to worry about it.

10 - USE UNITS

Theorems are quite a necessary part of many papers.

However, it is not necessary to give *only* a theorem with no

summary or context.

Summaries aid me as a reviewer to gain an intuitive

understanding before trying to dig into the math to make sure

what you have it correct

thus making my task easier and less error prone, I believe.

As a reader of a paper, I often just want the high -order bit

without slogging into the theorems and proofs to get said bit.

11 - WE'RE ALL NOT MATHEMATICIANS

One common mistake that authors make is failing to include

relevant references.

It is better to err on the side of citing too much related work

Next, when previous work must be criticized, do so in a

constructive manner.

In other words, do not say something like "Zevon's [1] work is easily

shown to be less effective than the work presented in this paper " --

Mr. Zevon might be reviewing your paper!

Something more along the lines of: "Zevon [1] first introduced the foo

algorithm … We show a more robust algorithm”

Finally, do not overstate what a paper you are citing says

It is often better to stick with the author's conclusions

12 - ON REFERENCES

S. Keshav

ACM CCR Editor

July 2011

WISE WORDS FROM A

WISEMAN

THE ACM SIGCOMM’S CCR - JULY 2011

13 technical submissions

All of them were rejected by the Area Editors on the advice of the reviewers

One could ask: were all the papers so terrible ?

Some submissions were too broad, others too narrow,

Some too incremental, some too radical,

Some were just not interesting enough

I feel that this attitude has permeated our community at large

A similar spirit of harsh criticism is used to judge papers at SIGCOMM, MOBICOM, CoNEXT, and probably every other top-tier computer science conference

Reviewers seem only to want to find fault with papers

rather than appreciate insights

Despite

inevitable errors

lack of technical completeness

THE ACM SIGCOMM’S CCR - JULY 2011

1. Subconscious desire to get one’s back

1. if my paper has been rejected from a venue due to sharp criticism…

2. Why not pay this back with sharp criticism of my own?

2. A desire to prove one’s expertise

1. if I can show that a paper is not perfect, that shows how clever I am

3. A biased view of what papers in a particular area should look like

1. I’m the expert in my field so I think I know what every paper in my field should look like

4. unrealistic expectations

1. I may not write perfect papers but I expect to read only perfect ones

5. Subconscious attitudes are exacerbated by

1. a ballooning of reviewer workloads

2. journals in computer science languishing in their roles, conference papers being held to archival standard

One negative consequence is the stifling of innovation

they scale the walls by adding epsilon to delta until the

incrementality threshold is breached

well-studied areas are further overstudied to the detriment of others

A second negative consequence is that it turns some

researchers off

do not want to take part in a game where they cannot respect the

winners or the system

PC chairs and Area Editors have to change our mental attitude

from finding reasons to reject a paper

to finding reasons to accept a paper

We will certainly be trying to do this from now on at CCR

As a reviewer of a paper, it is your duty to critique a paper and

point out its flaws.

But can you overlook minor flaws ?

Can you find the greater good?

With this small change, the system will also change.

One review at a time

Based on

true facts

EXAMPLES: FROM GOOD

TO BAD (AND VICE-

VERSA)

Don't judge peer review by its occasional failings

http://www.guardian.co.uk/science/blog/2008/apr/15/peerreviewersdoamagnificen

STARTING AN INTERNAL

PEER REVIEW PROCESS

• Title: be creative

• Abstract: act as a marketing person

• Introduction: be clear. State scenarios, need (motivation), topic importance, a quick view on previous work/your solution/results, paper organization

• Methodology: make it reproducible (at least, clear)

• Results: be creative on presentation (not much), interpret data, infer things (not much), discuss the lessons learned

• Related work: show both broad and close research papers; state the difference to yours (during and at the end of the section)

• Conclusion: do not add new material. Summarize main findings and make it coherent with title/abstract/intro

“Convince the referee

that it is worthwhile

reading the next section”

(me)

Author makes a

draft

Colleague

makes a revision

Professional

proofreading

LAST WORDS: THE DARK

SIDE OF PR

Open Access

Expla ined

PUBLISHERS

VS. OPEN

ACCESS

http://www.youtube.com/watch?feature=player_embedded&v=L5rVH1KGBCY

http://www.youtube.com/watch?feature=player_embedded&v=L5rVH1KGBCY