The Hidden Costs of Journal Peer Review

PEER REVIEW IN THE POLIT IC IZED ACADEMY

The Hidden Costs of Journal Peer Review

Robert Weissberg

Published online: 18 April 2013# Springer Science+Business Media New York 2013

Over the last thirty-five years as a political scientist I have had lots of

experience with peer review, as an author submitting articles to be reviewed and

as a referee. In principle, at least in political science, and I suspect the social

sciences more generally, it is a decent quality control system. Nevertheless, it

can also facilitate ideologically driven control over scholarship, no small matter

as real-world politics increasingly invades the academy and publishing in peer-

reviewed journals becomes the indispensable ticket to tenure.

First some background regarding how peer review works, at least in the

social sciences. You write a paper, circulate a draft among colleagues, and

then send it off to a peer-reviewed journal and hope for the best. The editor

recruits referees, assigns the paper a number to shield authorship, and after

perhaps a few months he must sort out the assessments. If referees initially

disagree or express qualms, the editor might request “revise and resubmit”

and rely on a third referee to resolve disagreements. Especially in the top

journals, outcomes are mainly negative, so rejected authors typically then

submit the paper to a less prestigious journal (this hierarchy is well known in

social science disciplines though it can be complicated for highly specialized

journals). This quest for publication may continue for years with multiple

submissions, particularly since only successes count regardless of the number of

rejections.

Acad. Quest. (2013) 26:157–165DOI 10.1007/s12129-013-9353-8

Robert Weissberg is professor emeritus of political science at the University of Illinois at Urbana–Champaign, Champaign, IL 61820; [email protected]. He is the author of many books, most recentlyThe Limits of Civic Activism: Cautionary Tales on the Use of Politics (2004), Pernicious Tolerance: HowTeaching to “Accept Differences” Undermines Civil Society (2008), and Bad Students, Not Bad Schools(2010), all published by Transaction.

Two features of this process warrant special mention. First, referee judgment

is anonymous and this secrecy opens the door to irresponsibility. A referee

wanting to decline a submission can misrepresent the author’s views, lie to rebut

rock-solid research, or otherwise behave capriciously with zero personal cost.

Petty jealousies may even surface when the referee can identify the author.

Writers are generally defenseless against such attacks, no matter how

outrageous or patently false. Editors and their staff generally lack the time and

energy to meddle in these spats, given the crush of other articles awaiting

consideration and their own lack of specialized expertise. Most editors look for

ways to say no, and truth be told, most articles do not deserve publication.

Potential irresponsibility also applies to the journal’s editing. No fixed rules exist

regarding who gets to review a submission and editors know from experience that

certain referees are “killers” who pick everything apart as deficient, while others

are far more generous. An editor can “honestly” therefore kill a piece that he

dislikes for whatever reason, including ideological aversion, simply by forwarding

it to an excessively demanding reviewer. A death sentence can also be given by

facilitating an ideological mismatch: sending a submission that reaches a

“conservative” conclusion to a referee famous for strident liberalism.

In other words, everything about publishing in refereed journals can, at

least potentially, resemble sausage-making—nobody can tell what went into

the process prior to the paper appearing in print.

The upshot of potential irresponsibly is for those submitting articles for

review to play it very, very safe. The odds are against you in any case (top

journals reject nearly everything), but an author should not create even more

obstacles by angering anonymous reviewers or the editor. A wise and

commonplace strategy would be to stay safely within the confines of accepted

wisdom and use ample amounts of professional jargon to indicate familiarity

with prevailing professional norms. Better to publish innocuous second-rate

stuff in a first-rate peer-reviewed journal than to try to hit a scholarly home run

and suffer rejection.

One can also up the odds of acceptance by heaping lavish praise on

potential reviewers (however undeserved) while copiously citing friends who

can be counted on to say “publish” if the submission comes their way. (These

friends have probably read drafts, so the author will be known to them.)

Most important, in all instances avoid trouble by steering clear of anything

even slightly “controversial.” Don’t kid yourself about speaking truth to

power. If you are writing about urban politics and the prevailing orthodoxy insists

158 Weissberg

The upshot of avoiding trouble is that whole areas of scholarship become

dead zones where only the politically ultra-orthodox can survive. This is

most obvious in anything having to do with “the groups”—African American

studies, women’s studies, Hispanic studies, queer studies, or any other

subfield populated by clearly identifiable ideologues.

Even the choice of explanatory factors can lead to papers being DOA

regardless of subject matter. Good luck to scholars who employ cognitive

ability as measured by IQ when examining economic development or civil

violence. How about submitting a paper that meticulously demonstrates that

lack of impulse control explains poverty? Actually, when all these off-limits

fields and subfields are added up, little remains open to anybody willing to

challenge the bien-pensant crowd.

So what’s a scholar to do who refuses to drink the ideological Kool

Aid but wants to publish in refereed journals? Stick to fields whose

obscurity and technical complexities virtually preclude ideological

intrusion. Prominent examples include rational choice, mathematical

modeling, and statistical methodology. A savvy publication-minded

scholar is also advised to dress up the paper with obtuse mathematics

to intimidate readers with limited knowledge of mathematics. After all,

nobody can antagonize the thought police for applying stochastic

calculus models to a series of wars involving Country A versus

Country B while Country C sat on the sidelines deciding whether to enter the

fray. All perfect insurance against rejection for being politically incorrect, a

throwback to the era when the “wrong” scholarship risked death so sages wrote

in convoluted Latin.

The same “keep it obscure and irrelevant” advice even applies to the

innumerate—become the world’s leading authority on an obscure

fourth-century French cleric and his previously unrecognized contribution to

the theory of republican government (and the clincher might be that this scholar

was a woman living as a man in a monastery). Again, it’s hard to get into trouble

if nobody reads your musings. And who will deny that this is “serious

scholarship”?

The Hidden Costs of Journal Peer Review 159

that black electoral control has been a boon for African Americans in Detroit,

don’t disagree. You can never win an argument with an anonymous reviewer.

Journal submissions are not evidence-based debates or marketplaces of ideas—

once the essay is in editorial hands, you have no voice in the outcome, no matter

how biased or wrong-headed.

This avoidance strategy also offers the advantage of reducing academic

competition, thanks to the technical knowledge barriers to entry. While there

may be thousands toiling in American or European politics generally, those

viewing the world from the perspective of rational choice (among several

other arcane subfields) may not exceed a hundred, and even better, they

may all know one another and I’d guess profitably engage in mutual

back-scratching.

All and all, this is bad news for scholarship. When viewed from outer

space the landscape is bleak and it is no wonder that fewer and fewer

academics writing for peer-reviewed journals become “public

intellectuals.” Actually, after years of writing in jargon, some

academics may have lost the ability to communicate in plain English.

How this affects recruitment of graduate students into the profession is a

topic of great concern, but I’ll leave that subject for another day.

An Untimely Death by Peer Review: A Personal Account

To illustrate the power of peer review to create academic dead zones, let

me offer a personal account that involved the hot-button topic of race. Back in

the late 1980s I presented a paper at the Midwest Political Science Association

offering an overview of current research on American racial politics. It

highlighted various research lacunae that, at least in my estimation,

warranted closer scrutiny.

The panel filled the room with many listening just outside the doorway and

others sitting on the floor. This, I should add, was highly unusual at the Midwest

convention, where most panels attract only a smattering of observers. Reaction to

all the papers including mine was exceptionally lively (but very civil) and might

have continued for hours afterward. Nobody criticized anything as hateful, racist,

or inappropriate for a disciplinary meeting. I reasonably concluded that a large

audience existed for a frank discussion on this “taboo” topic.

I then submitted a revised version of my paper to the American Journal of

Political Science, a prestigious journal in my field. After a few months an

editor’s rejection arrived accompanied by two reviews. One unequivocally

advised acceptance but added, “I disagree with some of the author’s points,

but the topic is a critical one, long under-examined and this paper might

instigate important, much needed research.” The second reviewer was

unambiguously negative.

160 Weissberg

The “reject” review failed to address my key points regarding how current

research on race was incomplete. Instead, it sidestepped everything with a

rambling, prolix literature review of sundry well-researched race-related

subjects, many about court cases, all of which had been around for decades. In

effect, this reviewer was saying “today’s research on race is fine, so no need to

venture elsewhere.” Evidently, the one negative review trumped the strong

positive one from the other reviewer.With this verdict, there was no need to get a

third opinion to settle the one “Yes,” one “No” outcome.

Then something highly unusual happened—the one positive reviewer called

me up. I had never met him but knew his work well and it was first-rate. He was

a self-defined leftist, a supporter of the civil rights agenda, but also, as he told

me, a strong believer in professionalism and let the chips fall where they may.

We chatted for over an hour and he told me that he had previously dabbled in

racial politics but then abandoned it. We both agreed that when it came to doing

research on race, it was futile to struggle to secure research funding, get

published, or otherwise move up professionally with so many other

noncontroversial research topics available.

What is also unusual about this tale is how far I got into the system before

quitting. Other academics but especially those early in their careers are unlikely

even to contemplate writing about race unless they desire a career in the African

American academic establishment. They know full well that the topic is a

minefield where the slightest deviation from the orthodoxy can end an academic

career, even for African Americans. Awareness of this reality probably begins in

graduate school and is likely hammered home by dissertation advisors who

quietly tell advisees about the scholarly version of racial apartheid.

Curing the Ideological Mischief of Peer Review

How can we maintain high scholarly standards while minimizing the

pernicious impact of ideology? Let me suggest four solutions—all feasible,

none excessively expensive—that can be implemented relatively quickly. The

first two involve making the process more transparent and therefore more

responsible. The third suggests creating an “alternative universe,” while the

fourth would use the Internet to eliminate any and all ideologically-imposed

choke points.

As noted, the current system’s anonymity permits bias—the ideologically

driven reviewer can irresponsibly say “publish” no matter how shoddy the


research, and this low standard further encourages yet more journal submissions

with the “right” bias that, ultimately, creates a collection of one-sided research

called “the scholarly consensus.” Some of this ideologically motivated exchange

of favors can be eliminated by listing those who consented to publication. With

this public acknowledgment in place, the strident ideologue might think twice

about green-lighting a thinly disguised, sloppily executed polemic. Indeed, for

time-pressed readers a quick glance at the referee names might be a shortcut to

avoid wasting time, while the editor might also think twice about being exposed

for cooking the books. There is also the benefit of curtailing cronyism, no small

matter when the intellectual value of a publication is weighed. A published article

exclusively approved by the author’s friends will surely be discounted, no

matter the quality.

A second route to transparency is to alter the meaning of “peer-reviewed.”

Just replace the anonymous referee system with a publicly identified board of

review (this already occurs at some journals and is considered legitimate). In

some instances, the review board may run in the dozens with multiple

specialties. In any case, authors submitting manuscripts could anticipate who

will judge their work and it would not take much to sniff out potential killer

bias. The other side of the coin would be that the publicly identified editorial

board members would have to take full responsibility for approving any

ideologically flavored rubbish.

A third possibility might be called “creating an alternative universe by

avoiding current, corrupt peer review.” Just establish new journals—even

electronic ones explicitly welcoming to those who might challenge prevailing

consensus—journals for the ultra-unorthodox, so to speak. (Highly regarded

but now defunct, The Public Interest exemplified this approach with its

penchant for articles probably not publishable by more “academic” journals.)

A variant might be an annual book series with titles like “Controversies in

Political Research.” Such publications would encourage multiple viewpoints,

not just tit-for-tat counter-balancing the other side’s partiality. This solution rests

on my observation that many academics, even liberals and perhaps many radicals

are reasonably open-minded when judging journal submissions (recall my personal

experience with such a leftist referee).

There is, however, a major problem with promoting openness to

“controversial” ideas. It is one thing to get “controversial” ideas in print,

but quite another to get these publications accepted as legitimate currency in the

academic marketplace. And without this certification, these “controversial”

162 Weissberg

contributions will be ignored, even judged as liabilities in departments dominated

by narrow-minded ideologues. The parallel is how many departments disregard

anything, nomatter how profound, that appears in journalistic outlets. Suchwritings

are disregarded as not being “serious scholarship.” In short, getting ideas out only

counts if the vehicle is certified as “academic” and this certification can be

politically motivated.

Lastly, we can eliminate all gatekeepers regardless of bias simply by

“publishing” everything on the Internet. This is far less radical than it may

initially appear and, to play the environmentalist card, it would save millions of

trees (and gallons of ink) while reducing carbon emissions. Meanwhile,

financially strapped academic departments would no longer have to subsidize

editorial costs (including expensive copyediting), but the big winners would be

libraries now freed from warehousing expensive (and seldom read) journals. A

further dividend might be the reduction of professional dues. And when I say

publish “everything,” I mean everything no matter how polemical or poorly

executed.

Here’s how. Anybody who wants to publish a paper would simply email an

electronic version to the headquarters of his profession, where it would be

immediately posted on the associationwebsite. Authorswould provide keywords to

sort out this avalanche and subscribers would automatically be sent daily

downloadable unscreened listings in their specialties. “Papers” would include the

author’s email address and a link to where readers can post comments. At last, a

wide open marketplace of ideas.

Actually, much of this exists in the form of informal networks where authors

share prepublication drafts for comments. Obviously, the principle benefit is to

eliminate ideologicallymotivated gatekeepers, but there are intellectual rewards.

First, since everything is now “published,” evaluations for tenure, salary

increases, and promotion must be based on actual substance, not the shortcut “it

must be decent since it was published in the Journal of Modern Obscure

Nonsense.”

A second, non-obvious intellectual benefit might be called “scholarly birth

control.” Since everything, no matter how dreadful, is now “published,”

those who unhappily waste years trying (usually unsuccessfully) to turn dross

into gold can better spend their time elsewhere, perhaps improving their

teaching or correcting student essays. Experienced professors know this

phenomenon all too well: the colleague who has been struggling with a

paper, constantly revising it and sending it out for consideration. Eventually,


it may see the light of day, though truth be told, few will read it and it will die

an obscure death. That understood, I’d predict that the number of papers

intended for journals might decline once authors recognize that getting

published has zero benefits, since everyone is published. Now only those

who really have something to contribute would contribute.

Conclusion

Illustrations aside, it should be clear that the potential for ideological control

of scholarship via peer review can apply across the political spectrum. Liberals

and radicals are today’s guilty party, but tomorrow it could be conservatives and

I suspect that in some obscure corners of the academy, conservatives dominate.

This analysis is about opening up the dissemination of knowledge, not some

scheme to give voice to “conservative” views currently excluded by peer review.

Now for the bottom line. What are the odds of any of these suggestions to

promote openness succeeding? Each recommendation is reasonable, not too

costly, and all, in one form or another, already exist.

The future looks bleak. With few exceptions today’s academics are not

interested in a wide open marketplace of ideas without gatekeepers in the social

sciences. After all, liberals and radicals—especially those pushing “group”

agendas—have spent decades securing control over the academy, including

many of its journals, and are hardly anxious to surrender their prize in the name

of some lofty appeal to intellectual openness.

Many of these academics will also insist that one-sided “scholarship” is vital to

achieving social justice. Moreover, permitting ideological dissenters a soapbox,

they argue, only gives voice to “hate,” homophobia, racism, and similar bad

thinking. The sad reality is that as the gap between their ideological fantasy that

dominates today’s social sciences and the real world widens, the need to quash

contrary views grows more pressing.

Even non-ideologues have a vested interest in keeping peer review somewhat

corrupt. More than a few professors, particularly in small, arcane fields, have

learned to manipulate the system. What could be more career-boosting than a

long vita of peer-reviewed articles in a reject-nearly-everything prestige journal

that barely anybody can comprehend? Why abandon a wonderful arrangement

that has taken years of mutual favors to achieve?

Despite all this, let me suggest one sliver of hope. This generation of

indecipherable social science “knowledge” facilitated by peer review’s incentives

164 Weissberg

may ultimately collapse other than as a currency to distribute professional rewards.

Recall the decline of turgid Latin-basedmedieval scholasticism—millions of words

now almost totally gone. At some point no one will actually read anything for

content; articles (or, more likely, their abstracts) will instead be examined only to

keep abreast with the literature or to populate one’s own research papers with

citations.

The result, hopefully, will be a shift in attention toward non-refereed

publications as a source of scholarly insights, perhaps publications sponsored by

think tanks, foundations, even “intellectual” magazines like Commentary. In

other words, with so little substance in peer-reviewed outlets, those seeking real

knowledge will turn to well-researched content independent of the

certification process. An essay will be deemed “good” for the simple reason

that it is good—and the way it managed to find its way into print will, happily,

become irrelevant.


Documents

The Hidden Costs of Journal Peer Review