E x p e r t O p i n i o n

use it to select and screen manuscript submissions, andfunding agencies use it to award research funds. The goalis to get authors to meet their discipline’s standards andthus achieve scientific objectivity. Publications and awardsthat haven’t undergone peer review are often regarded withsuspicion by scholars and professionals in many fields.However, peer review, although universally used, has manydrawbacks.

We propose replacing peer review with an auction-basedapproach: the better the submitted paper, the more scientificcurrency the author will likely bid to have it published. If thebid correctly reflects the paper’s quality, the author will berewarded in this new scientific currency; otherwise, theauthor will lose this currency. We argue that citations are anappropriate currency for all scientists.1 We believe that cita-tion auctions will encourage scientists to better control theirsubmissions’ quality. It will likely also inspire them to pre-pare more exciting talks for accepted papers and to invitediscussion of their results at congresses and conferences andamong their colleagues. In the long run, citation auctionscould have the power to greatly improve scientific research.

Peer review’s drawbacksWhile some believe that passing peer review is a certi-

ficate of validity, others are far more skeptical. One of themost common complaints is that peer review is slow: a sub-mitted paper typically takes several months, or even yearsin some fields, to appear in print. Such a delay in a fast-growing field is devastating for the propagation of ideas andneeds a solution. Another major drawback is its cost interms of hours of volunteer work devoted to refereeing.

In addition, some sociologists of science argue that peerreview makes publication susceptible to control by elitesand to personal jealousy. Peer review might suppress dis-sent against “mainstream” theories. Reviewers tend to be

especially critical of conclusions that contradict their ownviews but happily accept those that accord with them. Atthe same time, elite scientists are more likely than lessestablished ones to be sought as referees, particularly byhigh-prestige journals or publishers. As a result, someargue, ideas that harmonize with the elite’s views are morelikely to be published and to appear in premier journalsthan iconoclastic or revolutionary ideas. Consequently, thewhole process obstructs and delays the emergence of newideas and scientific revolutions.

However, others have pointed out that scientists havemany journals in which to publish, making control ofinformation difficult. In addition, peer review’s decision-making process, in which each referee gives an opinionseparately without consulting others, is intended to miti-gate some of these problems. Nevertheless, this processstill doesn’t address the cost of peer reviewing and publi-cation delays.

Moreover, peer review tends to accept those weakerpapers that have a mix of prestigious and unknown au-thors. This is because referees typically trust a paper withprestigious authors even if they don’t fully understand itscontributions, believing that “this must be true, must begood, and the final submission will address any potentialdrawbacks.” This might not systematically be bad, butthere are more elegant ways to give opportunities to newauthors. For the sake of improving the quality of science,this trust in important coauthors should be reduced. Blindpeer review, one possible way to deal with this problem, isstill an imperfect solution because it doesn’t decrease thecost of reviewing or publication delays.

Citations and their auctionsAccording to Philip G. Altbach, director of Boston Col-

lege’s Center for International Higher Education, the cita-tion system was invented mainly to understand how scien-tific discoveries and innovations are communicated andhow research functions.2 On the basis of our research onthe innovative use of auctions,3,4 in May 2006 we consid-ered using this approach to create an alternative to peerreview.5,6 Conferences often suffer from low participation

The scientific community has been suffering from

peer review for decades. This process (also called

refereeing) subjects an author’s scientific work or ideas to

the scrutiny of one or more experts in the field. Publishers

Selecting Scientific Papers for Publication via Citation Auctions

Josep Lluis de la Rosa, University of GironaBoleslaw K. Szymanski, Rensselaer Polytechnic Institute

16 1541-1672/07/$25.00 © 2007 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

and little discussion of the presented papers,because authors focus on getting their papersto the conference rather than widely dissem-inating their results. Moreover, organizersextend extraordinary effort selecting the bestconference paper, thus creating a heavy ref-ereeing workload. To avoid these pitfalls, wepropose an auction system combining cita-tion’s predictive value with authors’ intimateknowledge of their papers. (For other alter-natives to peer review, see the sidebar.)

We can characterize the situation as fol-lows. Nowadays, a scientist wants to pub-lish his or her results in conferences andjournals to gain citations and reputation. Asevidenced by the acceptance rates, highlyranked and read conferences and journalsselect only a small number of submittedpapers—hopefully, those that will generatethe most citations. In our approach, confer-ences and journals hold an auction to selectthose papers whose authors have bid thehighest. Each bid represents an author’s pre-diction of how many citations that paper willreceive. To make this bid trustworthy, weconsider the number of citations an authorhas received for previously published papersas that author’s citations wallet—the amountof “cash” the author possesses. So, the num-ber of citations in authors’wallets limits theirbids, and every auction-winning paper with-draws a number of citations from its authors’wallets.

So, what should the wallet contain? Cita-tions from one conference that has accepteda paper by the author? Citations from agroup of related conferences? Citations ofall the author’s papers? We use the citationsfrom either a group of conferences or all theauthor’s papers. This lets an author collectcitations in weaker conferences to get into

stronger ones or into a journal. Conse-quently, rational authors will bid the high-est number of citations they think accep-tance will require, but to the limit of theirwallet’s cash (that is, their past perfor-mance). Authors might lose cash if theirwinning bid exceeds the number of cita-tions the paper will generate. Conversely,they might win more cash if the publishedpaper generates more citations than theybid. An author’s ultimate goal will be tokeep his or her wallet growing.

Our approach has two main benefits. First,it reduces refereeing costs because paperselection via a citation auction doesn’t needprior understanding of the paper’s content toevaluate its quality. Second, authors will bemuch more committed to their papers’qual-ity. They will also focus much more on widedissemination and detailed explanations oftheir papers to maximize the number of cita-tions. In short, this novel approach empha-sizes active promotion of ideas while reduc-ing peer review’s high expense.

We’re fully aware that making this ideaworkable involves challenges. Here we dis-cuss the three most important ones.

First, if each author receives credit for allcitations to the paper, we’ll have wallet in-flation. A paper with five authors, five cita-tions, and a bid of five citations will contri-bute 25 citations total to all the authors’wallets, but the payment will be just fivecitations from one author’s wallet. To avoidthe inflation, each citation will earn oneunit of credit for the paper, regardless ofthe number of its authors. Additionally,we’ll assign fractions of each citation to theindividual authors according to their “cita-tions contract.” This contract might reflectthe percentage of each author’s participa-

tion. If no citations contract exists, all cita-tions will be assigned to the first author,who will then decide how to share them. Toencourage collaborations, we also let allthe paper’s authors contribute their cita-tions to a joint bid. This will solve the in-flation problem. Coauthors are often stu-dents, so they can build their wallets for afuture independent career while workingwith their advisors on their theses.

The second challenge involves self-references. An author could safely bid anumber of citations equal to the numberof that author’s self-citations in the paper.After the paper’s publication, the authorwould automatically receive the credit forthe citations, immediately rebuilding thewallet. To avoid this effect, none of theself-citations or citations of papers by abid’s contributors should be added to theauthor’s wallet.

The third challenge regards the wallet’sinitial content. We assume that the initialcontent is zero citations. So, new authors(for example, graduate students) or au-thors who depleted their wallets via too-aggressive bidding wouldn’t even be ableto try to get their papers published. Toavoid such situations, a certain fraction ofpapers should undergo peer review, butwith much higher acceptance criteria. Thiswill also avoid the problem of the bound-ary-quality papers, which are difficult andtime consuming to evaluate.

Alternatively, authors could receive“sponsorship.” For example, when one suchauthor looks for credit, another researchermight loan some citations from his or herwallet (likely after the paper’s review andafter the lender has suggested improve-ments). For a supervisor or advisor, the

NOVEMBER/DECEMBER 2007 www.computer.org/intelligent 17

In some fields, such as astronomy, much of the communica-tion about new results no longer takes place through peer-reviewed papers but through preprints submitted to electronicservers such as arXiv.org.

The recently launched online journal Philica (www.philica.com) exemplifies one new way to redress many of peer re-view’s problems. Unlike a traditional journal, it immediatelypublishes all submitted papers; review takes place after-wards. Reviews are still anonymous, but instead of an editorchoosing the reviewers, any researcher who wishes to canreview an article. Reviews appear at the end of each paper,thus giving the reader criticism or guidance about the paperinstead of determining whether it’s published. This means

that reviewers can’t suppress ideas with which they disagree.Some authors (for example, Stefano Mizzaro1) suggest scor-

ing papers and authors, letting readers act directly as referees,and receiving feedback on whether the readers providedgood-quality judgments. So, good readers would earn goodreputations.

Reference

1. S. Mizzaro, “Quality Control in Scholarly Publishing: A New Pro-posal,” J. Am. Soc. Information Science and Technology, vol. 54, no.11, 2003, pp. 989–1005.

Some Alternatives to Peer Review

motivation for a loan is clear: lending partof the prestige represented by the citationsin his or her wallet is a kind of investment.Hopefully, the student will eventually re-

turn the citations, perhaps with extra cita-tions as interest, which an agreement be-tween the two could formalize. We don’tbelieve that this approach would benefit

“bad students with famous supervisors”over “good students with unknown super-visors.” That’s because a donor systembased on one scientist’s rational criteria

Table 1. A simulation of four citation auction strategies.*

No. of citations

1999 2000 2001 2002 2003 2004

Author and paper Bid Earned Bid Earned Bid Earned Bid Earned Bid Earned Bid Earned

Author 1 (aggressive)

Author 1’s wallet 13 13 13 8 7 7

Paper 1

Paper 2 0

Paper 3 0 6 1

Paper 4 4 3

Paper 5 1 1

Author 2 (cautious)

Author 2’s wallet 0 6 7 7 5 3

2Paper 1 (loan) 8

1Paper 2 (loan) 1

Paper 3 2 3

Paper 4 1 1

Paper 5 2 2

Paper 6 1 1

Paper 7 1 0

Paper 8 1 0

Paper 9 1 0

Paper 10 1 0

Author 3 (very cautious)

Author 3’s wallet 0 1 1 2 2 1

1Paper 1 (loan) 3

1Paper 2 (loan) 0

Paper 3 0 1 0

2Paper 4 (loan) 4

Paper 5 1 1

Paper 6 1 0

Author 4 (no risk)

Author 4’s wallet 0 0 0 0 0 2

1Paper 1 0 0 (loan) 1 0

1Paper 2 (loan) 3

* Blue squares indicate unsuccessful bids.

18 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

should be more efficient than bureaucraticsystems based on collectives of scientistswho have reached a consensus of what’sgood or bad. Finally, even unrelated re-searchers might want to invest their cita-tions in a paper by a promising young tal-ent for a profit of future citations.

It’s beyond this article’s scope to discussthe citation auction diffusion and promo-tion mechanism, which requires deep in-sight into the auction model.3 Clearly, we’llneed a new economic model derived fromthe citation auction to foresee how auctionsincrease the quality of research, decreasepeer review costs and publication time, andso on.

A hypothetical caseTable 1 represents a possible scenario

with a simulation of four researchers usingdifferent auction strategies (aggressive, cau-tious, very cautious, and avoiding all risk).The Bid column indicates an author’s bid;blue squares indicate unsuccessful bids. TheEarned column indicates the number of cita-tions received by each paper published viathe auction. For each year in which a paperwas published, the Earned column showsthe number of citations that as of May 2006were made to that paper (to increase realism,we used actual data from a group of confer-ences). For simplicity, we use a first-pricesealed-bid auction, where every author sub-mits in a closed envelope a bid stating howmany citations he or she is offering. This bidsupposes to be lower than the number of ci-tations the author will collect after the paper’spublication. Some authors who couldn’t bidhigher than 0 because their citation walletswere empty borrowed citations from col-leagues and paid back the loans from thecitation earnings.

To measure papers’ impact, we can ex-

amine several rankings of authors. The firstis the citation wallet ranking, based on howmany citations remain in an author’s walletafter an auction. In this scenario, author 1clearly leads with seven citations, followedby author 2 with three citations, author 4with two, and author 3 with one. The walletvalues are lower than the total number ofcitations for each author, which leads toanother ranking (author 1, author 2, author3, and author 4). However, the citation wal-let ranking has the advantage of reflectingthe actual number of citations (not just thenumber of expected citations) that the en-tire set of publications has generated for anauthor in a given time period. This rankingcan track the behavior along a scientist’scareer, unlike the H-index,7 which just pro-vides a cumulative analysis of the scientist’sbest publications. So, these two measuredifferent things. If an author’s papers areranked in descending order of their num-bers of citations, the H-rank is the largestrank that is smaller than the number of ci-tations that the corresponding paper has.Hence, the H-index tries to capture the im-pact of the key papers throughout a scien-tist’s history. Citation wallet ranking triesto catch the impact of all the scientist’spublications, which could be equal to theH-index in the case of outstanding sci-entists but will differ for other scientists(most of us). Table 2 compares severalrankings, including the H-index.

Table 2 also shows the outcome comparedto expectations from the auctions shown intable 1. The Earned Citations and Total Pro-ductivity columns accurately measure theaccepted papers’ quality. So, authors haveincentive to gain the maximum number ofcitations, at least as many as they investedin the bid. Moreover, this system is self-regulating. If an author repeatedly under-

performs in citations (the author’s papersreceive fewer citations than what he or shebid), the author’s wallet will eventually ap-proach zero. This might happen even to pro-ductive authors, such as author 1 in table 2,whose bid productivity is negative althoughthe citation productivity is quite high. Suchauthors will have difficulty assuring the pub-lication of their papers, because they willlose many auctions. Conversely, authors canquickly increase their wallet’s size by sub-mitting papers that will be highly cited, thusmaking it easy to publish papers in the fu-ture. Although the wallets of the most con-servative strategies tend to grow, the otherstrategies might oscillate. This effect clearlydiffers from existing citation measures,which always grow through time.

Making citation auctions usable will in-volve many steps. The most important are to

• create a proof-of-concept of citation auc-tions, using a few selected publications;

• explore the most appropriate auctionmechanisms;

• develop the technology for a citationbank using standard citation engines; and

• study the use of “taxes” for stabilizingthe resulting economic model.

The adoption of citation auctions mighttake decades. However, the increased effi-ciency of scientific activities arising fromtheir use should let them prevail over theless efficient peer review.

Acknowledgments

This work was supported partly by the EUproject grant 34744 (ONE: Open Negotiation

NOVEMBER/DECEMBER 2007 www.computer.org/intelligent 19

Table 2. Author rankings generated by classic measures, the H-index, and citation auctions.

Accepted Total Earned Total BidAuthor papers citations citations productivity H-index Wallet Earnings productivity Losses

A1 4 18 5 1.25 2 or more 7 –6 –1.50 6

A2 10 16 16 1.60 2 3 5 0.50 0

A3 6 8 8 1.33 2 1 1 0.17 0

A4 2 4 4 2.00 1 2 2 1.00 0

The best A2 A1 A2 A4 A1 A1 A2 A4 A2, A3, A4

Second best A3 A2 A3 A2 A2 or A3 A2 A4 A2 A2, A3, A4

Third best A1 A3 A1 A3 A2 or A3 A4 A3 A3 A2, A3, A4

The worst A4 A4 A4 A1 A4 A3 A1 A1 A1

Environment, FP6-2005-IST-5, ICT-for Net-worked Businesses), and by US National Sci-ence Foundation grant OISE-0334667. This ar-ticle’s content doesn’t necessarily reflect thesponsors’ position or policy; no official endorse-ment should be inferred or implied.

References

1. C. Carrillo et al., Social Currencies andKnowledge Currencies, IOS Press, 2006, pp.266–274.

2. P.G.Altbach, “The Tyranny of Citations,” InsideHigher Ed, 8 May 2006, http://insidehighered.com/views/2006/05/08/altbach.

3. J.-S. Lee and B.K. Szymanski, “A Novel Auc-tion Mechanism for Selling Time-SensitiveE-services,” Proc. 7th Int’l IEEE Conf. E-Commerce Technology (CEC 05), IEEE Press,2005, pp. 75–82.

4. J.-S. Lee and B.K. Szymanski, “Auctions as aDynamic Pricing Mechanism for E-services,”Service Enterprise Integration, C. Hsu, ed.,Kluwer, 2006, pp. 131–156.

5. J.L. de la Rosa, Outline of Citation Auctions,IOS Press, 2006, pp. 299–307.

6. J.L. de la Rosa and B.K. Szymanski, “CitationAuctions as a Method to Improve Selection ofScientific Papers,” Proc. 2nd Int’l Conf. Dig-ital Information Management (ICDIM 07),IEEE CS Press, 2007, pp. 479–486.

7. J.E. Hirsch, “An Index to Quantify an Indi-vidual’s Scientific Research Output,” Proc.National Academy of Sciences, vol. 102, no.46, pp. 16569–16572; www.pnas.org/cgi/reprint/0507655102v1.pdf.

20 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Josep Lluis de laRosa is the found-ing director of theUniversity of Gir-ona’s EASY InnovaR+D TransferenceCentre for Innova-tion and BusinessDevelopment. Con-

tact him at peplluis@eia.udg.edu.

Boleslaw K. Szy-manski is the Clairand Roland SchmittDistinguished Pro-fessor of ComputerScience and thefounding director ofthe Center for Per-vasive Computing

and Networking at the Rensselaer Polytech-nic Institute. Contact him at szymansk@cs.rpi.edu.