Dealing with Noise

How to build a post-publication evaluation system

PDF downloads

Nature Genetics (N = 168) for 2005.

R = 0.71

From:Turning web traffic into citationsAction Potential blog

Connotea tag:Evolution

Journal Articles CitationsMBE (2003/2004) 480 6771

Nature (2003/2004) 5193 121932

Science (2003/2004) 5306 124377

Connotea Evolution (2003-2004) 86 3243

Connotea Evolution (2005-2006) 149 5059

November 2006 - Papers in Connotea with tag "evolution” published 2003 or 2004Early 2008 - Papers in Connotea with tag "evolution” published 2005 or 2006

Bloggers talk about interesting papers

There are 519 papers published in 170 journals in the period of 2003-2004 that were mentioned in blogs tracked by Postgenomic.

Quotes

“(…)what is obvious to everyone but bone-headed bean counters. The only way to assess the merit of a paper is to ask a selection of experts in the field. “ David Colquhoun Improbable Science

“In order to shortcut the work of looking up actual (real) citation counts for investigators the journal impact factor is used as a surrogate to estimate the count. I have always warned against this use”. Eugene Garfield (1998)

“Any measurement system will produce gamesmanship to increase success. Lawrence yearns for the one system that obviates it: face-to-face conversations between peers that allows a truly deep and well focused estimate of scientific merit. “ Philip Campbell

Dealing with Noise

Peer-review

Nature, Science, Cell

Lab notebooks, unpublished manuscripts

Jour

nal i

mpa

ct fa

ctor

Perc

eive

d im

pact

Dealing with NoisePe

rcei

ved

impa

ct

Peer-review

Highly recommended papers by your peers

Online lab notebooksPre-print servers

Aggr

egat

ed u

sage

dat

a

Post-publication journals

(…) nobody wants to have to wade through a morass of papers of hugely mixed quality, so how will the more interesting papers in such an archive get noticed as such? Philip Campbell

Post-publication journals

Automatic

Algorithms

Editors

Hubs/Portals