Transparency in measures of scientific impact

What do impact metrics measure?

klout.com

Metric opacity

Metric disconnected from what it measures

Metric consumer can:

•  Make metric-based inferences

•  Compare item score over time

•  Rank other items by the same metrics

Metric opacity: PageRank

Semi-public algorithm, arbitrary adjustments.

Metric opacity: PageRank

The 2002 Google vs Scientology case [1]

ad-hoc adjustment of rankings to comply with DMCA

SEO: reverse-engineering PageRank

The Google dance [2]

short-term fluctuation in PageRank

SEO strategies to control it

[1] McCullagh (2002). Google Yanks Anti-Church Sites, Wired http://www.wired.com/politics/law/news/2002/03/51233

[2] MetaMend. What is the Google Dance? http://www.metamend.com/google-dance.html

Metric opacity: Journal Impact Factor

Public algorithm, arbitrary adjustments

Metric opacity: Journal Impact Factor

Denominator variables (citable items)

Self-citation (from author-level to journal-level self-citation)

The JIF dance

Editorial policies to maximize JIF

publish more reviews

negotiate citable items

Controlling self-citation: the case of niche journals

Benefits of metric opacity

Increase trust in the metric itself

Provide easy to consume impact proxy

“My website has a PageRank 7”

“You increased your Klout score by 2 points”

“I have 12 papers published in 3+ JIF journals”

Better control of gaming

Consequences of metric opacity

Lack of verifiability

Metric lock-in

Metric abuse

Metric transparency: altmetrics

Alternative measures of the impact of scientific artifacts (papers, datasets, slides) based on large-scale online data [3]

Provide transparent, verifiable impact indicators linked to data sources

Provide real-time, multidimensional, semantically rich impact indicators

Disclose areas of impact invisible to traditional bibliometrics and usage factors

Track references and reuse of scholarly contents inside and outside academia

[3] Priem, Taraborelli, Roth, Neylon (2010). The altmetrics manifesto, http://altmetrics.org/manifesto

altmetrics services

readermeter.org

total-impact.org

citedin.org

altmetric.com

readermeter.org

altmetric.com

How to validate altmetrics-based impact? [4]

Using altmetrics to redefine impact

extra-academic usage (industry, education)

reuse of open scholarly contents [5]

[4] Priem et al. (2011). Evaluating altmetrics, in preparation

[5] Neylon (2011) Re-use as Impact. ACM WebSci ’11 Altmetrics Workshop http://altmetrics.org/workshop2011/neylon-v0/

Daniel Mietchen (2011) Reusing open-access materials. http://bit.ly/OA_reuse

stop worrying about impact help us rethink impact

create APIs to expose (and facilitate the reuse of) published contents usage data

help us measure reuse transparently/reproducibly

Conclusions

Thanks

Credits

VTVM 12 volts DC: CC-BY-NC image by 3D King Wildcat vs. Cheetah: CC-BY image by JD Hancock

Dario Taraborelli. Transparency in measures of scientific impact "http://nitens.org/docs/slides/coasp11.pdf"

"dtaraborelli@wikimedia.org ""@ReaderMeter"

Transparency in measures of scientific impact

Technology

corruption barometer - Towards Transparency · Vietnam by increasing demand and promoting measures for transparency, accountability and integrity in government, business, and civil

Transparency in Armaments / Military confidence-building measures

Unpacking Transparency and Accountability Measures...3 MKSS is a grassroots movement that has been working in the state of Rajasthan for the last twenty years on issues of transparency

Transparency Board 6 September 2011 Defra / Environment Agency presentation to Transparency Board Miles Parker, Defra: Deputy Chief Scientific Advisor,

09 Measures of Scientific Productivity and Current Philippine Performance - Dean Caesar a. Saloma

HIP: Transparency work group – session 1...... Price and Quality Transparency Work group title: Price and ... Track and analyze own performance on core measures ... negotiations

Opinion Scientific Committee Veterinary Measures Public Health, … · 2016-12-01 · 4 7(5062)5()(5(1&( The Scientific Committee on Veterinary Measures relating to Public Health

F1000 Bringing Transparency to Peer Reviewweburfist.univ-bordeaux.fr/wp-content/uploads/2016/... · F1000 Bringing Transparency to Peer Review Maaike Pols PhD Scientific Outreach

Banks' Survival during the Financial Crisis: The Role of ... · commercial banks in the United States, we find that transparency enhances stability. We use two measures of transparency

IFRA - SCIENTIFIC WORK PLAN ON ALLERGENS … · Scientific work plan on allergens ... x Measures taken by the industry on the specific material HICC ... management measures but can

Training Needs Assessmentthemisnetwork.rec.org/uploads/documents/ExeCom2015... · Ethics and transparency issues (anti-corruption measures) 5.39% Multilateral Environmental Agreements

ENCYCLOPAEDIA of SCIENTIFIC UNITS, WEIGHTS and MEASURES

Scientific Research Measures

ACHIEVING TRANSPARENCY AND COMBATING IUU FISHING IN … · 2 Contents Executive Summary 3 Essential measures to increase transparency and tackle IUU fishing in RFMOs 8 Identification

MEASURES FOR THE STUDY OF CREATIVITY IN SCIENTIFIC …

Prospects for Transparency and Confidence-Building ...and Telecom Law Program. The first day of the conference, entitled “Transparency and Confidence-Building Measures: Alternative

Scientific Integrity and Transparency Under Scrutiny: Lessons from Retraction Watch

Transparency Against Corruption. Abstract · increasing transparency should be accompanied by measures for strengthening peoples’ capacity to act upon the available information

CEFTA Conference on Transparency on Trade Policy and Measures - Means to Address Non-Tariff Measures – Sarajevo, 29 October 2013 Transparency Tools and

Grading Government Transparency Government Transparency 3 that goal within the federal government. In 2010, Holdren ... The media policy scoring rubric was based on six measures