Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?
37
Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review? Ivan Oransky, MD Co-Founder, Retraction Watch Executive Editor, Reuters Health Scripps Howard Institute on the Environment and Science May 23, 2012
Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?
1. Fraud and Why Studies are Flawed:Should Journalists Trust
Peer Review?Ivan Oransky, MDCo-Founder, Retraction Watch Executive
Editor, Reuters Health Scripps Howard Institute onthe Environment
and ScienceMay 23, 2012
2. Can You Trust Journal Studies? How good is peer review?
Positive publication bias Over-reliance on embargoed studies How
often it turns out to be wrong How to get it right
3. How Does Peer Review Work? Usually three outside reviewers
Usually anonymous Sometimes in as little as 72 hours
4. How Good Is Peer Review?
5. How Good Is Peer Review?
6. How Good Is Peer Review? Bad at detecting fraud Slow Costly
(and unfunded) Prone to bias
7. Peer Reviewers: Worse With Experience
8. Positive Publication Bias
9. Positive Publication BiasPublish a trial that will bring
US$100,000 ofprofit or meet the end-of-year budget byfiring an
editor. -- Former BMJ editor Richard Smith
10. Embargoes and the Ingelfinger RuleBy the late 20th century,
journals needed to competenot just with each other but with
newspapers and othermediaIn 1969, the Journal articulated this
relationshipin its Ingelfinger Rule, a policy against
publishinganything that had already appeared elsewhere.
Otherjournals followed suit. This rule, combined withembargo
policies, has led to a carefully choreographedproduction in which
medical journals and the popularpress work cooperatively and
competitively toinfluence the news cycle.-- NEJM, April 19,
2012
11. Even Without Embargoes, Wed Still HaveIngelfinger
12. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005;
2(8): e124
13. How Often Are Studies Wrong?
14. Retraction Watch http://retractionwatch.com Launched August
2010 with Adam Marcus Frequently cited in major news outlets, from
Natureto Der Spiegel to New York Times 250,000 pageviews per
month
15. Retractions on the Rise -The Wall Street Journal
16. Retractions on the Rise
17. Retractions on the Rise -Neil Saunders
18. The Unofficial Record Holder
19. Why Do Journals Retract? -Nature
20. Why Do Journals Retract? Error is more common than fraud
73.5% of papers retracted for error (orundisclosed reason) vs 26.6%
for fraud Most common cause of retraction: scientificmistake (234
papers; 31.5%) Fabrication (including data plagiarism) morecommon
than text plagiarism 67 retractions (9.0%) had multiple causes, but
134papers (18.1%) retracted for ambiguous reasons -Journal of
Medical Ethics 2010
21. Fraud: Image Manipulation
22. Fraud: Faked Data
23. Is Fraud on the Rise?Over the yearssurveys have asked
scientistsdirectly about their behaviour. [T]hese studieshave used
different methods and asked differentquestions, so their results
have been deemedinconclusive and/or difficult to compare. A
non-systematic review based on survey and non-surveydata led to
estimate that the frequency of seriousmisconduct, including
plagiarism, is near 1%. -- Fanelli, PLoS ONE, 2009
24. Is Fraud on the Rise?A pooled weighted average of 1.97% (N
= 7, 95%CI:0.864.45) of scientists admitted to havefabricated,
falsified or modified data or results atleast once a serious form
of misconduct by anystandard and up to 33.7% admitted
otherquestionable research practices. In surveys askingabout the
behaviour of colleagues, admission rateswere 14.12% (N = 12, 95%
CI: 9.9119.72) forfalsification, and up to 72% for other
questionableresearch practices.-- Fanelli, PLoS ONE, 2009
25. Is Fraud on the Rise?Meta-regression showed that self
reportssurveys, surveys using the words falsificationor
fabrication, and mailed surveys yieldedlower percentages of
misconduct. When thesefactors were controlled for, misconduct
wasreported more frequently bymedical/pharmacological researchers
thanothers. -- Fanelli, PLoS ONE, 2009
26. Is Fraud on the Rise?Considering that these surveys ask
sensitivequestions and have other limitations, it appearslikely
that this is a conservative estimate of thetrue prevalence of
scientific misconduct.-- Fanelli, PLoS ONE, 2009
27. This is Transparency?
28. Should Reporters Cover Conferences?
29. Conference Pitfalls Conferences select presenters based on
< 1000 words Urologists at U of Florida & Indiana U studied
126randomized controlled trials presented in 2002-2003
30. Conference Pitfalls RCTs are the gold standard of medical
evidence But the quality of that evidence wasnt pretty No abstract
said how trial subjects were randomlyassigned to different
treatments or placebos None told how the study ensured that neither
theresearchers nor their doctors knew which they got Only about a
quarter said how long researchersfollowed the subjects in the
trial
31. Just Say NoSometimes, its better not to cover something.But
if you must
32. Always Read the StudyWriting about a study after reading
just apress release or an abstract without reading the entire paper
is journalistic malpractice
33. How to Get Studies www.EurekAlert.org for embargoed
material Association of Health Care Journalists membershipincludes
access to Cochrane Library, Health Affairs,JAMA, and many other
journalswww.healthjournalism.org ScienceDirect (Elsevier) gives
reporters free access tohundreds of journals www.sciencedirect.com
Open access journals (e.g., Public Library of Sciencewww.plos.org)
Ask press officers, or the authors
34. Who Has an Interest? Disclose conflicts PharmedOut.org
Dollars For Docs
serieshttp://projects.propublica.org/docdollars/
35. Dont Rely Only on Study Authors Find outside sources. Heres
how:
36. A Dirty Little SecretKeep a biostatistician in your back
pocket Photo by Peyri Herrera, on Flickr