16 Scientifi c American, June 2011

Forum by John P. A. Ioannidis

Commentary on science in the news from the experts

COMMENT ON THIS ARTICLE ONLINE

Scientifi cAmerican.com/jun2011

Illustration by Ben Gibson

False positives and exaggerated results in peer-reviewed scien-tifi c studies have reached epidemic proportions in recent years. The problem is rampant in economics, the social sciences and even the natural sciences, but it is particularly egregious in bio-medicine. Many studies that claim some drug or treatment is ben-efi cial have turned out not to be true. We need only look to con-fl icting fi ndings about beta-carotene, vitamin E, hormone treat-ments, Vioxx and Avandia. Even when e� ects are genuine, their true magnitude is often smaller than originally claimed.

The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators—and the number of experiments, observations and analyses they pro-duce—has also increased exponentially in many fi elds, but ade-quate safeguards against bias are lacking. Research is fragment-ed, competition is fi erce and emphasis is often given to single studies instead of the big picture.

Much research is conducted for reasons other than the pursuit of truth. Confl icts of interest abound, and they infl uence out-comes. In health care, research is often performed at the behest of companies that have a large fi nancial stake in the results. Even for academics , success often hinges on publishing positive fi ndings. The oligopoly of high-impact journals also has a distorting e� ect on funding, academic careers and market shares. Industry tailors research agendas to suit its needs, which also shapes academic priorities, journal revenue and even public funding.

The crisis should not shake confi dence in the scientifi c meth-od. The ability to prove something false continues to be a hall-mark of science. But scientists need to improve the way they do their research and how they disseminate evidence.

First, we must routinely demand robust and extensive external validation—in the form of additional studies—for any report that claims to have found something new. Many fi elds pay little atten-tion to the need for replication or do it sparingly and haphazardly. Second, scientifi c reports should take into account the number of analyses that have been conducted, which would tend to down-play false positives. Of course, that would mean some valid claims might get overlooked. Here is where large international collabora-tions may be indispensable. Human-genome epidemiology has re-cently had a good track record because several large-scale consor-tia rigorously validate genetic risk factors.

The best way to ensure that test results are verifi ed would be for scientists to register their detailed experimental protocols be-fore starting their research and disclose full results and data when the research is done. At the moment, results are often selectively reported, emphasizing the most exciting among them, and outsiders frequently do not have access to what they need to replicate stud-

ies. Journals and funding agencies should strongly encourage full public availability of all data and analytical methods for each published paper. It would help, too, if scientists stated up front the limitations of their data or inherent fl aws in their study de-signs. Likewise, scientists and sponsors should be thorough in disclosing all potential confl icts of interest.

Some fi elds have adopted one or several of these mechanisms. Large international consortia are becoming commonplace in epi-demiology; journals such as Annals of Internal Medicine and the Journal of the American Medical Association instruct authors to address study limitations; and many journals ask about confl icts of interest. Applying the measures widely won’t be easy, however.

Many scientists engaged in high-stakes research will refuse to make thorough disclosures. More important, much essential re-search has already been abandoned to the pharmaceutical and biomedical device industries, which may sometimes design and report studies in ways most favorable to their products. This is an embarrassment. Increased investment in evidence-based clinical and population research, for instance, should be designed not by industry but by scientists free of material confl icts of interest.

Eventually fi ndings that bear on treatment decisions and poli-cies should come with a disclosure of any uncertainty that surrounds them. It is fully acceptable for patients and physicians to follow a treatment based on infor-mation that has, say, only a 1 percent chance of being correct. But we must be realistic about the odds.

COMMENT ON THIS ARTICLE ONLINE

Scientifi cAmerican.com/jun2011

John P. A. Ioannidis is C. F. Rehnborg Professor in Disease Prevention, professor of medicine and of health research and policy, and director of the Stanford Prevention Research Center at the Stanford University School of Medicine.

An Epidemic of False ClaimsCompetition and confl icts of interest distort too many medical fi ndings

sad0611Foru3p.indd 16 4/14/11 6:08:24 PM