Breaking Bad: Combating Scientific Misconduct€¦ · Grant removal 3% Debarment from obtaining grants and contracts Suspension Removal from a scientific organization Other 8% 52%

©2018 The Science Advisory Board • scienceboard.net 1

Breaking Bad: Combating Scientific Misconduct


Do you believe everything that you read in a nationally recognized scientific journal? How confident are you that scientists arereleasing trustworthy and accurate data? Unfortunately, the answer to these two questions is probably not the resounding “yes” that we want to hear.

Scientific misconduct is an umbrella term to describe actions in the scientific community that have been deemed unethical, inexact, or inaccurate. Although governments around the world have attempted to counter these actions through rules and regulations, poor research integrity can be found in many areas of science. Around the world, scientific misconduct severely diminishes the quality of research journals and their publications. Additionally, it can lead to the production of ineffective medical treatments and inaccurate knowledge.

At The Science Advisory Board®, we’ve decided to harness our research and analytical capabilities to learn more about scientific misconduct. After surveying over 600 scientists around the world, we’ve gathered several significant findings that gives us a far better understanding of the most common forms of misconduct. Through statistics, graphical images, and textual responses, we’ve also attempted to explain why it’s so common in the first place, and how we can potentially improve methods of detection.

Introduction

Introduction


Introduction

• More than half of the scientists surveyed believe the public neither trusts nor distrusts the scientific community, and a further third think that the public highly distrusts the scientific community.

Does the Public Trust Scientists?

Highly Distrust32%

Neither Trust nor Distrust53%

Highly Trust15%

In your opinion, how much does the public trust the scientific community?


Types of Scientific Misconduct


• Misconduct is fairly common, since almost half of the scientists we surveyed witnessed or have been informed of a colleague that has engaged in some form of misconduct.

• Given that the subject of scientific misconduct was disclosed to scientists in their invitation, this high incidence rate may indicate a self selection bias among respondents. This means that those who witnessed misconduct were more likely to participate than those who had not.

• In 2012, Grant Steen analyzed 800 papers published from 2000-2010 on PubMed. He found only 21.3% of retractions occurred due to error, whereas 67.4% occurred due to misconduct, including fraud or suspected fraud (43.4%), duplicate publication (14.2%), and plagiarism (9.8%).

• We will be discussing and focusing on misconduct, as opposed to honest errors throughout this report.

Yes49%No

41%

Prefer not to answer

10%

How Prevalent is Scientific Misconduct?

Misconduct and retractions Ferric C. Fang, R. Grant Steen, Arturo Casadevall, Proceedings of the National Academy of Sciences, Oct 2012, 109 (42) 17028-17033; DOI: 10.1073/pnas.1212247109

Have you ever personally witnessed or been informed that a colleague has engaged in scientific misconduct?

Categorizing and Locating



Where is Misconduct Most Prevalent?

• Respondents from regions other than North America and Europe were more likely to report having witnessed or been informed of a colleague’s misconduct.

North America

Rest of World

Asia

Europe

Yes47%No

44%


9%

Yes49%

No36%


15%

Yes45%

No48%


8%

Yes68%

No23%


8%

Have you ever personally witnessed or been informed that a colleague has engaged in scientific misconduct?


• The top three reported forms of misconduct are related to data manipulation or exclusion to support a narrative.

• 55% of respondents from Asia selected “assembling or reporting data not collected experimentally” compared to just 35% of North Americans and Europeans surveyed.


What Are the Most Common Forms of Misconduct?

Insufficient supervision or mentoring of junior coworkers

Using published ideas or phrases of others without referencing

Manipulation of figures or tables to support claims

39%

Selectively dropping data for a publication after performing an initial analysis

Disregarding other interpretations of data to support a specific hypothesis

28%

Performing statistical tests on data until one receives a desirable result to report

Assembling and reporting data that was not collected experimentally

34%

Purposefully utilizing an experimental design with obvious shortcomings

Claiming or denying authorship wrongfully

Selectively citing research to please editors, reviewers, or colleagues

Deviating from established procedures or protocol

Re-using work from a previous experiment or paper

55%

48%

42%

34%

29%

29%

25%

24%

21%


• The most damaging types of misconduct are outright data fabrication followed by data manipulation.

• Importantly, almost a quarter of scientists consider harassment based on gender or race to be the most damaging form of misconduct, but it does not appear to be frequently reported as a common form of misconduct.


What Are the Most Damaging Forms of Misconduct? 48%

Performing statistical tests on data until one receives a desirable result to report

Selectively citing research to please editors, reviewers, or colleagues

Disregarding other interpretations of data to support a specific hypothesis

Assembling and reporting data that was not collected experimentally

Lack of informed consent from test subjects

Ignoring the quality of treatment for animal test subjects

Harassment based on gender or race

16%

Manipulation of figures or tables to support claims

Using published ideas or phrases of others without referencing

Deviating from established procedures or protocol

Ignoring substantial safety risks of the study participants, workers, or environment

Claiming or denying authorship wrongfully

Insufficient supervision or mentoring of junior coworkers

14%

Selectively dropping data for a publication after performing an initial analysis

Re-using work from a previous experiment or paper

Purposefully utilizing an experimental design with obvious shortcomings

33%

27%

24%

23%

21%

19%

19%

16%

16%

14%

9%

7%

6%


Witnesses and Perpetrators


• Most scientists inform another colleague/superior or warn the colleague directly after witnessing misconduct.

• 16% of scientists took no action when they witnessed misconduct.


Do Witnesses React?46%

Warned my colleague of possible consequences

for his/her actions

Informed a colleague or supervisor

Informed my local office of research integrity

I took no action

Other

44%

Gave information to local press organizations

16%

14%

12%

3%

https://grants.nih.gov/grants/research_integrity/process.htm

https://grants.nih.gov/grants/research_integrity/process.htm


• 18% of respondents in academia/university reported taking no action, while only 8% of respondents in biotech/pharma reported taking no action.

• 10% of respondents in academia/university setting reported that they informed a local office of research integrity, while 22% of respondents in biotech/pharma reported informing a local office.


Do Witnesses React?“I took no action because I had not enough expertise to check the claim”

“We dealt with it internally by repeating the questionable study”

“The colleague in question is a student; he has been required to do a write-up regarding the consequences of the type of misconduct in which he has engaged (plagiarism)”

“Notify editor of journal paper’s (sic) submitted to”

“Informed the responsible office, in the cases (sic) not local.”


• Many perpetrators experienced no negative consequences for their actions.

• 20% of perpetrators had their publications retracted and 17% lost their jobs.

• This may indicate a lack of real consequences or little actual follow up for most scientists who conduct fraudulent research.


Punishing Misconduct

Loss of authorship

Termination of employment

No negative consequences

Loss of status or demotion

Retraction of a publication

Letter of reprimand

Grant removal

Debarment from obtaining grants and contracts

Suspension

Removal from a scientific organization

Other

7%

46%

6%

20%

17%

12%

12%

7%

10%

6%

5%



Punishing Misconduct• Respondents working in pharma/biotech companies were less likely to

report no negative consequences, and almost three times as likely to report termination of employment. They were also more likely to report all other “negative” consequences.

• This may be due to a real risk in pharma/biotech of lawsuit due to injury, death, or illness caused by products or research conducted improperly.

• In an academic settings, punishment and investigations for fraud and misconduct are determined by the institution. If the institution or department is less willing to prosecute research misconduct or is inclined to get rid of the issue by moving the accused along (either out of the program or to a new job), then consequences are less likely to be seen.

• One academic respondent gave a clear example of this approach, “It was a graduate student who felt pressure to graduate in time for a job offer. They were awarded a PhD because it was the only occurrence and a relatively minor offense.”

32%

Termination of employment

No negative consequences

5%

Retraction of a publication

Loss of status or demotion

Loss of authorship

Letter of reprimand

Grant removal

3%

Debarment from obtaining grants and contracts

Suspension

Removal from a scientific organization

Other

8%

52%

20%21%

12%35%

13%

11%17%

11%

6%

9%11%

5%10%

8%

4%14%

5%

Pharma/Biotech

Academic/University


• Almost half of scientists that reported misconduct experienced no negative outcomes themselves.

• Respondents working in pharma/biotech who reported misconduct were more likely to experience ostracization and negative association than academics.

• The consequences varied widely depending on the institution’s policies and how co-workers reacted.

• Some were bullied to the extent that they “found it difficult to continue to work in that environment”, so they left.

• Others experienced the “guilty by association” educational measures: “several seminars on scientific integrity and misconduct.”


Consequences of Reporting Misconduct

2%

No noticeable outcomes47%

Other

Felt ostracized by co-workers

Became negatively associated with said colleague by others

Felt pressure from co-workers to drop allegations

49%

Threats of lawsuits

41%

22%18%

32%

4%

15%

18%

11%24%

9%

14%13%

14%17%

6%

Total Respondents

Pharma/Biotech

Academic/University


Lab Characterization


• More than half of scientists either were mentored on what constitutes scientific misconduct or are currently mentoring someone about scientific misconduct.

• Surprisingly, 14% of scientists report not having received guidance about what constitutes misconduct and a further 12% are unclear as to if they even received training.


Lab Characterization - Mentorship

Yes45%

No14%

Unsure12%

I do not/did not have a

mentor11%

I am currently mentoring someone,

and I have done so

17%

I am currently mentoring someone, and have not done so 1%

Was your mentor explicit and thorough when describing what constitutes honest and ethical science?


• Scientists feel the most strongly that there is pressure to obtain funding, followed by pressure to publish a paper and/or receive recognition.

• Sources of pressure and stress like funding, publications, and desire for recognition are often seen as a strong motivator behind scientific misconduct, but other structural factors may also influence people to behave in morally questionable ways when performing research.


Lab Environment - Sources of Stress or Pressure

48%

41%

34%

25%

28%

34%

31%

36%

24%

25%

36%

39%

There is pressure to publish a paper and/or receive recognition

There is pressure to obtain funding

There is pressure to obtain tenure

There is often not enough time to complete necessary work

Highly Agree Neither Agree nor Disagree Highly Disagree


• Respondents working in academiafeel more pressure to obtain funding, tenure, and publish than respondents working in pharma/biotech.

• The unique pressures felt by academic research scientists may create an environment where scientific misconduct is inadvertently encouraged.



(% “highly agree”)

36%




There is pressure to publish a paper and/or receive recognition 28%

51%

47%

36%

26%

26%

22%

Academic/University

Pharma/Biotech


• Those who have witnessed misconduct are more likely to characterize their lab environment as having more pressure to obtain funding, to publish, and to obtain tenure than those who had not witnessed misconduct.

• These pressures are not the only reasons misconduct occurs, but the perception of these pressures is indicative of scientists whose labs may be at risk of misconduct and those who may not be.





35%

There is pressure to publish a paper and/or receive recognition

41%

52%


48%

40%

25%

27%

24% Witnessing Misconduct - Yes

Witnessing Misconduct - No

(% “highly agree”)



Lab Characterization - Ethical Environment• When describing the nature of the work done in their labs, most scientists identified a collaborative and collective effort.

• When looking for ethical guidance, scientists are evenly divided between their own ideas of right and wrong and the ideas of others within their organization.

39%

30%

24%

37%

40%

35%

25%

30%

41%I mostly look to rules or codes of

conduct established outside of my organization when making decisions

I mostly rely on my individual cognition and morals when

making decisions

I mostly consult other members of my organization, its rules, and

its policies when making decisions

46%

29%

28%

39%

34%

37%

15%

37%

35%

Completing work is a collective endeavor and is completed for

the collective interest of its members

There is a high level of competition and/or

introversion among researchers

There is a strict adherence to rules and policies

Highly DisagreeHighly Agree Neither Agree nor Disagree


Prevention


• More than half of respondents consider the current peer review process to be ineffective at preventing scientific misconduct.

Prevention

Does Peer Review Help Prevent Misconduct?

Highly Effective

8%

Neither Effective

nor Ineffective

32%

Highly Ineffective

53%

• Peer review does not have access to original, raw data.

“Peer review only checks the end product. There is not information on how it was done.”

“It is very difficult to discover scientific misconduct when reviewing a manuscript.”

“More original data should be presented to the reviewers in the form of original gels, western blots etc, especially if selected regions of gels or blots are shown.”

• Peer review does not have the people or resources it needs.

“Peer review is highly demanding in relation to intellectual work and time consuming to be carried out properly. I feel that sometimes reviewers are not interested or not enough involved in the task.”

“I see it fairly consistently, usually just cutting corners. But you need experts to review papers and with multi-disciplinary papers it’s tough to find experts in all the different aspects. There is unequal expectations depending on the authors level of prestige (i.e. junior authors are held to a higher standard).”

“Peer review is reasonably effective at finding trivial mistakes (intentional or not). However, deliberate misconduct is nearly impossible to spot without some analytical tools (image analysis etc.) that compare to known publications. Also, erroneously performed statistical analyses (and other data handling) can be very difficult to spot as this requires not only expert skills but may also require the reviewer to redo the entire data analysis with additional checks”

“When reviewing, I am very careful, but I am not an expert in all areas. I am always concerned that I might miss something in the statistics, or an aspect of technology in which I am not an expert, even though I am an expert in my subject itself. I am also not an expert at detecting image manipulation.”


Prevention

Does Peer Review Help Prevent Misconduct?• Peer review is not intended to assess misconduct in the first place.

“{Peer review} I think is ok, in my opinion the conduct is based in the individual integrity, so, education is the more important consideration.”

“We are not policemen.”

“Reviewers are tasked with evaluating the scientific merit and impact of the research question, approach, design, statistics, team and resources. Very few tools and very little, if any, guidance is provided to Reviewers for evaluating plagiarism, detecting misconduct, or identifying manipulated figures/tables/data.”

“The only way to prevent this is to have experiments duplicated by another research group. But that is of course difficult to achieve.”

“Peer review only examines the scientific merit of the project/study. Reviewers are not capable of scrutinizing potential misconduct”

“It is not the purpose of peer-review to identify scientific misconduct. In the current system, it would be difficult to do so anyway (although clear cases of plagiarism and data re-use may be detected).”

“All peers are in the same boat -- publish and get funding or perish. There is no value in exerting proper pressure against misconduct. Much to the contrary, the system works more smoothly for individuals who do not rock the boat, or in some cases who ignore issues in order to not rock the boat.”

• Are the reviewers part of the problem?

“I think that a significant number of reviewers are prone to give their decisions considering how the paper/project being evaluated impacts on and shakes their own research/results and not the scientific merits of the study itself.”

“Much to the contrary, the system works more smoothly for individuals who do not rock the boat, or in some cases who ignore issues in order to not rock the boat.”

“Senior people are supporting papers that are not threatening for their own research while applying much more severe critique to competitors.”

“Peer review is not open to the whole scientific community to make objections before publishing.”

“Peer review is broken. Too many papers get through based on author names, reputations and prestige.”


• Scientists believe that improvements in data management (both in documentation, practices, and management), as well as the mentoring and supervision of junior scientists, will be the most effective at preventing misconduct.

• Increases in connected lab equipment and laboratory notebooks enable auditing and tracking.

• Mentoring and “passing down” standards and clearly stating rules and expectations so that they become internalized touch points for future decision making is a must.

• Interestingly, among pharma/biotech scientists, quality inspection and tracking of stored materials, instruments and equipment calibration logs is considered more important for preventing fraud than among academics(54% vs. 33%).

• Academics considered increased funding (to decrease pressure on the lab) as much more important than pharma/biotech scientists (33% vs. 14%).

Prevention

If Not Peer Review, What Else is There?Meticulous management and

storage of all data, including primary experimental results

Utilizing computer programs that detect plagiarism

Documentation of expected practices and standards for recording, storing, backing up

or archiving primary and collateral data

Consistent involvement from a Principal Investigator

Inspections of data collection systems

Explicit requirements for data sharing among colleagues and other scientists prior to publication

Mentorship/education delineating scientific misconduct and/or ethical standards

Meeting with supervisors prior to leaving a lab to ensure that all pertinent activities are properly inventoried, and that all original data are retained

Quality inspection and tracking of stored materials, instruments and equipment calibration logs

Rewarding open and reproducible practices in hiring and promoting researchers

Providing more funds to researchers

13%

Accurate and complete financial data recorded for each project

Research protocols take account of, and are sensitive to, relevant differences in age, gender,

culture, religion, ethnic origin and social class

Other

59%

56%

49%

49%

39%

32%

31%

30%

27%

27%

24%

17%

3%


Characterizing Scientists


• Generally, scientists are not concerned about the general level of research integrity and are comfortable discussing the topic.


Characterizing Scientists - Are Scientists Worried?

22%

8%

27%

13%

51%

79%I feel uncomfortable discussing

topics related to research integrity with fellow researchers

I often feel concerned about the level of research integrity



• Regionally, scientists in Asia and the Rest of the World are much more concerned about the research integrity in their labs and are not as comfortable discussing it.


Characterizing Scientists - Are Scientists Worried?

8%

15%I often feel concerned about the level of research integrity

I feel uncomfortable discussing topics related to research integrity

with fellow researchers

17%

43%

30%

5%

11%

11%

North America

Europe

Asia

Rest of World


• Generally, scientists are happy with their current role in the workplace and value knowledge over awards or fame, suggesting these factors are not contributing to misconduct.

• Subjectivity is not valued as a part of the scientific process by most scientists surveyed.

“From the hospital inspections to scientific research, subjective decisions have no place in the academic/clinical/medical world.”


Characterizing Scientists - What Do Scientists Value?

76%

58%

44%

24%

21%

27%

39%

30%

15%

17%

46%

Accuracy, precision, and reproducibility are extremely

important when assessing the quality of data

Subjectivity is considered a necessary part of the

scientific process

I am happy with my current role in the

workplace

The quest to attain knowledge is more important

than recognition or awards



Methodology and References


• A total of 634 respondents completed the survey between June 28th and July 9th, 2018.

• Respondents were from 38 different countries.

• Agreement was measured on an 11-point scale where 10=“Strongly Agree” and 0=“Strongly Disagree”. Values of 9 and 10 were grouped to indicate “Highly Agree”, 7 and 8 to indicate “Neither Agree nor Disagree” and 0 through 6 to indicate “HighlyDisagree”.

Methodology

Methodology

North America

Europe

Australia

Asia

Middle East

South America

Africa

35%

34%

22%

4%

3%

1%

1%

Region Market Segment

Academic/University

68%

Pharma/Biotech

28%

Other Industry

4%


Established in 1997, The Science Advisory Board is an international online community of scientists and medical professionals who share the goal of being the catalyst for future scientific and medical breakthroughs. Our website provides a space for scientists and medical professionals to share, interact, and discover. Members share their knowledge and experience with the community, and through surveys, interviews and focus groups, advise and consult with science-focused companies, scientific publishers and government agencies. Registered members are compensated for their participation in most research studies conducted by The Science Advisory Board.

The Science Advisory Board offers access to an exclusive community of science and medical experts. We provide you with an opportunity to share, network, and collaborate with peers. You can ask and answer important research questions. Join us to influence the development of the next generation of research, development, manufacturing, and medical technologies. All qualified individuals are encouraged to apply for membership by registering at The Science Advisory Board's Signup Page.

Only professionals actively conducting life science research, development, or manufacturing; clinical care and testing; or related disciplines in the applied sciences or engineering are eligible for membership. Applications from full-time graduate students and post-doctoral fellows are also welcome.

We invite you to join us in helping to shape the future of scientific and medical technologies! More information can be found on our website, or email us at [email protected].

www.scienceboard.net

References

About The Science Advisory Board®

https://www.scienceboard.net/index.aspx?sec=log&radio=newmember

mailto:[email protected]

Documents

Breaking Bad: Combating Scientific Misconduct€¦ · Grant removal 3% Debarment from obtaining grants and contracts Suspension Removal from a scientific organization Other 8% 52%