Mysteries of Peer-Review - University of Calgary in Alberta · Cover Letter • Address it to the correct individual and spell their name correctly • Indicate importance of the

Mysteries of Peer-Review Howard Bauchner, MDEditor-in-Chief, JAMA and The JAMA NetworkProfessor of Pediatrics and Public HealthBoston University School of Medicine (on leave)

© 2016 American Medical Association. Confidential and Privileged.

JAMA Pediatrics and JAMA • JAMA Pediatrics

– First published in 1911 (oldest pediatric journal in the world)– Renamed in 2012– Major focus on digital since 2012– All articles published online ahead of print– Part of The JAMA Network

• JAMA– First published in 1883– General medical journal– Appears 48 times per year– Flagship publication of AMA, but editorially independent– Importance of the brand

Mysteries of Peer-Review and Decisions of Journals • Slide 2


Quality of A Journal• Reputation• Reach – website, eTOC, views, social media,

press coverages• Value added material • Respect for Authors• Attention to detail• Impact Factor• Do you like the contents



JAMA Pediatrics - Quality• Reputation - part of the JAMA Family, brand recognition, preeminent

publishing group • Reach – website (design responsive, fast, split screen, surfaces all

content), eTOC (57K), social media (34K twitter and Facebook), views (4.3M), press (12K media mentions, NYT 20, NPR 2, WP 10, WSJ 2, Reuters 98, LA Times 5, US Today 2), Altmetric Score

• Value added material – editorials, VPs, clinical material• Respect for Authors – many innovations, easier submission system,

love notes, depositing in PMC, cascading• Attention to detail – among best copy editors in the world• Impact Factor – 10.25 (No. 1 ranked general pediatric journal in the

world)• Do you like the contents – hope so!



Altmetric Score



JAMA Pediatrics – Hybrid Open Access • What is open-access?

– Began in 2000s– Sense that science should be free– Reverses traditional business model of publishers owning content

and create business model vs authors own content and publishers generate income by charging author fees

– Increasingly mandated by funders

• What is a hybrid journal?– Author presented with option upon acceptance, standard journal policies

or open-access (author pays model)

• How does it affect you!



Content: JAMA and JAMA Pediatrics• Viewpoints• Tabstracts• Clinical Evidence Synopsis• Visualizing Health Policy • Medical Letter Reports• Special Communications• Guideline Reviews

• Revival of Users’ Guides• JAMA Forum• Redesign of MMWR/CDC• Diagnostic Test Interpretation • Guide to Statistics and Methods• IHI Experimentation • Clinical Trials Update• JNC 8 Microsite (with Spotify)• White Board Videos



Talking Points• Correct journal• Instructions• Cover letter• Peer-Review• Structure of an article• Major themes of the paper• Rejection without review• Paper reviewed• Editors • Responding to reviews• Other issues• Keys to success• http://www.equator-network.org



Is it the Correct Journal?• This is a critical issue for authors and journals• Can be subjective decision by journal• Beware of case-reports• Is the topic timely • Most journals can reject without review• Review the journals you are considering• Is it worth shooting high, but failing (impact factor)• Must read journal prior to submission


© 2016 American Medical Association. Confidential and Privileged. Mysteries of Peer-Review and Decisions of Journals • Slide 10


Impact Factors* – 2016• AIM 17 (5586/326)• BMJ 21 (10268/494)• BMJ Open 2.4 (5996/2531)• Cell 30 (26548/873)• Health Affairs 5.0 (2435/489)• JAMA 45 (18872/425)• JAMA-IM 17 (5226/316)• JAMA-Neuro 10 (3139/313)• JAMA-Peds 10.3 (2532/247)• JAMA-Psych 15 (3643/238)• Lancet 48 (27742/580)• Nature 40 (70761/11763)• NEJM 72 (50322/695)• Pediatrics 5.7 (7582/1329)• Journal of Pediatrics 3.9 (3219/831)• PLOS Medicine 12 (2669/225)• PLOS One 2.8 (183193/58157)• Science 37 (61612/1656)

*No. of citations/no. substantive reports previous 2 years



Follow the Instructions!• Article type – original, review, etc.• Cover page – title, word count• Length – critical issue (less is more)• Abstract – single most important page• Tables and figures• References• Supplemental material• Speak with editor(s) first – although often generates a

standard response• Make sure clinical trials are registered before you recruit

patients



Cover Letter• Address it to the correct individual and spell their name

correctly• Indicate importance of the paper• Suggest why this journal is a “good” fit• Can site some articles from that journal• Short and sweet



Structure of an Article• Introduction

– 2-3 paragraphs– Brief 300-400 words– The frame for your study– Invoke higher authority

• Methods– 3-5 paragraphs– Explain in English and justify use of unusual statistics

• Results– 5 paragraphs– 1st paragraph describe the sample– Text should not duplicate tables/figures

• Discussion (structured)– Principal findings– Strengths and limitations– Strengths and Limitations vis a vis other studies– Meaning of study– Unanswered questions/future research

• References – landmark papers and most recent publications (last 5 years)



Major Theme of Paper• What are your 2-3 important points (that is remembered)• Emphasize in results section of the abstract• Conclusion of the abstract should reflect these points • Highlight in results section of paper• Emphasize in tables• Highlight in first paragraph of conclusion



A Good Abstract 90% of us read ONLY the abstract Structured Concise (300-400 words, 10,000 characters) Keep odd abbreviations to a minimum Some data Accuracy of data Beware dataless abstracts

Conclusions Best to have outside reader



A Good Abstract


A Good Abstract


Dataless Abstracts!!!• Results Mixed-modeling analyses were used to examine differences in the rate of

weight gain over time based on the extent to which children exhibited the ability to self-regulate in the behavioral procedures. Compared with children who showed high self-regulation in both behavioral protocols at ages 3 and 5 years, children who exhibited a compromised ability to self-regulate had the highest BMI z scores at each point and the most rapid gains in BMI z scores over the 9-year period. Effects of pubertal status were also noted for girls.

• Results Multivariate linear regression analyses showed that increased sugar-sweetened beverage intake was independently associated with increased HOMA-IR, systolic blood pressure, waist circumference, and body mass index percentile for age and sex and decreased HDL cholesterol concentrations; alternatively, increased physical activity levels were independently associated with decreased HOMA-IR, low-density lipoprotein cholesterol concentrations, and triglyceride concentrations and increased high-density lipoprotein cholesterol concentrations. Furthermore, low sugar-sweetened beverage intake and high physical activity levels appear to modify each others' effects of decreasing HOMA-IR and triglyceride concentrations and increasing high-density lipoprotein cholesterol concentrations.



Poor AbstractPurpose: To determine the prognostic value of early electroencephalograms (EEGs) in full-term neonates suffering from hypoxic ischemic encephalopathy (HIE) exposed to whole body hypothermia (cooled group), compared to neonates treated conventionally (control group). Design: This retrospective observational study included all full-term neonates born at Grenoble Hospital between 2000 and 2006 with symptoms of HIE. The first two EEGs were reviewed retrospectively and classified according to current electrophysiological criteria. EEGs in the cooled group were performed at a mean body temperature of 33°C. Neurological outcome was correlated with EEG pattern. Results: An EEG pattern of inactivity (stage 4) on the first EEG was associated with death in 60% of the control group, while all survivors had neurological sequelae. In the cooled group, this EEG pattern was only predictive of death in 40% and all survivors but one had a normal outcome. Mild abnormalities on the first EEG correlated with a good prognosis in both groups. The second EEG had a high predictive value, particularly in the cooled group. Persistence of stage 4 three days after birth was always associated with death.Implications: Early EEGs are good prognostic indicators of outcome in neonates with HIE. Stage 4 EEGs may not always indicate a poor prognosis in neonatal HIE treated with hypothermia.



A Good Introduction• Short• Focused• Evoke higher authority• 10-15 references• Can be 25% shorter• Criticize others with care



Introduction• Numbers 5 weeks after publication

• Slide 22


Good Introduction

• Slide 23

• First paragraph-General introduction to the topic along with the scope of the problem

– 6.6 million people see primary care clinicians for sore throat and often get antibiotics to reduce symptoms

• Second paragraph-Narrows in on the aspect of the problem this article is about

– Steroids reduce inflammation and may accelerate symptom reduction

• Third paragraph-States research question (preferably using the word hypothesis)

– Does a single dose of dexamethasone increase the resolution of sore throat symptoms?


Good IntroductionArtemisinins are established antimalarial agents with an excellent safety profile.1Artemisinin-based combination therapies are now recommended by the World Health Organization (WHO) as first-line treatment of uncomplicated falciparum malaria in all areas in which malaria is endemic.2 Replacing ineffective, failing treatments (chloroquine and sulfadoxine–pyrimethamine) with artemisinin-based combination therapies has reduced the morbidity and mortality associated with malaria.3,4,5 Parenteral artesunate is replacing quinine for the treatment of severe malaria.6 Recently, there have been signs that the efficacy of artemisinin-based combination therapy and artesunate monotherapy have declined in western Cambodia.7,8,9,10 Artemisinin resistance would be disastrous for global malaria control. To characterize treatment responses to artemisinin derivatives and provide evidence for planning containment-measure strategies, WHO and the National Malaria Control Programmes of Cambodia and Thailand established a multipartite task force. The trials reported here were conducted as part of this initiative.



Concise Introduction

Introduction:A car occupant could be killed if struck by another occupant who was catapulted forward, backward, or sideways in a crash.1- 3 We used a matched cohort study design to estimate the association between the death of a car occupant and the restraint use of another occupant in the same car.

Cummings and Rivara, JAMA 2004


Methods and Results• Methods

– Clarity – could follow subject through the study– IRB approval– Statistics – reproduce the analysis if given the data set

(all assumptions justified)– Primary vs secondary vs exploratory outcome– Focused

• Results– Marry text to tables– First section – who is in the study– Primary then secondary then exploratory outcomes



Rejection without Review• Usually editors/sometimes committee• JAMA – Pediatrics (2017)

– 1795 original research reports• 71% rejected without review (average 1 day)• Acceptance rate 6%

• JAMA – Neurology (2017)– 1473 original research reports

• 65% rejected without review (average 4 days)• Acceptance rate 9%

• JAMA – Internal Medicine (2017)– 1963 original research reports

• 80% rejected without review (average 2 days)• Acceptance rate 5%

• TJN Journals: time to rejection without review (range) – 1-3 days; reject without review ~70%; acceptance rate ~ 10%



Rejection without Review - Why?• Wrong journal – journals have specific missions• Incorrect format (editors are sensitive beings)• Data too old to be relevant• Not new or novel• Similar paper recently accepted/published (read journal before

submitting)• Poorly written abstract• Poorly designed/wrong analysis• Sweeping conclusion• Case-report• You did not read the journal• Editor having a bad day (this happens) !!!



Paper Accepted for Review • Assigned to editor (not EIC)• Most editors have areas of expertise• Editors may send article out for review (rejection

without review can occur here)• No magic number of reviewers – 2 to 5• At JAMA every research paper undergoes

statistical review prior to manuscript meeting• At JAMA Pediatrics every research paper

undergoes statistical review prior to decision



Variations in Peer Review Models & Methods

Process/ Task

Traditional/Conservative

New/Liberal

Type of review Double-blind Single-blind Open

Reviewerassignment

By editor Automated from defined database

Post-publication and open to all

Acceptancecriteria/quality control

Soundness, importance,originality, contribution, fit, and presentation

Technical soundness only

Cursory check

Transparencyof peer review

None; or perhaps only lists of peer reviewers published

General stats about journal acceptance rates and turnaround times

Specific information on reviewers, history, and comments available during peer review and published with articles

Reuse of reviews

None Shared within family or group of journals

Pre-obtained reviews from services and published with articles

Based on Bjork BC, Hedlund T, Emerging new methods of peer review in scholarly journals. Learned Publishing. 2015; 28(2)85-91.


Sample of Leading Journals and Peer Review

Double-blind

Single-blind

Open Options/Comments

NEJM X

Lancet X

JAMA X

Annals X

BMJ X For research articles, attributed reviews and all editorial comments posted with publication

Nature X X Authors can choose single- or double-blind

Science X

PNAS X NAS members can choose a limited number of papers for which they select their reviewers

PLOSBiology

X Reviewers can sign their reviews if they wish

eLife X Open and collaborative throughout the process


Manuscript Processing at JAMA Nework


Triaging Editor(paper can be

rejected)

Associate Editor (paper can be

rejected)

Assigned to 2-5 Peer-Reviewers (paper can be

rejected)

Triaged to Manuscript

Meeting (stats review mandatory)

Rejected or Revised or ERBR

Manuscript Returned to Author for Revision

Returns to Manuscript

Meeting

ACCEPTED or REJECTED !!!


What’s Inside the box?Example: JAMA’s and JAMA Pediatrics single-blind peer review process


Peer-Reviewers• You can recommend reviewers to editor (and individuals

not to review – plus/minus)– Recommended reviewers rate paper the same as others, but more

often recommend acceptance

• Chosen from “list” of reviewers that journal generally uses• Some subject areas difficult to find reviewers – editors

search reference list or Medline• Process takes 1-3 months



Peer-Review • Little science – a fair amount of research• Quality varies, best reviewers are 35 to 45• Time – 2-4 hours depends upon manuscript• Not much difference between blinded and unblinded

reviews• Increasing concern about biases/conflicts of interest of

reviewers• Transparency being debated



What Do Reviewers Assess• Importance• Clarity• Design and analysis• Should review abstract, text, tables, figures, references,

acknowledgements/support• Make recommendation to editor• Opinions of reviewers are not binding• Usually provide comments to authors and separate

comments to editors



Editors• Review paper• Review comments from peer-review• May request statistical help• Make recommendation to and participate in manuscript

review meeting – Accept; accept with revision; reject with revision; reject; short

report; research letter

• Discussed vis a vis importance and validity



Responding to Reviews• Answer completely, answer politely, answer with evidence• Most times the reviewer/editor are correct• Reviewers provided conflicting suggestions - ask editor• You do not have to respond to every issue, but must

articulate why not• Follow directions – i.e. number responses, indicate

changes in manuscript and where they can be found• Long explanations to editor in cover letter is not the same

as modifying the text



Polite Responses• We agree with the referee that ---- but…• The referee is right to point out ---- yet…• Although we agree with the referee that…• We, too were disappointed by the low response rate.• We support the referee’s assertion that ---, although…• With all due respect to the reviewer, we believe that this

point is not correct.Data not words are a better response



Responding to ReviewsDear Dr. Moyer,We are pleased to resubmit the revision of our A/R - Manuscript 2010-3686, Management of Children with Sickle Cell Disease: A Comprehensive Review of the Evidence” for Pediatrics. We appreciate the valuable and detailed comments provided by the reviewers. In addition to the point-by-point responses, we have provided our responses to the items from the editors below. Please do not hesitate to contact us if you have additional suggestions on improving the paper. Thank you.I. Response to Editors:1) Your revised paper should not exceed 2700 words (excluding the abstract, acknowledgments, references, tables, figures, and appendices). Response: The word count of this revised manuscript is 2699 words. Minor edits were made throughout this manuscript in order to address the reviewers’ comments.2) Your main title in the Scholar One title box (see above) is different from your main title on the manuscript itself. They should be the same. Response: We apologize for this confusion. The title in the Title Box now matches the title on the Manuscript, “Management of Children with Sickle Cell Disease: A Comprehensive Review of the Evidence.”



A Note To GodDear God:

Thank you for submitting your paper about “Creating Life.” Your paper was reviewed by external editors and the consensus is that it will not be acceptable in its present form. We were quite concerned about your methodology. The main issue raised by the Reviewers were:

- How do we know Life would not have happened without you? You need better controls. The fact that you did it a few billion times since then does not matter. It was not randomized and might be subject to selection bias. Finally, we need more outcome data about your creation. What is Life supposed to achieve?

Please resubmit addressing all the reviewers concerns point by point. Given the extensive changes required, your paper will be treated as a new submission.

We look forward to seeing more high impact research from you.




JAMA’s Interest• Novelty• Randomized clinical trials• Effect on clinical care or population health• Large effect – rare disease• Small effect – common disease• Public health emergency



JAMA Pediatrics Interest• Novelty• Randomized clinical trials• Effect on clinical care or population health• Large effect – rare disease• Small effect – common disease• Public health emergency



Lower Priority• Surveys• Older data • Associations without clinical implications• Small information slices• Qualitative research



JAMA & TJN – Common Mistakes-1• Abstract – must be perfect – with numbers• Relative vs absolute differences• Too many messages and comparisons• Inconsistencies• Lack of clarity• Exaggeration of findings (conclusion of abstract must

match data in abstract)• Methodological/statistical clarity



Common Mistakes - 2• Too much first person• Too many references• Numbers that do not match• Misuse of trend, marginal significance, or P value (highly

significance)• Response letter – poorly organized, not responsive



Other Issues• Authorship• Fabrication, Falsification, Plagiarism • Ethics • Duplicate publication• Honorary authors• Ghost authors• Conflicts of interest• Spin and boasting• General mistakes



Authorship – ICMJE Requirements• Substantial contributions to the conception or design of the

work; or the acquisition, analysis, or interpretation of data for the work; AND

• Drafting the work or revising it critically for important intellectual content; AND

• Final approval of the version to be published; AND • Agreement to be accountable for all aspects of the work in

ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.



Fabrication and Falsification• Fabrication – making up data or results and

recording or reporting them• Falsification – manipulating research

materials, equipment, or processes, or changing or omitting data or results



Plagiarism Definition:

To steal and pass off (the ideas or words of another) as one's own To use (another's production) without crediting the source To commit literary theft To present as new and original an idea or product derived from an existing source.

Types: Ghost author – author turns in another’s work, word-for-word Photocopy – writer copies significant portions of text straight from a single source Self-stealer – borrows generously from previous work (self-plagiarism) Forgotten Footnote – writer mentions user’s name, but neglects the reference Too-Perfect Paraphrase – writer properly sites source, but neglects quotation marks

*http://www.plagiarism.org/, last accessed March 20, 2011


http://www.plagiarism.org/


Ethics• Always seek ethics committee approval• Researchers should never decide for themselves if IRB approval is necessary

• Quality improvement versus research



Duplicate Publication• Definition: is the publication of a paper that overlaps substantially with one

already published. Both papers need not be identical to be considered a duplicate/redundant publication. Other terms that have been used include dual, divided, republication, fragmented, prior, repetitive, and salami slicing. It is especially inappropriate when an author does not notify the editors nor reference the similar/redundant work in the body of the submitted paper

• Types:– Identical articles or articles with identical paragraphs– Highly similar articles with similar data, patients, experiments– Segmented articles where “salami slicing” produced several articles when

one would have been appropriate– Sequential research articles based on previously methodology with no

new concepts• Ask/Inform the editor – it is not worth it!

http://www.wame.org/wame-listserve-discussions/duplicate-publication, last accessed March 20, 2011


http://www.wame.org/wame-listserve-discussions/duplicate-publication


Honorary Authors • Definition: individuals who are named as authors but have

not met ICMJE authorship criteria– Substantial contribution to conception and design, or acquisition of data, or

analysis and interpretation of data; AND– Drafting and revising it critically; AND– Final approval of version submitted for publication AND– Agreement to be accountable for all aspects of the work in ensuring that

questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

• Approach:– Authorship should be discussed at start of project– Never circulate a draft with names without discussion– Your mentor can –and should - help with this



Ghost Authors Definition: individuals who have made a substantial

contributions to the work reported in an article but are not named as authors Wislar et al (BMJ, 2011): queried 896 authors of research

and review articles and editorials published in JAMA, AIM, Lancet, Nature Medicine, NEJM in 2008 630 responded (70.3%) 21% had honorary or ghost authorship or both

17.6% had honorary authorship 7.9% had ghost authorship



Conflicts of Interest• “For an investigator, a conflict of interest exists when the prospect of some

personal advantage is strong enough to pose a realistic possibility that he or she might compromise the researcher’s primary obligation to adhere to sound scientific procedures in an unbiased search for the truth.”

Cohen J, Academic Medicine, 2001

• “Conflicts of interest are ubiquitous and inevitable in academic life. The challenge for academic medicine is not to eradicate them, which is fanciful and would be inimical to public policy goals, but to recognize and manage them sensibly and effectively.”

Korn D, JAMA, 2000



Spin and Boasting• Exaggeration of Importance

– “has reached alarming proportion,” drastic increase”

• Unfairly disparaging previous research– “crisis of credibility, ”methodologic flaws”

• Using words to convince, not illuminate– “these analyses provide clear answers," the robust association”

• Boasting – “findings open new frontiers,” “strong new evidence”



Be Modest• They could have said “These crucially important and novel findings will

transform the way scientists think about genetics”• But they did not:


Watson and Crick


General Mistakes• Circulating a draft before discussing authorship• Rushing the abstract at the end• Poorly referenced paper• Spelling errors in text and references• Data in abstract that are not in the paper• Data in abstract that are different from the paper• Bait and switch – emphasizing secondary rather

than primary outcomes



Your First Draft• Just write!• Do tables or abstract first• Circulate as a full paper• Double space except for tables/references• Do not list authors without talking with senior author• Date all drafts• Prompting your co-authors• Get examples of similar papers• http://www.equator-network.org


http://www.equator-network.org


Keys to Success• Clarity (abstract)• Brevity (2500 words)• Novelty (why this journal)• Modesty (some)• Read the journal (often)



Getting Articles Published• Revise and revise – 10 drafts• Senior colleagues are critical• Good luck – it feels great!!!



What We Have Learned

• Writing is a learned skill• Try to do state-of-the-art research – be at the leading

edge• Collaborate with others – more fun, better research• Do not write something you cannot publish• There is (nearly) always a journal out there that will

publish your work. Rejection does not mean it is a poor paper (most times).


Resources (www.equator-network.org)

© 2017 American Medical Association Confidential and Privileged

• Slide 65

Documents

Mysteries of Peer-Review - University of Calgary in Alberta · Cover Letter • Address it to the correct individual and spell their name correctly • Indicate importance of the