Reporting statistics (and surviving peer review)

Gary Collins and Elaine Beller(and Mark Jones)

The peer-review process• Most journals will perform some sort of statistical

review of your manuscript– Some journals have appointed statistical editors– Some use casual stats reviewers (when needed)– Some ask the authors to get an independent stats

reviewer to review their paper!

• Statistical editors – are (or can be) part of the decision-making process– have an aim to identify errors, improve clarity and

ultimately the quality of the study

Statistician input needed for

• Statistical input for protocol• Statistical analysis plan• Formulating hypotheses (from objectives)• Sample size calculation• Statistical methods section• Data analysis and reporting• Dealing with missing data• Reviewing the final manuscript• Help with responses to peer reviewers

Formulating hypotheses

• Hypotheses are tricky!• Best to get a statistician to help with writing

hypotheses for your study• Based on the study objectives• Are worded in a way that directly relates to

the statistical testing to be performed• Hypotheses may not be relevant for some

studies e.g. exploratory studies

Sample Size• Should be reported in sufficient detail to enable

replication– stats reviewers/editors will often try to replicate the

calculation based on what was described in the paper– report the type I error (alpha), type II error (power)– report the basis for all estimates (ideally from

previous research, pilot study) used in the sample size calculation

• Need to specify your clinically important difference and the assumed variation in the population

• If no sample size calculation was done, then say so

Sample size example

To detect a reduction in PHS (postoperative hospital stay) of 3 days (SD 5 days), which is in agreement with the study of Lobo et al with a two-sided 5% significance level and a power of 80%, a sample size of 50 patients per group was necessary, given an anticipated dropout rate of 10%. To recruit this number of patients a 12-month inclusion period was anticipated**Vermeulen H, Hofland J, Legemate DA, Ubbink DT. Intravenous fluid restriction after major abdominal surgery: a randomized blinded clinical trial. Trials 2009;10:50

Statistical Analysis [Methods]

“Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data […] to verify the reported results” [ICMJE]

Statistical Analysis [Methods]

• All statistical analyses described in the methods section should have a corresponding set of results (and vice versa) – this applies to abstract as well

• Explain the purpose of your analysis• If the analysis is using complex statistical methods:

– don’t dumb down but limit text to describe them– describe rationale for the approach– citations to appropriate papers should be given– further details can be presented in an appendix– if possible, select appropriate reviewers

Statistical Analysis [Methods]

• Multivariable analyses (i.e. regression) should be clearly explained (e.g. multiple [linear], logistic, Cox)

• Specify the outcome being analysed in the regression model

• Specify all variables included in the regression analysis– are all important key confounders (prognostic variables)

being adjusted for?

• Specify whether and how variables were selected for inclusion in the model

A diagram may be useful

Redrawn from Conan MacDougall, and Ron E. Polk Clin. Microbiol. Rev. 2005; doi:10.1128/CMR.18.4.638-656.2005

Stats Methods exampleThe primary endpoint was change in body- weight during the 20 weeks of the study in the intention-to-treat population … Secondary efficacy endpoints included change in waist circumference, systolic and diastolic blood pressure, prevalence of metabolic syndrome … We used an analysis of covariance (ANCOVA) for the primary endpoint and for secondary endpoints waist circumference, blood pressure, and patient-reported outcome scores; this was supplemented by a repeated measures analysis. The ANCOVA model included treatment, country, and sex as fixed effects, and bodyweight at randomisation as covariate. We aimed to assess whether data provided evidence of superiority of each liraglutide dose to placebo (primary objective) and to orlistat (secondary objective)*

*Astrup A, Rössner S, Van Gaal L, Rissanen A, Niskanen L, Al HM, et al. Effects of liraglutide in the treatment of obesity: a randomised, doubleblind, placebo-controlled study. Lancet 2009;374:1606-16

Results: Describing data• Describe characteristics of your data

– often a “Table 1” in an article– report means (standard deviations)

• Normally distributed data– or report medians (interquartile ranges)

• non-Normally distributed data– make sure tables add up

• e.g. columns for group A, group B and total– Report n (%) for binary or categorical data

• If the primary analysis involves comparing groups describe the characteristics of each group

Example Table 1^

^Micek ST, Ward S, Fraser VJ, Kollef MH. A randomized controlled trial of an antibiotic discontinuation policy for clinically suspected ventilator-associated pneumonia. Chest. 2004 May;125(5):1791-9.

Missing data• DESIGN:

– Did you anticipate drop-out prior to starting your study and adjust your sample size accordingly?

– If yes, then clearly report this in the sample size section– Prespecify analytic strategy for dealing with missing data

• METHODS & RESULTS: – When describing your data:

• how many had missing data? • what was missing?

– What did you do with the missing data?• Omit them from the analysis?• Impute them?

P-values

• P-values provide no indication of the direction or magnitude of the effect (which was the focus of the study)– “the effect of the drug was statistically significant”

• Misinterpreted– non-statistically significant P-values (i.e. >0.05) are often

misleadingly described suggesting significance– e.g. “trend towards significance”, “approaching

significance”, “narrowly missed significance’ and many many more => all nonsense

Still not significant…

• ‘a barely detectable statistically significant difference (p=0.073)’• ‘a distinct trend toward significance (p=0.07)’• ‘a favourable statistical trend (p=0.09)’• ‘just tottering on the brink of significance at the 0.05 level’• ‘narrowly escaped significance (p=0.08)’• ‘significant to some degree (0<p>1)’• ‘on the cusp of significance (p=0.058)’• ‘not formally significant (p=0.06)’

mchankins.wordpress.com/2013/04/21/still-not-significant-2/

Results• Don’t just report a P-value

• Report the measure of effect– treatment effect, correlation, differences, odds / hazard ratio– report absolute effects as well as relative effects (if possible)

• Provide a measure of uncertainty around the estimate (e.g. a 95% confidence interval [CI])

– if different CIs are reported (e.g. 90% or 99%), then make this clear

• Report the exact P-value– very small P-values can be reported as P<0.001– but don’t report P<0.05, P<0.01– AND avoid *, **, ***– AND avoid NS or >0.05 to denote not statistically significant– AND avoid P=0.000

How not to report p-values

Example on reporting effects*

*BMJ 2010;340:c869 doi: 10.1136/bmj.c869

Anything special?

For each dataset would you believe that the:

Mean of x is 9Variance of x is 11Mean of y is 7.5Variance of y is 4.122Correlation between x and y is 0.816Regression line: y = 3 + 0.5x

Called ‘Anscombe’s Quartet’ Illustrates the importance of showing your data

Justifying ‘negative findings’

• NEVER report post-hoc sample size calculations– they do not justify non-statistically significant findings– they do however annoy the statistical reviewer

• Don’t report non-statistically significant results as negative findings– the important thing is whether you have adequately

answered the research question– null findings are important (and need to be published)

Additional considerations

• Be aware of your area of research e.g. statistics reporting in psychology quite different to statistics reporting in medicine

• Always a good idea to check journal guidelines for how they prefer you to report the statistical aspects of your study