BMC Medical Research Methodology - Why statistical ......“Why Most Published Research Findings are False” [11], John Ioannidis sought to explain this phenomenon. He assumed the

DEBATE Open Access

Why statistical inference from clinicaltrials is likely to generate false andirreproducible resultsLeonid Hanin1,2

Abstract

One area of biomedical research where the replication crisis is most visible and consequential is clinical trials. Whydo outcomes of so many clinical trials contradict each other? Why is the effectiveness of many drugs and othermedical interventions so low? Why have prescription medications become the third leading cause of death in theUS and Europe after cardiovascular diseases and cancer? In answering these questions, the main culprits identifiedso far have been various biases and conflicts of interest in planning, execution and analysis of clinical trials as wellas reporting their outcomes. In this work, we take an in-depth look at statistical methodology used in planningclinical trials and analyzing trial data. We argue that this methodology is based on various questionable andempirically untestable assumptions, dubious approximations and arbitrary thresholds, and that it is deficient inmany other respects. The most objectionable among these assumptions is that of distributional homogeneity ofsubjects’ responses to medical interventions. We analyze this and other assumptions both theoretically andthrough clinical examples. Our main conclusion is that even a totally unbiased, perfectly randomized, reliablyblinded, and faithfully executed clinical trial may still generate false and irreproducible results. We also formulatea few recommendations for the improvement of the design and statistical methodology of clinical trials informedby our analysis.

Keywords: Clinical trial, Heterogeneity, p-value, Permutation test, Random sample, Randomization, Reproducibility,Sample size, Significance, Stochastic independence

BackgroundOver the past several decades, biomedical sciences havemade remarkable progress in understanding molecularand genomic mechanisms of life and disease. However,there remains an enormous gap between our under-standing of life’s molecular machinery and our ability toexplain behavior of living organisms as a whole and theirresponses to various interventions. In the medical arena,clinical trials aim to fill this gap using empirical means.One of the more dramatic manifestations of this gap isunexpected catastrophic events in early clinical trials.For example, a phase I trial of drug BIA 10-2474 aimedat treating anxiety, motor disorders and chronic pain

that was conducted in 2015 in Rennes, France had unex-pectedly led to the death of one volunteer and irreversiblebrain damage in five others [1]. In a phase I trial con-ducted in 2006 in London, a monoclonal antibodyTGN1412 intended for treating autoimmune diseases andleukemia had caused multiple organ failure in six healthyvolunteers [2]. As yet another example, fialuridine, anantiviral agent tested in 1993 by NIH for treatment ofhepatitis B in a phase II clinical trial, had resulted inthe death of five human volunteers due to severe hepatictoxicity and lactic acidosis [3]. Remarkably, the dose givento volunteers in the London trial was 500 times smallerthan the one found to be safe in animals [4].Insufficient knowledge of biological mechanisms and

complex interactions associated with the action of drugsand other medical interventions creates a considerableuncertainty in predicting and interpreting trial outcomes.In particular, we are still unable to predict whether a

Correspondence: [email protected] of Mathematics and Statistics, Idaho State University, 921 S. 8thAvenue, Stop 8085, Pocatello, ID 83209-8085, USA2Department of Applied Mathematics, Institute of Mathematics andMechanics, St. Petersburg Polytechnic University, Polytekhnicheskaya ul. 29,195251 St. Petersburg, Russia

© The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Hanin BMC Medical Research Methodology (2017) 17:127 DOI 10.1186/s12874-017-0399-0

http://crossmark.crossref.org/dialog/?doi=10.1186/s12874-017-0399-0&domain=pdfhttp://orcid.org/0000-0003-4842-8965mailto:[email protected]://creativecommons.org/licenses/by/4.0/http://creativecommons.org/publicdomain/zero/1.0/

tested intervention will work for an individual patient or acategory of patients and what side effects it will produce.This causes the samples of trial participants to be hetero-geneous in numerous unpredictable ways. Further, ethicalconsiderations lead to recruitment of subjects that aregenerally younger, healthier and less medicated than thetargeted population, which causes the sample of trial par-ticipants be not representative of the population. Theseuncertainties, combined with various biases and conflictsof interest involved in funding, planning, analyzing andreporting clinical trials, is a major reason behind surpris-ingly low effectiveness of many drugs and high incidenceof severe side effects. As reported in [5], ten top-sellingdrugs on the American market fail to improve the condi-tions in 75% to 96% of patients who take them. Thus, lin-gering question remain as to whether even some highlyprescribed drugs actually work. More ominously, prescrip-tion medicines have become the third leading cause ofdeath in the US and Europe after cardiovascular diseasesand cancer ([6], p. 1). Doubts about utility of clinical trialsare not limited to the medical community; they are alsocirculating in the media [7].On the academic front, the validity and reproducibility

of biomedical research including clinical trials has beena matter of grave concern for more than two decades [8, 9].As one striking example, a meta-analysis study [10] foundthat, out of 26 most highly cited reports of controlledrandomized clinical trials that appeared in top medicaljournals, claimed positive effects of medical interventionsand were later retested on larger groups of patients, 9studies (35%) were either refuted or their claims of effectwere found to be greatly exaggerated.What are the root causes of falsity and irreproducibil-

ity of biomedical research findings? In an article titled“Why Most Published Research Findings are False” [11],John Ioannidis sought to explain this phenomenon. Heassumed the following model of scientific discovery:several groups of investigators independently study anumber of research questions by performing statisticalanalysis of empirical data. Each research finding has aprior probability to be true; however, this probability ismodulated by random effects of statistical analysis withcertain rates of false positive and false negative discov-eries. A key model parameter is “bias,” defined as theprobability to report as true a finding that was foundby a research team to be false. The model leads to aBayes-type formula for the probability of false discoveryas a function of model parameters. Sample computationsbased on this formula have led the author of [11] to theconclusion encapsulated in the title of his paper.An in-depth look at various instances of deliberate bias

and institutional corruption in clinical trials conductedor sponsored by industry was taken by Peter Gøtsche inhis remarkable book [6]. It documents numerous cases

of withholding and falsifying data, selective reporting,suppressing information on adverse side effects, posthoc changes of endpoints, manipulation of patient inclu-sion criteria and study duration to achieve more favor-able outcomes, intentional handicapping of comparatorsas well as an assortment of other unethical profit-drivenactivities. This has led to numerous major public healthcalamities. As one example, concealment and fabricationof trial data on cardiovascular side effects of COX-2 in-hibitor Vioxx (rofecoxib), manufactured by Merck andmarketed primarily as a NSAID painkiller, has causedabout 120,000 deaths worldwide from 1999 to 2004 ([6],p. 161). Based on his extensive study, Gøtsche concludedthat in the hands of Big Pharma clinical trials have be-come nothing more than marketing tools in disguise.While the above-mentioned explanations are valid and

instructive, they ignore perhaps the most critical compo-nent of any research study – the scientific methodologyadopted by the research team. In this article, we focusexclusively on statistical methodology used for planningand analyzing phase III clinical trials. We show that stat-istical inference from even a totally unbiased, properlyrandomized, reliably blinded, perfectly executed andfaithfully reported controlled clinical trial is still likely togenerate false knowledge and irreproducible results. Anumber of specific problems associated with the designof clinical trials, statistical inference from trial data andassessment of trial quality were discussed in [12].As a reminder and point of reference, we review very

briefly the methodology of clinical trials, see e.g. [13].Phase III clinical trials are conducted to make large-scaleempirical comparisons of medical interventions (one ofthem normally being a placebo or standard treatment).Two arms of a trial are compared by computing the differ-ence between the average values of a certain measure ofeffect over all subjects in the respective arm of the trial. Atypical measure of effect for an individual subject is eitherthe indicator of occurrence of certain event (responseto treatment, cure, death, etc.) or the value of some ob-servable clinical variable (e.g. disease-free survival time,systolic or diastolic blood pressure, count of some typeof blood cells, concentration of certain biomarker, andso on). The normalized difference between the aboveaverages usually serves as a test statistic employed forboth trial planning and analysis of trial data.Trials are designed in such a way that the null hypoth-

esis of the treatments’ equal performance can be rejectedwith sufficiently low probability (called significance level)if they in fact perform equally and with high enoughprobability (called statistical power) if the alternative hy-pothesis that one treatment performs sufficiently betterthan the other is true. Normally, precautions againstvarious biases are taken. In particular, patients are ran-domly assigned to treatments and, when feasible, patients

Hanin BMC Medical Research Methodology (2017) 17:127 Page 2 of 12

and investigators are blinded to this assignment. After thetrial’s completion, statistical analysis of its results isconducted to determine the actual significance level andpower, estimate parameters of interest and computeconfidence intervals, assess incidence and severity ofside effects, etc.Inherent in clinical trials are four principal sources of

variation: (1) sampling of trial participants from a rele-vant population; (2) intra-subject (individual) variationin the effects of compared interventions; (3) between-subject variation in these effects; and (4) random assign-ment of trial participants to treatments. Only patientswho meet the study entry criteria and give informedconsent are enrolled in clinical trials. This and manyother reasons cause sampling of subjects from the popu-lation to be non-random. This cannot be mitigated byeven the best statistical analysis of trial data and thus isnot a subject of this study. However, non-random sam-pling represents a major obstruction to extrapolation oftrial findings to the population and their reproducibility.Variation types 2, 3 and 4 will be addressed in subse-quent sections.Our critique of statistical methods used in clinical tri-

als proceeds along two dimensions. First, we identifyseveral basic assumptions and principles underlying stat-istical methodology and argue that in the case of clinicaltrials their validity is uncertain, questionable or verylikely false. Second, we address a few important tech-nical aspects of statistical analysis commonly employedin clinical trials and conclude that they too are likelycontributors to the generation of false and irreproducibleresults. Specifically, we start with the randomness vsdeterminism dilemma vis-à-vis individual response vari-ables. Next, we discuss the averaging principle as a pref-erence rule for selection of medical interventions. Thenwe take a close look at the fundamental assumptions ofindependence and homogeneity; we do so first by meansof an example (also discussed in [14]) and then approachthis topic from a more formal standpoint of statisticalanalysis. Further, we discuss individual case studies as analternative to clinical trials. In the section StatisticalAnalysis of Clinical Trials as a Ritual, we review severalstatistical concepts and tools commonly employed in theanalysis of trial data and argue that ignoring underlyingassumptions, uncritical use of various approximationsand arbitrary thresholds, and disregardingrandomization may lead to false results. Finally, wesummarize our findings, frame them in historical andphilosophical perspectives, and formulate our conclu-sions and specific recommendations.The presence of serious deficiencies in statistical meth-

odology utilized in clinical trials does not mean thatclinical trials should be abandoned. When carefullyplanned and properly conducted, they can produce a

wealth of empirical knowledge about the disease and pa-tients’ responses to the compared treatments. This mayprove especially valuable when the disease and/or theeffects or side effects of the tested interventions are veryheterogeneous. However, for trials to be effective andinference from their results valid, statistical methodologyshould be considerably tightened, fortified with rigorousmathematical and computational sensitivity analyses,and combined with biomedical knowledge of the diseaseas well as biological and/or pharmacological action ofthe compared treatments.

Individual response to treatment: Thedeterminism vs chance dilemmaNowadays, statistical analysis has become a mandatorycomponent of biomedical research. This compliance pres-sure has caused biomedical scientists to adopt, mostly un-wittingly, the assumption that every health-related eventoccurring in a given subject depends in essential ways onchance and that every measurable quantity is a randomvariable. That this assumption is not merely philosophicalis clear: detection of the occurrence of a non-randomevent over the duration of a study is a matter of singleobservation while a random event occurs with certainprobability whose estimation requires a large number ofobservations. It is an empirical fact that responses of dif-ferent subjects to the same treatment display a wide vari-ation. However, how strong is scientific evidence for theinvolvement of chance in individual responses?The deterministic side of the dilemma enjoys a strong

backing from basic science. The effects of drugs andother medical interventions typically manifest throughthe action of various biochemical systems that constitutethe molecular basis of life. What we know about theirfunctioning suggests that they essentially act as de-terministic machines governed by differential equationsof biochemical kinetics. If the initial concentrations ofall molecular species, kinetic constants, and various ex-ternal and internal conditions are known, then the fu-ture states of a biochemical system can be predictedwith great accuracy. Such a predictable operation withina wide range of internal and external conditions is thereason why the genetic apparatus of a cell, which isnothing more than an extremely complex, self-regulatingbiochemical system, displays such a great fidelity in pres-ervation and replication of the genome as well as intranscription, translation, and adaptive regulation ofgene expression. Additional features of biochemical sys-tems, such as activation thresholds and the presence ofinhibitors and feedback loops, ensure stable execution ofbiological functions even in randomly changing environ-ments, which contributes to physiologic homeostasis.The deterministic behavior of biochemical systems is a

collective result of a very large number of random


microevents governed by stochastic laws of quantummechanics. Therefore, before embarking on statisticalanalysis of individual response data one has to envisiona mechanism by which stochasticity re-emerges, in theparticular setting at hand, at the level of whole-bodyclinical effects. Or is this merely an illusion that masksunknown mechanisms of deterministic causation?The hypothesis of deterministic response can some-

times be tested empirically. For example, if repeatedoccurrences of an acute illness in a given subject arecured by the same dose of a drug then the patient’s re-sponse to the drug is likely to be deterministic. Likewise,our casual experience with various drugs and medicalprocedures suggests that many of them have a stable,well-defined patient-specific effect in terms of both mag-nitude and timing. Why couldn’t this, then, be the casefor an experimental drug tested in a clinical trial?On the other hand, individual effects of certain treat-

ments are undoubtedly stochastic. A classic example isexposure to radiation that can cause observable effectsthrough cell killing, mutagenesis and carcinogenesis.Here, lethal damage to a cell or a harmful mutation canresult from a random amount of energy deposited by asingle particle of ionizing radiation, or one of its secon-dary particles, if the particle track happens to pass closeenough to the cell’s DNA.The choice between deterministic and stochastic

approaches to mathematical or statistical modeling ofindividual effects of treatment should be deliberateand follow the preponderance of biomedical evidenceregarding the nature of the disease and treatment. Ifsuch evidence is inconclusive then both approachescan be pursued competitively and the resultscompared.

The methodology of clinical trials: Does averagingwork?The core methodological idea in clinical trials is thecomparison of averages. The power of averaging lies inthe combination of (1) essential cancellation of randomindividual variation and (2) retention of systematic clin-ical effects assumed to result from medical interventions.This idea, however, hinges on a hidden assumption ofhomogeneity, i.e. that the magnitude of the mean indi-vidual responses to the assigned treatment is about thesame for most patients. The reality of most clinical trials,however, is very different. Typically, a sizeable fractionof subjects enrolled in a trial does not respond to theassigned treatment while responses of other subjectsdisplay a large variation in the magnitude and timing ofthe effect. Additionally, a wide variety of side effects ran-ging from minor and transient to permanent and life-threatening are observed.

The seemingly appealing idea that the best interven-tion is the one that works best on the average may betrue in the case of homogeneous responses. However, asa general comparison principle, it represents a funda-mental fallacy. What may appear best on the averagemay not be the best intervention even for a single pa-tient in the population of interest. As a simple schematicexample, suppose there are three competing drugs, A, Band C, compared on a population of patients. Let the ef-ficacy of drug A be 2 units on a certain scale on one halfof the population and 0 on the other half, and let drug Bhave efficacy of 2 on the latter half of the populationand 0 on the former. Suppose drug C has efficacy of 1.1across the board. Then drug C is superior to A and B onthe average but for each particular patient it is almosttwice less effective than the better of the drugs A or B!To compare treatments based on their average re-

sponses, trialists have to minimize heterogeneity of theanticipated arm-specific individual responses at the plan-ning stage of the trial. To this end, they are advised touse all available prior biomedical information about thetargeted disease and mechanisms of drug action andadopt strict study entrance criteria.

The heterogeneity curse: An exampleTo see how unreasonable is our reliance on the homoge-neity of responses to a given treatment in clinical trials,consider a hypothetical randomized and appropriatelyblinded clinical trial that compares survival or metastasis-free survival of stage I-III breast cancer patients undertwo treatment plans involving surgery and various com-binations and regimens of adjuvant cytotoxic chemother-apy, external beam radiation and hormonal therapy. Whatare the leading factors that determine individual survivaloutcomes? The single most important among them is thepresence or absence of subclinical metastases at the timeof surgery. Such metastases may be in three distinct states:(1) solitary cancer cells that were released into the blood-stream during surgery or were already present at the timeof surgery as circulating tumor cells or quiescent cancercells lodged at various secondary sites; (2) dormant orslowly growing avascular micrometastases; and (3) aggres-sively growing vascular secondary tumors that at the timeof diagnosis have not yet reached detectable size. If a pa-tient was metastasis-free at surgery then, barring primarytumor recurrence, she will be cured. If only state 1 and 2metastases were present immediately after surgery thenthe outcome depends critically on how long the state ofmetastatic dormancy will be maintained. (For an extensivediscussion of the significance of metastatic dormancy inbreast cancer, see [15]; a quantitative assessment of thecontribution of the above-defined states 1-3 of the meta-static cascade to the timing of metastatic relapse based on


a mathematical model applied to real data can be found in[16]). Whether or not metastases will escape from dor-mancy in a particular patient depends not only on the ef-fects of treatment, functioning of the immune system,concentrations of circulating angiogenesis promoters andinhibitors, and other internal factors; exacerbation of thedisease may also be triggered by intercurrent sporadic ex-ternal events such as surgery unrelated to breast cancer,infection, trauma, radiation, stress, etc. Another highlysignificant prognostic factor is the intrinsic aggressivenessof the disease; however, its reliable assessment at earlystages of the disease has proven so far to be elusive. Thus,the most critical determinants of the trial outcome arelargely unobservable and/or unpredictable.In practice, the above unobservable prognostic factors

are substituted with less informative observable surro-gates such as (1) age at trial entry; (2) stage and histo-logical grade of the disease at surgery; (3) localizationand size of the primary tumor; (4) whether or not thetumor invaded surrounding tissues; (5) the extent ofnodal involvement; (6) menopausal status; (7) estrogenand progesterone receptor status; (8) presence of specificmutations in BRCA1 or BRCA2 genes; (9) family historyof breast cancer; and (10) individual history of othermalignancies. Even this rough and incomplete set of sur-rogate clinical variables creates a large number of cat-egories of women in both arms of the trial withpotentially very different characteristics of survival andmetastasis-free survival. Importantly, randomizationwon’t eliminate the observable and hidden heterogeneity;it will only reduce the difference in the extent of hetero-geneity between the treatment and control arms.The aforementioned inter-subject heterogeneity is

quite typical of clinical trials (as opposed to in vitro ex-periments with cell lines or studies on animal modelswith tightly controlled inter-subject variation). Thus, in-dividual responses of subjects in both arms of a trialcannot even approximately be viewed as homogeneous,let alone distributionally identical.

Statistical inference from clinical trials: Are theassumptions met?Like all mathematical sciences, theoretical statistics isbased on theorems consisting of assumptions and con-clusions. The validity of the arguments by which theconclusions are derived from the assumptions is therefor anyone to verify. In applied statistics, including infer-ence from clinical trials, statistical methods and testsresulting from these theorems are employed to generatenew knowledge based on empirical data. But are the as-sumptions behind the methods and tests valid?The most fundamental assumption that underlies vir-

tually every application of statistics is that the set of ob-servations, say x1, x2, …, xn, is a random sample from a

certain probability distribution. Informally, this meansthat the observed values result from independent repli-cations of the same random experiment, just like a se-quence of “heads” and “tails” results from flips of a coinor numbers 1-6 result from repeated rolls of a die. Theexact meaning of the “random sample” assumption is asfollows: there exist a sample space S with a probabilitymeasure on it and jointly stochastically independent (i)and identically distributed (id) random variables X1, X2,…, Xn on S such that X1(s) = x1, X2(s) = x2, …, Xn(s) = xnfor some point s in S. Critically, the iid assumption can-not be verified empirically, for each of these random var-iables is represented in the data set by a single value.The hypothesis that random variables X1, X2, …, Xn havea given distribution (say, standard Gaussian) can only betested, with certain probability of error, under the iid as-sumption. Beyond this premise, most statistical methods,tests and tools fail; even measures as simple as the sam-ple mean lose their inferential significance. Thus, themost basic assumption underlying statistical inferencefrom data is necessarily and invariably taken on faith.How strong is our faith in the iid hypothesis in the

case of clinical trials? Turning to the independenceproperty first, note that selection of trial participants isassociated with clinical characteristics of their diseaseand other medical conditions and thereby constitutes asystematic source of dependence between individual re-sponse variables. The latter may also be induced by apre-randomization run-in period during which all partic-ipants receive the same treatment [12]. Another factor isthe significance of family history for the incidence ofvarious health-related events, both sporadic and trig-gered by a medical intervention. For identical twins, theoccurrence of such an event in one of them typicallysharply increases the probability of the same event hap-pening to the other twin, thus making these eventshighly dependent. The same effect, albeit possibly to alesser extent, is often observed in siblings and other rela-tives. Furthermore, because in a relatively homogeneoushuman population it is very likely that two given mem-bers have a common ancestor, their disease states andresponses to treatment should, at least in principle, beviewed as dependent random variables. Finally, sto-chastic dependence between individual responses mayresult from various post-randomization events such asexchange of information between subjects participatingin a clinical trial, which may modify the placebo effectand lead to partial unblinding of the study.One may argue, of course, that the aforementioned

dependence is weak and therefore negligible. Toanalyze claims about the strength of dependence, thelatter has to be quantified. A simple and almost uni-versally used measure of dependence between two ran-dom variables is their correlation coefficient. This


measure, however, is inadequate because (1) zero cor-relation does not imply independence; and (2) a collec-tion of pairwise independent random variables may notbe jointly independent. Thus, even designing a practic-able quantitative measure of the deviation from jointindependence represents a considerable challenge.The central question in assessing the effects of sto-

chastic dependence between individual response vari-ables is how much the deviation from independencemodifies the distribution of the test statistic under thenull (no relative effect of intervention) and alternative(relative effect exceeding a given threshold) hypotheses.The distributions of the test statistics typically used inparametric analyses of clinical trial data (such as stand-ard Gaussian, Student’s t and chi-squared) depend invery essential ways on the independence postulate. Ab-sent this assumption, one cannot rely on these standarddistributions anymore. An equally important questionis how to assess the impact of misspecification of thedistribution of the test statistic on the outcome of stat-istical analysis (such as the p-value of the test statisticunder the null hypothesis, statistical power of the trial,sample size, estimates and confidence intervals for pa-rameters of interest, trial stopping time, etc.).As we have argued in the previous section, along

with the possible lack of independence, statistical ana-lysis of clinical trial data is bound to encounter an evenmore consequential violation of the equidistribution(id) assumption. As with stochastic dependence, to as-sess the extent of the deviation from the id assumptionand its impact on the outcome of statistical analysis oftrial data quantitatively, one has to use certain distance,d, between probability distributions. This can be doneby employing one of the well-known probability metricssuch as the total variation, Kantorovich, Kolmogorov-Smirnov, Cramér-von Mises, Lévy and other distances[17].We emphasize once again that neither the absolute

value, r, of the correlation coefficient for a pair of indi-vidual observations nor the distance, d, between theirdistributions is estimable from the observations alone.Suppose, however, for the purpose of our argument,that we know the values of r and d for all pairs ofunderlying random variables exactly and that they aresmall, say, less than some positive number ε. Let y bean output of statistical analysis. Computation of y isbased on the known distribution, P0, of a test statisticunder the null or alternative hypothesis provided theiid assumption is met. Let P be the “true” distributionof the same statistic under the hypothesis in questionwithout the iid assumption. How much does P deviatefrom the “ideal” distribution P0? It can be envisionedthat in some cases distribution P0 is robust, i.e. the dis-tance d(P, P0) will be small for small ε, while in other

cases d(P, P0) will be found to be large regardless ofhow small ε is. Similarly, the dependence of the outputy on the “ideal” distribution P0 may be robust to theperturbations of the latter or not at all. Finally, evenunder the total robustness scenario, the utility of such asensitivity analysis depends on availability of tight andrelatively simple estimates for the deviation of the out-put y as a function of ε. Obtaining such estimates inmost cases goes far beyond the reach of contemporaryprobability theory and statistics.The patients’ hidden and observable clinical variables

associated with the disease, responses to the comparedinterventions and susceptibility to the placebo effectpartition the queried population into a large number ofcategories with distinct distributional characteristics ofthe response. Even if we assume that each category isdistributionally homogeneous, both trial arms will con-tain a large yet unknown number of categories, eachcontaining an unknown number of subjects. Moreover,the number of such categories in each arm and thenumbers of their representatives are dependent randomvariables with unknown distributions. The unknownpopulation weights of these categories are nuisance var-iables that will confound statistical analysis of trial out-comes. Furthermore, variation in the number ofcategories and subjects representing each categorybetween different trials will make their results poten-tially irreproducible.Theoretically, greater distributional homogeneity of

responses in a clinical trial could be achieved throughstratification with respect to observable clinical variables.However, two impediments will most likely underminethe feasibility of this approach. First, due to the largenumber of strata the requisite sample size in many in-dividual strata will be unachievable, thus making the trialunderpowered. Second, the presence of substantialhidden variation of the type discussed in the abovebreast cancer example will still leave individual strataheterogeneous.In summary, even for large-scale randomized con-

trolled clinical trials commonly viewed as the goldstandard of biomedical research, the “heterogeneitycurse” will likely make the results of statistical inferencefrom the collected data dubious and potentially lead tofalse and irreproducible conclusions.

Optimal sample size? Try n = 1!One of the primary design parameters in a clinical trialis sample size. Large sample size is supposed to ensurestatistical power of the study when the knowledge ofcauses and mechanisms of the underlying biomedicalprocesses and effects of the compared interventions isinsufficient for outcome prediction. As discussed above,such lack of knowledge likely means that, in spite of all


the effort, the queried population is still heterogeneousand so is the sample of trial participants randomizedbetween two or more arms of the study.There is one case, however, where the heterogeneity

and independence problems are non-existent, namelywhen a sample consists of just one subject. A great ad-vantage of individual case studies is that learning every-thing that is there to know is quite feasible and that inthis case inference from the generated data is not con-founded by inter-subject variation. If one believes thatbiomedical processes are governed by natural laws andhave causes, mechanisms and effects, then studying asingle subject thoroughly should be very informative. Itcan be expected that doing so for many individual pa-tients will eventually reveal all major types and charac-teristics of the disease and will enable evaluation, andeven prediction, of effects, and side effects, of variousinterventions. However, in contrast to clinical trials, in-dividual case studies is an open-ended process with un-certain inferential value, which makes them unfit formaking expeditious public health decisions regardingmedical interventions.Since time immemorial medical doctors used the

method of trial and error to find effective individualtreatments while minimizing the harm to the patient’shealth. In the cases where the attempted ineffectivetreatments did not change significantly the natural his-tory of the disease, the patients served as best-matchingself-controls. Multiple individual case studies, especiallythose involving controls, is the way medicine accu-mulated enormous empirical knowledge. Over the lasttwo centuries, this process has been greatly acceleratedby the advancement of basic biomedical sciences, andthere is no reason to believe that it won’t bear fruit inthe future. Focused on individual rather than populationdimension of medicine, individual case studies representa natural complement to clinical trials. Thus, beforestarting a clinical trial on 1000 patients, it is reasonableto ask if it would be more beneficial to science andhealth care (as well as more cost-effective) to conduct amore sophisticated, state-of-the-art individual case-controlled study on 100 subjects randomly selected froma larger pool of qualifying and consenting patients.

Statistical analysis of clinical trials as a ritualDistributional heterogeneity of responses within thequeried population and potential lack of independenceare by no means the only factors that may call intoquestion the results of statistical inference from trialdata. Statistical analysis of clinical trials involves awhole host of hidden and untestable assumptions, va-rious approximations and arbitrarily selected thresholdsdiscussed below. They all require careful justificationand thorough theoretical, or at least numerical,

sensitivity analysis. Without this, statistical inferencefrom clinical trials would essentially be a ritual thatlacks rigorous scientific underpinnings and may havedisastrous effects on public health.

The mantra of large n and the invocation of normalityThe distributions of the test statistics most widely usedin the analysis of clinical trials are the standard Gaussian(normal) and closely related distributions (such as χ2

and Student’s t). Strictly speaking, their legitimacy iscontingent on the assumption that individual responsevariables are iid with normal distribution. Recall that thenormality assumption is empirically untestable withoutthe iid hypothesis that, as we have argued above, can byno means be taken for granted. The iid assumption isalso required if one employs asymptotic results such asthe Central Limit Theorem that allows one to concludethat for large sample size the distribution of the samplemean of the response variables over a trial arm is ap-proximately normal. An important question is how largethe sample size should be for the true finite sample dis-tribution be sufficiently close to the asymptotic distribu-tion. Estimates of various distances between thesedistributions as function of sample size are extremelyhard to obtain. Even for the Central Limit Theorem,only a basic estimate of the Kolmogorov-Smirnov dis-tance given by the Berry-Esseen theorem [18] is avai-lable. Yet another difficult question concerns the effectsof such estimates on the accuracy of the outcomes ofstatistical analysis.Additional challenge to the normality assumption

comes from the obvious fact that individual responsevariables, and hence their sample means, are boundedabove and below, so that their true distributions arealways confined to a finite interval; in particular, theycan never be exactly normal, a point eloquently made in[12]. The correction to the assumed asymptotic distri-bution with an infinite tail that arises from such a trun-cation, as measured by some probability metric, may besmall; however, the downstream effects of this error onthe distribution of the test statistic and outcomes of sta-tistical analysis may be significant.

Sample size: Fixed or random?Although statistical analysis of trial data assumes fixedsample size, it is often applied to sample size that is inreality random. Variation of the sample size has severalsources. One is randomization of patients between thetrial arms; here, sample size variation in each arm maybe substantial unless block randomization or more ad-vanced schemes [19, 20] are employed. Random samplesize also arises in trials that require a fixed number ofevents of interest. Finally, many patients drop out of a


trial due to the lack of benefit, severe side effects orother reasons, which leads to a difficult dilemma: eitherto perform “intent-to-treat” analysis with fixed samplesize under intractable informative censoring or to dealwith a random number of patients followed throughthe entire study duration. Statistical methods intendedfor fixed sample size lead to erroneous results if appliedto samples of random size. For example, even the Cen-tral Limit Theorem that underpins many statisticalmethods fails beyond a few special cases, e.g. when thesample size has Poisson distribution ([21], p. 4699).

The idol of statistical significanceStatistical significance emerged as a means to accountfor random variation in the context of hypothesis test-ing. Let, for example, δ be the observed value of the dif-ference Δ = A1 –A0 between the average measures ofeffect for experimental and control arms of a clinicaltrial. Is the observed relative effect due to the true dif-ference between the compared interventions or tochance, or perhaps both? A reasonable question to ask,then, in order to distinguish between these two pos-sibilities, is as follows: What is the probability to ob-serve the value δ under the (strong) null hypothesisthat, for each trial participant, the effects of the com-pared treatments are identical? In other words, what isthe probability that Δ = δ due to chance alone? If thedistribution, P, of the statistic Δ under the null hypoth-esis were discrete, then the required probability wouldbe P(δ). The problem with this answer, still somewhatpopular among natural scientists, is that it does notprovide a clear way to separate two interrelated factors:(1) the magnitude of P(δ) relative to the probabilitiesP(x) of other admissible observations x; and (2) thesample size n. (Note that as n increases all the probabi-lities P(x) tend to become small). In the opposite caseof a continuous distribution P, the proposed answer isutterly uninformative, for in this case P(x) = 0 for anyobservation x.A way to resolve this conundrum was proposed by

Sir Ronald Fisher in his famous book [22]. To quantifysignificance of an observation δ, he suggested to usethe probability, under the null hypothesis, that Δ ≥ δ ifδ > 0 and Δ ≤ δ if δ < 0 (or the corresponding two-tailprobability if the sign of Δ is of no particular impor-tance). This probability, termed the p-value, representsthe asymptotic fraction of hypothetical independentidentical trials, if one conducts them indefinitely, inwhich the size of the observed effect will be at least asextreme as that in the given trial. Thus, sufficientlysmall p-values can be used for rejecting the nullhypothesis.Fisher’s approach to significance is not without a

blemish. First, it employs, contrary to the empirical

nature of biomedical sciences, the values of statistic Δthat were not observed in a given study and perhapswill never be, even if the study were to be replicated in-definitely; the sole basis for these counterfactual valuesof Δ is its imputed distribution under the null hypoth-esis. Second, Fisher’s idea is based on a tacit assump-tion that the probability density function of statistic Δunder the null hypothesis has a one- or two-sided bell-shaped tail. For other shapes, it may lose its appeal(think, for example, of Δ uniformly distributed on asymmetric interval whose endpoints represent realisticbounds for Δ). Furthermore, if the null distribution ofΔ is multimodal then the blanket definition of p-valueas tail probability is unequivocally wrong.Normal distribution of Δ, almost universally assumed

in the parametric analyses of clinical trials, is merely anapproximation, based on the Central Limit Theorem, toits true distribution (or perhaps not even an approxi-mation if individual response variables are not iid). Be-cause p-value is a tail probability, the resulting error inits determination may be as large as the Kolmogorov-Smirnov distance between the two distributions. Thelatter can be estimated through the Kolmogorov-Smirnov distances between the distributions of theaverages A1, A0 over trial arms and their normal ap-proximations. Under the iid assumption, each of thesedistances, according to the Berry-Esseen theorem [18],does not exceed 0.5Cn-1/2, where n is the sample sizeand C ≥ 1 is the ratio of the third absolute central mo-ment of the distribution of individual response variablesto the cube of its standard deviation. Importantly, thedependence of the Berry-Esseen bound on n cannot beimproved; specifically, with the upper bound of 0.4Cn-1/2,it is in general not true anymore [23]. Therefore, forsample sizes typically encountered in clinical trials(from a few hundred to a few thousand subjects), themaximum error in p-value determination may becomparable to, or even exceed, the small p-valuesused for rejecting the null hypothesis. Such samplesizes can only guarantee the correctness of the firstdecimal digit of the p-value! Thus, pursuit of small p-values in parametric analysis of clinical trials is in-defensible. A way to somewhat mitigate this problemis discussed next.

Randomization ignoredThe above-described parametric p-values ignorerandomization, an essential aspect of the design of ran-domized controlled clinical trials [12]. Suppose that p-values are computed under the (strong) null hypothesis.Then, for a properly blinded trial, it is reasonable to ex-pect the responses of all trial participants to be exactlythe same regardless of their allocation to treatment arms(this claim is unequivocally true if individual responses


are deterministic). This enables computation of the stat-istic Δ for every possible allocation of subjects to treat-ments that the employed randomization algorithm mayproduce, not only the “real” one. This leads to thepermutation-based p-value of the observation δ [24, 25].If individual responses are deterministic (in which casestochastic dependence and heterogeneity are non-issues), this is the only way to compute the effective sig-nificance of the trial outcome. However, in general thesep-values are only partial in that they do not account forthe variation of individual response variables. This iswhere parametric analysis, that capitalizes on the asymp-totic normality of Δ, could be invoked. (Observe thatbecause the generalized Berry-Essen inequality [26] re-quires only independence of individual response vari-ables, the error in p-value determination can inprinciple be controlled even if the assumption of distri-butional homogeneity is lifted). The average of para-metric p-values computed over all admissible outcomesof the randomization process represents a permutation-based parametric p-value that will most likely reducethe error in the conventional parametric p-valuecomputation.

The magic 5% and other arbitrary thresholdsIn the aforementioned book [22], Ronald Fisher alsoproposed to use 0.05 as a p-value threshold for rejectingthe null hypothesis. Since then this low key suggestionhas become almost a religious commandment for adop-tion of statistical significance levels and computation ofconfidence intervals. For example, a recent massivemeta-analysis study [27] found that among almost 2 mil-lion biomedical papers published over the last 25 years,96% appealed to p-value ≤ 0.05 to claim significance oftheir results. As discussed above, numerous factors maylead to misspecification of the test statistic under thenull hypothesis, which may have a considerable impacton the effective significance level of a clinical trial. As aresult, a trial may produce false results even if the p-value happens to be < 0.05 and, conversely, true andvaluable results may be discarded or self-censored justbecause their statistical significance falls short of themagic 5%. The same is true for the deviation of theeffective statistical power of clinical trials under the al-ternative hypothesis from the nominal value, typicallyassumed at the planning stage to be 80 or 90%.

Discussion, conclusions and recommendationsThat logic, including careful formulation of premises, iscritical for the correctness of all sorts of arguments, hasbeen recognized since Aristotle. However, it was HenriPoincaré, a genius French mathematician, theoretical

physicist and philosopher of the 19th and early twenti-eth century, who was the first to keenly understand thefundamental importance of hypotheses and assum-ptions for the validity of scientific research [28]. Appa-rently, his ideas appeared so much ahead of their timethat even more than a century later they have not beenfully recognized and taken to heart by the scientificcommunity.In this work, we focused on the following basic as-

sumptions that underpin statistical methodology used inclinical trials and are critical to the validity of statisticalinference from trial data: (1) substantial role of intra-subject variation in individual responses to the com-pared interventions; (2) stochastic independence of indi-vidual response variables; and (3) their distributionalhomogeneity within each trial arm. We found that formany health conditions and treatments assumption 1 isunlikely to be true; assumption 2 is possibly approxi-mately true with some exceptions and in general re-quires careful analysis in each particular case; andassumption 3 is likely to be false. The last point has twoimportant implications.

(a)The average response over a trial arm may prove tobe a poor preference function for selection of thebest intervention and likewise the normalizeddifference between arm-specific averages may appearto be a suboptimal test statistic under the null andalternative hypotheses. As a historical note, the ideathat indiscriminant use of averages in biology andmedicine may lead to obfuscation of scientific truthwas passionately argued 150 years ago by ClaudeBernard, one of the greatest experimentalphysiologists of all times. He also insisted that theduty of a scientist is to find the unique immediatecause behind every health-related event in anindividual patient, thus siding unequivocallywith the deterministic paradigm ([29], p. 137).

(b)Under the assumption of stochastic individualresponses, a key condition that makes or breaksstatistical analysis of clinical trial data is thedistributional homogeneity of individual responsevariables. On the one end of the homogeneityspectrum, one encounters the situation whereindividual response variables in each arm of the trialare iid. Here comparison of treatments by the valueof arm-specific averages and inference from theirdifference, when made correctly and rigorously, iswell justified. Achieving a larger degree ofhomogeneity requires tightening of the subjectrecruitment criteria based on observable clinicalvariables including genomic, molecular, cellular,histologic and other markers of the disease andresponses to compared treatments. Nonetheless, it is


quite possible that in spite of all the effort, responsesof trial participants are still extremely heterogeneous.There we meet the other end of the homogeneityspectrum where all individual responses have verydissimilar distributions. Here each individual subjectrepresents a separate clinical case while populationapproach has only descriptive utility. Eachcombination of a disease, medical intervention andtargeted population falls somewhere in betweenthese extremes but its position within thehomogeneity spectrum is difficult to determine. Apractical way to address this uncertainty wouldbe to conduct clinical trials and controlled individualcase studies competitively and compare theiroutcomes.

From its early beginnings medicine has been “person-alized” in that physicians were primarily focused ontreating individual patients, one at a time, and their in-terventions were tailored to the unique patient’s condi-tion and specific course of the disease. The advent ofclinical trials with its focus on average population ef-fects signified a dramatic departure from this paradigm.One conclusion of this work is that perhaps the timehas come to expand the approach of personalized medi-cine from treatment to drug development and othermedical innovations. Due to scientific and technologicalbreakthroughs in molecular biology and genomics com-bined with increased computational power and moderninformation technologies, such expansion may bringabout greater effectiveness and prognostic accuracy ofmedical treatments than in the past, see e.g. [5].In this work, we have also taken a close look at statis-

tical significance invoked for supporting the claim ofsuperiority of a tested intervention over a comparator.Many factors, such as (1) violation of the iid assump-tion for individual responses; (2) deviation from thenormality of the individual responses or their arm-specific averages; (3) randomness of the sample size;and (4) failure to take into account randomization oftrial participants between treatments, may cause the p-value computed for the postulated distribution of thetest statistic to diverge considerably from the true sig-nificance. For example, as we have argued in the previ-ous section, to guarantee the nominal significance α =0.05 (after rounding) in parametric analysis, a trial hasto be run with tens of thousands subjects! This casts aserious doubt on the utility of parametric p-values forplanning and analyzing clinical trials, suggests thatsmall parametric p-values are likely meaningless, andimplies that reliance on fixed thresholds (typically 0.05)for rejection of the null hypothesis is scientifically un-founded. The last point is also true for statistical power.Finally, the conventional sample size computation at

the planning stage of a clinical trial may also lead to er-roneous results.To make things even worse, in the vast majority of

biomedical studies (including reports on clinical trials),p-values are deployed without even defining the measureof effect, stating the null hypothesis or specifying the teststatistic, let alone verifying the assumptions under whichthe test statistic has a postulated distribution. As a re-sult, systematic misuse, overuse and misinterpretation ofp-values has become a major source of false and irrepro-ducible results. Although these abuses have been exten-sively criticized [30–33], p-values remain the single mostimportant numerical measure invoked to analyze the re-sults of clinical trials, confirm the validity of biomedicalresearch, and make critical health care decisions. Yet alltoo often they provide a convenient cover for poor dataquality, all sorts of biases and conflicts of interest per-taining to the collection, analysis and reporting of clin-ical trial data, and for outright fraud.Throughout history the practice of medicine was

rooted in tradition, authoritative opinion, personal ex-perience, and clinical intuition. Clinical trials emergedas an attempt at a more “objective” and “evidence-based” approach. If a treatment has invariably large ef-fect then a small controlled trial would give a definitiveanswer even without formal statistical analysis. A text-book example is the finding that eating citrus fruits ordrinking their juice cures scurvy, a discovery made in1747, long before it was found that scurvy is caused byvitamin C deficiency, by Royal Navy surgeon JamesLind through the first controlled trial in history. As SirAustin Bradford Hill has forcefully stated in his classicwork [34], in cases where the observed effect is uni-formly very large, very small or practically inconse-quential, formal statistical analysis is unnecessary. It isthe case of heterogeneous effects of small or variablesize that calls for carefully designed large-scale con-trolled randomized clinical trials and rigorous statisticalanalysis of their outcomes. Paradoxically, as we haveargued in this work, this is precisely the situation wheredistributional heterogeneity of individual responses andnumerous other factors may invalidate the basic as-sumptions upon which statistical analysis of clinicaltrial data rests and result in false and irreproducibleconclusions.What should be done to restore the value and integrity

of statistical methods in clinical trials and beyond? I be-lieve the answer lies in (1) resisting statistical orthodoxyand creatively using a multitude of statistical methods(while occasionally openly admitting that these methodshave failed); (2) rigorously validating all the assumptionsunderlying statistical analysis; and (3) closely coordinat-ing statistical analyses with biomedical research, whichprovides statistical methods with both context and


means of external validation. As an example of howmuch enrichment, innovation and modification biomed-ical research may bring to statistical methodology, thereader is referred to the article [35], which represents atwenty-first century version of the Bradford Hill’s 9 princi-ples of inference from association to causation in epi-demiological studies that he has formulated in 1965 [34].Each clinical trial should be treated as a unique scientificproject that brings the full arsenal of knowledge of mecha-nisms associated with the disease and its treatments tobear on selection, or invention, of statistical methods.Meanwhile, advanced mathematical and computationalmethods could be used to validate the assumptionsbehind statistical methods and estimate errors result-ing from various approximations.We conclude with a few specific recommendations in-

formed by the analysis undertaken in this work.

1. Clinical trials should be publicly funded andconducted by biomedical researchers, medicaldoctors and statisticians with no relation toindustry and no conflicts of interest.

2. Health care decisions based on outcomes of clinicaltrials should rely on a combination of statistical andbiomedical evidence.

3. Scientific and health care benefits resulting fromclinical trials should be compared to those of state-of-the-art controlled individual case studiesincurring comparable costs.

4. Trials should be populated in such a way that theanticipated individual responses in all arms of thetrial are as homogeneous as possible given all theavailable prior information.

5. Results of statistical analyses of randomized clinicaltrial data should be compared with those based ondeterministic individual responses and permutation-based p-values, unless there is strong scientificevidence that individual responses are stochastic.

6. The use of fixed levels of significance and statisticalpower as well as pursuit of small p-values inparametric analyses of trial data should be discouraged.

7. Computation of parametric p-values for randomizedclinical trial data should involve averaging over theset of permutations produced by the randomizationalgorithm.

AcknowledgementsThe author is grateful to the handling Editor and the two Reviewers for theirthoughtful and constructive comments. The author is obliged to Drs AlexanderGordon, Boris Hanin and Allen Roginsky for their insightful remarks on an earlierversion of the manuscript. The author also thanks Dr. Mark Hanin, JD, for hishelpful suggestions on the substance and style of the article.

FundingNot applicable.

Availability of data and materialsNot applicable.

Authors’ contributionsLH is responsible for all components of the manuscript and all aspects of itspreparation.

Authors’ informationLH is a rigorously trained Ph.D. mathematician who has more than 30 yearsof experience working in statistics, biostatistics and various biomedical sciences,mostly cancer biology, cancer epidemiology, radiation biology and radiationoncology.

Ethics approval and consent to participateNot applicable.

Consent for publicationNot applicable.

Competing interestsThe author declares that he has no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Received: 25 March 2017 Accepted: 2 August 2017

References1. Butler D, Callaway E. Scientists in the dark after French clinical trial proves

fatal. Nature. 2016;529:263–4.2. Wadman M. London’s disastrous drug trial has serious side effects for

research. Nature. 2006;440:388–9.3. Honkoop P, Scholte HR, de Man RA, Schalm SW. Mitochondrial injury.

Lessons from the fialuridine trial. Drug Saf. 1997;17:1–7.4. Attarwala H. TGN1412: from discovery to disaster. J Young Pharm.

2010;2(3):332–6.5. Schork N. Time for one-patient trials. Nature. 2015;520:609–11.6. Gøetsche P. Deadly medicines and organised crime: how big Pharma has

corrupted healthcare. London: Radcliffe Publishing; 2013.7. Leaf C. Do clinical trials work? New York: The New York Times; July 13, 2013.8. Altman DG. The scandal of poor medical research. BMJ. 1994;308(6924):283–4.9. Horton R. Offline: what is medicine’s 5 sigma? Lancet. 2015;385:1380.10. Ioannidis JPA. Contradicted and initially stronger effects in highly cited

clinical research. JAMA. 2005;294(2):218–28.11. Ioannidis JPA. Why most published research findings are false. PLoS Med.

2005;2(8):e124.12. Berger VW. Conflicts of interest, selective inertia, and research malpractice in

randomized clinical trials: an unholy trinity. Sci Eng Ethics. 2015;21(4):857–74.13. Friedman LM, Furberg CD, DeMets DL, Reboussin DM, Granger CB.

Fundamentals of clinical trials. 5th ed. New York: Springer; 2015.14. Hanin L. Do breast cancer patients benefit from surgery? Hypotheses,

mathematical models and false beliefs, in: Perioperative inflammation as atriggering origin of metastasis development (Retsky M and Demicheli R,eds). New York: Nature/Springer; 2017. p. 161-82.

15. Demicheli R, Retsky MW, Swartzendruber DE, Bonadonna G. Proposal for anew model of breast cancer metastatic development. Ann Oncol.1997;8:1075–80.

16. Hanin L, Pavlova L. A quantitative insight into metastatic relapse of breastcancer. J Theor Biol. 2016;394:172–81.

17. Rachev ST, Klebanov L, Stoyanov SV, Fabozzi F. The methods of distances inthe theory of probability and statistics. New York: Springer; 2013.

18. Durrett R. Probability: theory and examples. Wadsworth & Brooks/Cole:Pacific Grove; 1991.

19. Soares JF, Wu CFJ. Some restricted randomization rules in sequentialdesigns. Comm Stat Theory Methods. 1982;12:2017–34.

20. Berger VW, Ivanova A, Deloria-Knoll M. Minimizing predictability whileretaining balance through the use of less restrictive randomizationprocedures. Stat Med. 2003;22(19):3017–28.


21. Hanin L, Zaider M. Cell-survival probability at large doses: an alternative tothe linear-quadratic model. Phys Med Biol. 2010;55:4687–702.

22. Fisher RA. Statistical methods for research workers. 14th ed. Edinburgh:Oliver and Boyd; 1970.

23. Esseen C-G. A moment inequality with an application to the central limittheorem. Skand Aktuarietidskr. 1956;39:160–70.

24. Berger VW, Lunneborg C, Ernst MD, Levine JG. Parametric analyses inrandomized clinical trials. J Mod Appl Stat Methods. 2002;1(1):74–82.

25. Berger VW. Pros and cons of permutation tests in clinical trials. Stat Med.2000;19:1319–28.

26. Berry AC. The accuracy of the Gaussian approximation to the sum ofindependent variates. Trans Amer Math Soc. 1941;49(1):122–36.

27. Chavalarias D, Wallach JD, Li AHT, Ioannidis JPA. Evolution of reporting Pvalues in the biomedical literature, 1990-2015. JAMA. 2016;315(11):1141–8.

28. Poincaré H. Science and hypothesis. New York: Dover Publications; 1952.29. Bernard C. An introduction to the study of experimental medicine. New

York: Dover Publications; 1957.30. Goodman SN. Toward evidence-based medical statistics, 1: the P value

fallacy. Ann Intern Med. 1999;130(12):995–1004.31. Goodman SN. A dirty dozen: twelve p-value misconceptions. Semin Hematol.

2008;45(3):135–40.32. Gelman A. P values and statistical practice. Epidemiology. 2013;24(1):69–72.33. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process

and purpose. Am Stat. 2016;70(2):129–33.34. Hill AB. The environment and disease: association or causation? Proc R Soc

Med. 1965;58(5):295–300.35. Fedak KM, Bernal A, Capshaw ZA, Gross S. Applying the Bradford Hill criteria

in the 21st century: how data integration has changed causal inference inmolecular epidemiology. Emerg Themes Epidemiol. 2015;12:14.

• We accept pre-submission inquiries • Our selector tool helps you to find the most relevant journal• We provide round the clock customer support • Convenient online submission• Thorough peer review• Inclusion in PubMed and all major indexing services • Maximum visibility for your research

Submit your manuscript atwww.biomedcentral.com/submit

Submit your next manuscript to BioMed Central and we will help you at every step:


AbstractBackgroundIndividual response to treatment: The determinism vs chance dilemmaThe methodology of clinical trials: Does averaging work?The heterogeneity curse: An exampleStatistical inference from clinical trials: Are the assumptions met?Optimal sample size? Try n = 1!Statistical analysis of clinical trials as a ritualThe mantra of large n and the invocation of normalitySample size: Fixed or random?The idol of statistical significanceRandomization ignoredThe magic 5% and other arbitrary thresholds

Discussion, conclusions and recommendationsFundingAvailability of data and materialsAuthors’ contributionsAuthors’ informationEthics approval and consent to participateConsent for publicationCompeting interestsPublisher’s NoteReferences

Documents

BMC Medical Research Methodology - Why statistical ......“Why Most Published Research Findings are False” [11], John Ioannidis sought to explain this phenomenon. He assumed the