Why statistical inference from clinical trials is likely to generate false and irreproducible results
Open accessAbstract
One area of biomedical research where the replication crisis is most visible and consequential is clinical trials. Why do outcomes of so many clinical trials contradict each other? Why is the effectiveness of many drugs and other medical interventions so low? Why have prescription medications become the third leading cause of death in the US and Europe after cardiovascular diseases and cancer? In answering these questions, the main culprits identified so far have been various biases and conflicts of interest in planning, execution and analysis of clinical trials as well as reporting their outcomes. In this work, we take an in-depth look at statistical methodology used in planning clinical trials and analyzing trial data. We argue that this methodology is based on various questionable and empirically untestable assumptions, dubious approximations and arbitrary thresholds, and that it is deficient in many other respects. The most objectionable among these assumptions is that of distributional homogeneity of subjects' responses to medical interventions. We analyze this and other assumptions both theoretically and through clinical examples. Our main conclusion is that even a totally unbiased, perfectly randomized, reliably blinded, and faithfully executed clinical trial may still generate false and irreproducible results. We also formulate a few recommendations for the improvement of the design and statistical methodology of clinical trials informed by our analysis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The absence of associations between broader result claims and larger, more nationally diverse samples, or reports of generalizability strengthening factors, is indicative of hasty generalizations.
“We treated the absence of such associations as indicative of hasty generalizations because broader result claims tend to require larger, more nationally diverse samples, or generalizability supporting background assumptions [8,12].”
From Hasty generalizations and generics in medical research: A systematic review (2024) · cites this paper as [12]
Broader result claims tend to require larger, more nationally diverse samples, or generalizability supporting background assumptions.
“We treated the absence of such associations as indicative of hasty generalizations because broader result claims tend to require larger, more nationally diverse samples, or generalizability supporting background assumptions [8,12].”
From Hasty generalizations and generics in medical research: A systematic review (2024) · cites this paper as [12]
In clinical trials, individual outcomes within tested groups are aggregated but outcomes between individuals are rarely the same (e.g., the variance in individuals can be up to four times greater than in groups).
“in clinical trials, individual outcomes within tested groups are aggregated but outcomes between individuals are rarely the same (e.g., the variance in individuals can be up to four times greater than in groups [12,13]).”
From Hasty generalizations and generics in medical research: A systematic review (2024) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,302 · see citing works
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · see citing works
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Probability: Theory and Examples.
Prewitt, K., & Durrett, R. T. (1992) · Journal of the American Statistical Association · doi:10.2307/2290301 · Cited by 3,238 · see citing works
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237 · see citing works
Statistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1930) · Journal of the American Statistical Association · doi:10.2307/2278215 · Cited by 1,565 · see citing works
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Personalized medicine: Time for one-person trials
Schork, N. J. (2015) · Nature · doi:10.1038/520609a · Cited by 1,229 · see citing works
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,188 · see citing works
An Introduction to the Study of Experimental Medicine
Bernard, C., Wolf, S., & Greene, H. C. (2018) · Routledge eBooks · doi:10.4324/9781351320764-1 · Cited by 939 · see citing works
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897 · see citing works
The accuracy of the Gaussian approximation to the sum of independent variates
Berry, A. C. (1941) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1941-0003498-3 · Cited by 822 · see citing works
Poincaré, H. P. H. (2024) · doi:10.55094/holistence.185 · Cited by 821 · see citing works
A dirty dozen: twelve p-value misconceptions.
Goodman, S. (2008) · Seminars in hematology (Print) · doi:10.1053/j.seminhematol.2008.04.003 · Cited by 694 · see citing works
Fedak, K. M., Bernal, A. J., Capshaw, Z. A., & Gross, S. A. (2015) · Emerging Themes in Epidemiology · doi:10.1186/s12982-015-0037-4 · Cited by 682 · see citing works
An introduction to the study of experimental medicine
Lasagna, L. C. (1958) · Journal of Chronic Diseases · doi:10.1016/0021-9681(58)90110-3 · Cited by 506 · see citing works
Evolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
(1953) · Journal of the Franklin Institute · doi:10.1016/0016-0032(53)90763-7 · Cited by 327 · see citing works
TGN1412: From Discovery to Disaster
Attarwala, H. (2010) · Journal of Young Pharmacists · doi:10.4103/0975-1483.66810 · Cited by 253 · cited 1× · see citing works
“Remarkably, the dose given to volunteers in the London trial was 500 times smaller than the one found to be safe in animals [4].”
BackgroundDirty Dozen: Twelve P-Value Misconceptions
Goodman, T. (2008) · Cited by 238 · see citing works
Offline: What is medicine's 5 sigma?
Horton, R. (2015) · The Lancet · doi:10.1016/s0140-6736(15)60696-1 · Cited by 219 · see citing works
Gelman, A. (2012) · Epidemiology · doi:10.1097/ede.0b013e31827886f7 · Cited by 216 · see citing works
Berger, V. W., Ivanova, A., & Knoll, M. D. (2003) · Statistics in Medicine · doi:10.1002/sim.1538 · Cited by 200 · see citing works
The Methods of Distances in the Theory of Probability and Statistics
Rachev, S. T., Klebanov, L. B., Stoyanov, S. V., & Fabozzi, F. J. (2013) · Springer eBooks · doi:10.1007/978-1-4614-4869-3 · Cited by 187 · see citing works
A moment inequality with an application to the central limit theorem
Esseen, C.-G. (1956) · Scandinavian Actuarial Journal · doi:10.1080/03461238.1956.10414946 · Cited by 186 · see citing works
Deadly medicines and organised crime: How big pharma has corrupted healthcare
Dickinson, J. A. (2014) · Europe PMC (PubMed Central) · Cited by 174 · see citing works
Some Restricted randomization rules in sequential designs
Soares, J. F., & Wu, C. (1983) · Communication in Statistics- Theory and Methods · doi:10.1080/03610928308828586 · Cited by 167 · see citing works
Proposal for a new model of breast cancer metastatic development
Demicheli, R., Retsky, M. W., Swartzendruber, D. E., & Bonadonna, G. (1997) · Annals of Oncology · doi:10.1023/a:1008263116022 · Cited by 140 · see citing works
Deadly Medicines and Organised Crime: How Big Pharma Has Corrupted Healthcare
Gøtzsche, P. C. (2013) · Cited by 133 · see citing works
Lin, H., McCulloch, C. E., Turnbull, B. W., Slate, E. H., & Clark, L. C. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000530)19:10 · Cited by 87 · see citing works
Mitochondrial Injury Lessons from the Fialuridine Trial
Honkoop, P., Scholte, H. R., de Man, R. A., & Schalm, S. W. (1997) · Drug Safety · doi:10.2165/00002018-199717010-00001 · Cited by 80 · see citing works
Scientists in the dark after French clinical trial proves fatal
Butler, D., & Callaway, E. (2016) · Nature · doi:10.1038/nature.2016.19189 · Cited by 77 · see citing works
London's disastrous drug trial has serious side effects for research
Wadman, M. (2006) · Nature · doi:10.1038/440388a · Cited by 60 · see citing works
Cell-survival probability at large doses: an alternative to the linear-quadratic model
Hanin, L. G., & Zaider, M. (2010) · Physics in Medicine and Biology · doi:10.1088/0031-9155/55/16/005 · Cited by 45 · see citing works
Self-citationParametric analyses in randomized clinical trials
Berger, V. W., Lunneborg, C. E., Ernst, M. D., & Levine, J. G. (2002) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1020255120 · Cited by 43 · see citing works
Berger, V. W. (2014) · Science and Engineering Ethics · doi:10.1007/s11948-014-9576-2 · Cited by 15 · see citing works
A quantitative insight into metastatic relapse of breast cancer
Hanin, L. G., & Pavlova, L. (2016) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2016.01.014 · Cited by 7 · see citing works
Self-citationDo Breast Cancer Patients Benefit from Surgery? Hypotheses, Mathematical Models and False Beliefs
Hanin, L. G. (2017) · Springer eBooks · doi:10.1007/978-3-319-57943-6_7 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Matthay, E. C., Hagan, E., Gottlieb, L. M., Tan, M. L., Vlahov, D., Adler, N. E., & Glymour, M. M. (2019) · SSM - Population Health · doi:10.1016/j.ssmph.2019.100526 · Cited by 84 · see citing works
Marshall University, Gress, T. W., Denvir, J., Marshall University, Shapiro, J. I., & Marshall University (2018) · Marshall Journal of Medicine · doi:10.18590/mjm.2018.vol4.iss2.9 · Cited by 60 · see citing works
PK-DB: pharmacokinetics database for individualized and stratified computational modeling
Grzegorzewski, J., Brandhorst, J., Green, K., Eleftheriadou, D., Duport, Y., Barthorscht, F., Köller, A., Ke, D. Y. J., Angelis, S. D., & König, M. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa990 · Cited by 60 · see citing works
Hill, C., Creswell, C., Vigerland, S., Nauta, M. H., March, S., Donovan, C., Wolters, L., Spence, S. H., Martin, J. L., Wozney, L., et al. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.02.002 · Cited by 44 · see citing works
Assessing assumptions for statistical analyses in randomised clinical trials
Nielsen, E. E., Nørskov, A. K., Lange, T., Thabane, L., Wetterslev, J., Beyersmann, J., de Uña‐Álvarez, J., Torri, V., Billot, L., Putter, H., et al. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111174 · Cited by 44 · see citing works
Mechanisms and uncertainty in randomized controlled trials: A commentary on Deaton and Cartwright
Suzuki, E., & VanderWeele, T. J. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.04.023 · Cited by 20 · see citing works
Multifractal foundations of biomarker discovery for heart disease and stroke
Mangalam, M., Sadri, A., Hayano, J., Watanabe, E., Kiyono, K., & Kelty-Stephen, D. G. (2023) · Scientific Reports · doi:10.1038/s41598-023-45184-2 · Cited by 18 · see citing works
van Boekel, R. L. M., Timmerman, H., Bronkhorst, E. M., Ruscheweyh, R., Vissers, K. C. P., & Steegers, M. A. H. (2020) · Pain Research and Management · doi:10.1155/2020/1050935 · Cited by 14 · see citing works
Osimani, B. (2020) · Boston studies in the philosophy of science · doi:10.1007/978-3-030-29179-2_15 · Cited by 12 · see citing works
Hasty generalizations and generics in medical research: A systematic review
Peters, U., Sherling, H. R., & Chin-Yee, B. (2024) · PLOS ONE · doi:10.1371/journal.pone.0306749 · Cited by 11 · see citing works
Cheng, W., Chen, Y., He, F.-M., Zhao, J.-G., Mao, Z.-R., Qin, B.-Y., Hu, C., Feng, S., Zhang, F., Dong, X., et al. (2026) · Critical Care Medicine · doi:10.1097/ccm.0000000000007011 · Cited by 6 · see citing works
Mheissen, S., Khan, H., Almuzian, M., Alzoubi, E. E., & Pandis, N. (2021) · European Journal of Orthodontics · doi:10.1093/ejo/cjab069 · Cited by 6 · see citing works
Accounting for clustering is still not routinely undertaken in orthodontic studies
Sudiskumar, N., Cobourne, M. T., Pandis, N., & Seehra, J. (2022) · European Journal of Orthodontics · doi:10.1093/ejo/cjac066 · Cited by 5 · see citing works
Hanin, L. G. (2021) · Mathematics · doi:10.3390/math9060603 · Cited by 5 · see citing works
Self-citationMignot, S., Ménard, J.-P., & Iraola, É. (2026) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.70227 · Cited by 4 · see citing works
Paradoxical Effects of Tumor Shrinkage on Long-Term Survival of Cancer Patients
Hanin, L. G. (2020) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2020.00027 · Cited by 3 · see citing works
Self-citationRandomization and Permutation Tests
Berger, V. W., Onghena, P., & Matthews, J. R. (2019) · Springer eBooks · doi:10.1007/978-3-319-52677-5_129-1 · Cited by 2 · see citing works
Rebrova, O. Y., Fediaeva, V. K., & Va, A. (2021) · Problems of Endocrinology · doi:10.14341/probl12797 · Cited by 2 · see citing works
Zhao, Q., Zhang, Y., & Xi, Q. (2025) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-025-03456-1 · Cited by 1 · see citing works
The active construction of past episodes
Parr, T., Pezzulo, G., & Friston, K. J. (2026) · Translational Neuroscience · doi:10.1515/tnsci-2025-0391 · Cited by 1 · see citing works
Chandu, V. C. (2021) · Journal of Orofacial Sciences · doi:10.4103/jofs.jofs_259_21 · Cited by 1 · see citing works
Mheissen, S., Khan, H., Seehra, J., & Pandis, N. (2022) · Journal of Dentistry · doi:10.1016/j.jdent.2022.104182 · Cited by 1 · see citing works
Randomization and Permutation Tests
Berger, V. W., Onghena, P., & Matthews, J. R. (2022) · Springer eBooks · doi:10.1007/978-3-319-52636-2_129 · Cited by 1 · see citing works
Brownstein, N. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04721 · Cited by 1 · see citing works
Final orders at the last chance saloon?
Wickham, H. (2019) · Drug and Therapeutics Bulletin · doi:10.1136/dtb.2018.000052
Effect of right censoring bias on survival analysisPreprint
Barrajón, E., & Barrajón, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08649
Bayesian censored piecewise regression mixture models with skewness
Dagne, G. A. (2022) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2021.2009496
Sachs, R. K., Huang, E. G., & Hanin, L. G. (2023) · Radiation Research · doi:10.1667/rade-22-00189.1
Self-citationFlattening the Truth Pyramid: Reconsidering the Evidence Hierarchy
Bierman, S. (2023) · The Journal of Medical Research · doi:10.31254/jmr.2023.9606
Nonparametric assessment of regimen response curve estimators
Pham, C., Baer, B. R., & Ertefaie, A. (2025) · Biometrics · doi:10.1093/biomtc/ujaf066
Preregistration in Animal Experimentation – A Qualitative Study on Researchers’ Views in SwitzerlandPreprint
Priboi, C., Lalor, S., Sarra, G., Vergauwe, E., Elger, B. S., & Würbel, H. (2025) · Research Square · doi:10.21203/rs.3.rs-8194213/v1
FAIRsharing record for: Pharmacokinetics Database
FAIRsharing Team, & König, M. (2018) · FAIRsharing.org · doi:10.25504/fairsharing.ayegqk
Qualitative study reveals perceived barriers and facilitators of preregistration in animal research
Priboi, C., Lalor, S. M., Vergauwe, E., Elger, B., & Würbel, H. (2026) · Royal Society Open Science · doi:10.1098/rsos.261156
- Authors
- Hanin, L.
- Affiliations
- Idaho State University (US)
- Corresponding author
- Hanin, Leonid
- ORCID
- Hanin, Leonid
- Venue
- BMC Medical Research Methodology
- Published
- 2017-08-22
- Volume
- 17
- Issue
- 1
- Pages
- 127-127
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC5568363
- OpenAlex
- W2746050295
- Cited by
- 32
- References
- 40
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 1 · 2023: 3 · 2022: 4 · 2021: 4 · 2020: 5 · 2019: 4 · 2018: 4
- FWCI
- 2.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- clinical trials, replication crisis, statistical inference, questionable research practices, trial design, statistical methodology, bias, conflicts of interest, false positives
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Humans, Stochastic Processes, Reproducibility of Results, Bias, Treatment Outcome, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
