An estimate of the science-wise false discovery rate and application to the top medical literature
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The accuracy of published medical research is critical for scientists, physicians and patients who rely on these results. However, the fundamental belief in the medical literature was called into serious question by a paper suggesting that most published medical research is false. Here we adapt estimation methods from the genomics community to the problem of estimating the rate of false discoveries in the medical literature using reported $P$-values as the data. We then collect $P$-values from the abstracts of all 77 430 papers published in The Lancet, The Journal of the American Medical Association, The New England Journal of Medicine, The British Medical Journal, and The American Journal of Epidemiology between 2000 and 2010. Among these papers, we found 5322 reported $P$-values. We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, $P = 0.18$) or with respect to journal submissions (0.5% more FP per 100 submissions, $P = 0.12$). Statistical analysis must allow for false discoveries in order to make claims on the basis of noisy data. But our analysis suggests that the medical literature remains a reliable record of scientific progress.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Analyses based on the distribution of P values reported in the medical literature, for example, suggested a false-discovery rate of only 14% (9).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [9]
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_25 · Cited by 45,181 · cited 1×
“This is a similar problem encountered in the analysis of survival data, when patients are lost to follow-up (Kaplan and Meier, 1958).”
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501452 · Cited by 39,158
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Statistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692 · cited 2×
“…similar statistical methods for estimating the rate of false discoveries in genomic (Kendziorski and others, 2003; Newton and others, 2001; Efron and Tibshirani, 2002; Storey and Tibshirani, 2003) or brainimaging studies (Genovese and others, 2002, where many hypotheses are tested simultaneously.”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
The interaction of TIGIT with PVR and PVRL2 inhibits human NK cell cytotoxicity
Stanietsky, N., Simic, H., Arapovic, J., Toporik, A., Levy, O., Novik, A., Levine, Z., Beiman, M., Dassa, L., Achdout, H., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0903474106 · Cited by 3,524
Research works printed but not linked: 4
- 1989, Journal of Political Economy, 100, 1257
- 2011, STATISTICAL APPLICATIONS IN GENETICS AND MOLECULAR BIOLOGY, 10, 28
- 2012, PNAS, 334, 1226
- 2011, PROCEEDINGS OF THE WORLD CONGRESS ON ENGINEERING, 1, 1
Cited by
Items 101–178 of 178
Does bad inference drive out good?
Marozzi, M. (2015) · Clinical and Experimental Pharmacology and Physiology · doi:10.1111/1440-1681.12422 · Cited by 6
A Two-Sided Discussion of Preregistration of NLP Research
Søgaard, A., Hershcovich, D., & de Lhoneux, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.6 · Cited by 9
Fanelli, D. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0004 · Cited by 7
Avidan, M. S., & Wildes, T. S. (2014) · British Journal of Anaesthesia · doi:10.1093/bja/aeu263 · Cited by 5
Assessing the Probability that a Finding Is Genuine for Large-Scale Genetic Association Studies
Kuo, C., Vsevolozhskaya, O. A., & Zaykin, D. V. (2015) · PLoS ONE · doi:10.1371/journal.pone.0124107 · Cited by 4
At the Crossroads: Lessons and Challenges in Computational Social Science
Borge-Holthoefer, J., Moreno, Y., & Yasseri, T. (2016) · Frontiers research topics · doi:10.3389/978-2-88945-021-3 · Cited by 4
Williamson, J. C., & Silverstein, T. P. (2021) · Journal of Chemical Education · doi:10.1021/acs.jchemed.0c01072 · Cited by 4
Reid, N. M. (2015) · Elsevier eBooks · doi:10.1016/b978-0-08-097086-8.42170-7 · Cited by 4
Publication Bias in Meta-Analyses from Psychology and Medicine: A Meta-Meta-AnalysisPreprint
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/czdp3 · Cited by 4
Zeng, B., Wang, H., Chen, J., Wu, S., & Xie, J. (2026) · Journal of Neurology · doi:10.1007/s00415-026-13674-8 · Cited by 3
“Inconsist-ency between direct and indirect evidence was examined using loop-specific, local, and global inconsistency tests; when the P value for inconsistency was greater than 0.05, a consistency model was adopted for the analysis [24].”
InfluentialUsing controls to limit false discovery in the era of big data
Parks, M., Raphael, B. J., & Lawrence, C. E. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2356-2 · Cited by 5
Satterthwaite, W. H. (2023) · Fisheries Research · doi:10.1016/j.fishres.2023.106763 · Cited by 3
Randomness and the Games of Science
Goeman, J. J. (2016) · The frontiers collection · doi:10.1007/978-3-319-26300-7_5 · Cited by 5
“Jager and Leek (2014) estimated the proportion of unreliable results in the whole medical literature at 14 %.”
BackgroundA Statistical Model to Investigate the Reproducibility Rate Based on Replication Experiments
Pauli, F. (2018) · International Statistical Review · doi:10.1111/insr.12273 · Cited by 4
“In general, a sensible option would be to use a more flexible specification for their distribution, for example, Jager and Leek (2014) mention the possibility of using a mixture of Beta distributions, but a non-parametric specification could also be used.”
BackgroundMethodInfluentialMűködési zavarok és megújulási törekvések a tudományos pszichológiában, a 21. század elején
Szokolszky, Á. (2018) · Magyar Pszichológiai Szemle · doi:10.1556/0016.2018.73.4.3 · Cited by 3
Contributions towards understanding and building sustainable sciencePreprint
Hartgerink, C. H. J. (2019) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/4wtpc · Cited by 3
The Published Research Findings are Trustable?: Review
DOĞAN, İ., & DOĞAN, N. (2017) · Turkiye Klinikleri Journal of Biostatistics · doi:10.5336/biostatic.2016-53445 · Cited by 2
Hall, L. M., & Hendricks, A. E. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03817-7 · Cited by 5
The ABC’s of the alternative risk premium: academic roots
Gorman, S. A., & Fabozzi, F. J. (2021) · Journal of Asset Management · doi:10.1057/s41260-021-00234-0 · Cited by 3
Augustine, F., Doss, S. M., Pellicciotti, J., Mahate, S., & Singer, H. S. (2025) · Neuroscience · doi:10.1016/j.neuroscience.2025.01.005 · Cited by 3
The Reproducibility Crisis and Kinesiology: Confronting the Human & Statistical Limitations
Twietmeyer, G. (2025) · International Journal of Kinesiology in Higher Education · doi:10.1080/24711616.2025.2468469 · Cited by 2
“For criticisms of Ioannidis’ position see: (Goodman & Greenland, 2007; Jager & Leek, 2014).”
Has It Really Been Demonstrated That Most Genomic Research Findings Are False?
Samsa, G. P. (2014) · The American Statistician · doi:10.1080/00031305.2014.951127 · Cited by 2
“…(Jager and Leek 2014a) and discussion in the January 2014 issue of the journal Biostatistics (Benjamini and Hechtlinger 2014; Cox 2014; Gelman and O’Rourke 2014; Goodman 2014; Ioannidis 2014; Jager and Leek 2014b; Schuemie et al. 2014), also Goodman and Greenland (2007) and Ioannidis (2007).”
BackgroundPublication Policies for Replicable Research and the Community-Wide False Discovery Rate
Habiger, J., & Liang, Y. (2022) · The American Statistician · doi:10.1080/00031305.2021.1999857 · Cited by 4
“See Jager and Leek (2013) and Ioannidis (2013).”
BackgroundMethodKelly, R. E., Ahmed, A. O., & Hoptman, M. J. (2022) · Brain Sciences · doi:10.3390/brainsci12020165 · Cited by 2
“The corresponding values for fifth-graders were height [139, 5] and age [10, 0.”
BackgroundClassifying Positive Results in Clinical Psychology Using Natural Language Processing
Schiekiera, L., Diederichs, J., & Niemeyer, H. (2024) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000563 · Cited by 3
Treatment Effect Bias from Sample Snooping: Blinding Outcomes is Neither Necessary nor SufficientPreprint
Fisher, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02514 · Cited by 3
A glass half full interpretation of the replicability of psychological science
Leek, J. T., Patil, P., & Peng, R. D. (2015) · arXiv · doi:10.48550/arxiv.1509.08968 · Cited by 1
Self-citationWhat proportion of published research findings are false?
Lydersen, S., & Langaas, M. (2021) · Tidsskrift for Den norske legeforening · doi:10.4045/tidsskr.21.0703 · Cited by 1
A Tutorial on Multiple Testing: False Discovery Control
Chatelain, F. (2016) · EAS Publications Series · doi:10.1051/eas/1678008 · Cited by 2
Howard, J. (2018) · Springer eBooks · doi:10.1007/978-3-319-93224-8_30 · Cited by 1
The replication crisis in science and its overlap into kinantropology: an introduction
Baloun, L., & Mališ, J. (2023) · Tělesná kultura · doi:10.5507/tk.2023.002 · Cited by 1
Recent Applications of Web Semantics in eLifeScience
Clark, T., & Roos, M. (2014) · Journal of Web Semantics · doi:10.1016/j.websem.2014.11.005 · Cited by 1
“Lastly, the reproducibility of reported experimental results in the life sciences cannot be guaranteed – there seem to be many false positives strewn about the literature and the underlying false positive rate is a subject of debate [16] [17] [18].”
BackgroundJournal of Controversies in Biomedical Research - the Need of the Hour
Morais, C., & Vitetta, L. (2015) · Journal of Controversies in Biomedical Research · doi:10.15586/jcbmr.2015.5 · Cited by 1
“There are many reasons for the ‘reproducibility crisis’ (3, 4).”
BackgroundThe Economic Value of Firm-Specific News SentimentPreprint
Coqueret, G. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3248925 · Cited by 1
A Novel Framework for Constructing Sport-Based Rating Systems
Patel, A. (2020) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.17151083.v1 · Cited by 2
(2018) · PeerJ · doi:10.7287/peerj.6035v0.1/reviews/1 · Cited by 1
“A range of other applications for our methodology are also possible 452 by adapting our regression framework, including estimating false discovery rates for gene sets 453 [2], estimating science-wise false discovery rates [11], or improving power in high-throughput 454 biological studies [9].”
Publication Bias in Psychology
Kühberger, A. (2023) · Oxford University Press eBooks · doi:10.1093/obo/9780199828340-0307 · Cited by 3
Fine-scale modelling of the dynamics of phytoplankton communities
Picoche, C. (2022) · theses.fr (ABES) · doi:10.70675/fc1722e4z6066z40f0z8fdaz93eab2f9c461 · Cited by 1
Bormann, B. V. (2014) · Journal of Blood Disorders & Transfusion · doi:10.4172/2155-9864.1000205
“His conclusions have been strictly opposed by Jager and Leek estimating after all a 14%, thus ‘tolerable’ rate of false information [30].”
BackgroundFibroblast Differentiation and Models of Human Skin
Rakar, J. (2016) · Linköping University medical dissertations · doi:10.3384/diss.diva-125925
Statistical Blue lines: Biologist's A-Priori Need
Saikia, S. K. (2015)
Assessing Reproducibility and Value in Genomic Signatures
Patil, P. (2016)
A Bayesian Nonparametric Test of Significance-Chasing Biases
Karabatsos, G. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1269
Genomic Prediction and Genetic Dissection of Yield-Related Traits in Soft Red Winter Wheat
Ward, B. (2017) · VTechWorks (Virginia Tech)
Deep Phenotyping in the -Omic Era: Methodological Issues Facing Neuropsychiatric Disorders
Fisch, G. S. (2018) · Journal of Psychiatry and Brain Science · doi:10.20900/jpbs.20180011
Hall, L., & Hendricks, A. E. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/500660 · Cited by 1
“For more details, see the Supplemental Materials of Jager and Leek (20).”
BackgroundMethodInfluentialFallstricke in der statistischen Welt
Kiese-Himmel, C., & Plontke, S. K. (2019) · HNO · doi:10.1007/s00106-019-00750-x
Decision-making Risks of Operational Tests from a Statistical Point of View
Bo, Y., Bai, Y., & Liao, X. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1544/1/012182
Questionable Research Practices: p ‐Hacking, Replication, and Fraud
Arendt, F. (2020) · Wiley eBooks · doi:10.1002/9781119011071.iemp0008
Call for papers: special issue of Journal of Critical Realism on health and wellbeing
Price, L. (2020) · Journal of Critical Realism · doi:10.1080/14767430.2020.1843122
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/uqz2e
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/mjvwu
(2016) · Wiley eBooks · doi:10.1002/9783527681969.biblio
de Winter, J., & Dodou, D. (2017) · SpringerBriefs in applied sciences and technology · doi:10.1007/978-3-319-56964-2_3
Estimating the false finding rate across scientific fieldsPreprint
Piraino, S. W. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/97cwq
Reply letter to: False discovery rate in laser studies
Corrales, M., Panthier, F., Solano, C., Ventimiglia, E., & Traxer, O. (2023) · World Journal of Urology · doi:10.1007/s00345-023-04457-1
“[10] analyzed the FDR in the published medical literature using the reported p-values as the data and concluded that the medical and scientific literature remain a reliable record of scientific progress [10].”
MethodPhase III Failures for a Lack of Efficacy Can Be, in Significant Part, Recovered
Martini, D. D. (2019) · Epidemiology Biostatistics and Public Health · doi:10.54103/2282-0930/20638
“14), because: a) in some phase II trials the launching rule is based on statistical significance with threshold 5% or higher; b) it has been estimated that the FDR in top medical literature is 14% [8], where phase II clinical trials represent an even higher class of experiments, so that FDRII has be assumed to be at m…”
BackgroundRøge, K. M. (2022) · Journal of Pragmatic Constructivism · doi:10.7146/jopracon.v12i1.135312
Galwey, N. W. (2024) · Wiley eBooks · doi:10.1002/9781119889809.ch1
Estimating the false finding rate across scientific fieldsPreprint
Piraino, S. W. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/97cwq_v1
Hall, L. M., & Hendricks, A. E. (2020) · Figshare · doi:10.6084/m9.figshare.13357508
Li, X., Yang, S., & Yao, J. (2026) · Statistical Papers · doi:10.1007/s00362-025-01788-6 · Cited by 1
“Empirical evi - dence suggests that the prevalence of p-hacking is particularly notable in fields such as psychology, medical research, and economics (Simmons et al. 2011; Begg and Berlin 1988; Goodman 1999; Jager and Leek 2014; Brodeur et al. 2020).”
Caquelin, L. (2023) · theses.fr (ABES) · doi:10.70675/13ebb540z3938z4806z9ac3z3fac95f76f1e
The role of measurement practices in the replicability of empirical findings
Böhm, C. M. (2026) · Publication Server of Kaiserslautern University of Technology (Kaiserslautern University of Technology) · doi:10.26204/kluedo/13582
Metadata
- Authors
- Jager, L. R., & Leek, J. T.
- Affiliations
- United States Naval Academy (US)
- Johns Hopkins University (US)
- Corresponding author
- Leek, J. T.
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-09-25
- Volume
- 15
- Issue
- 1
- Pages
- 1-12
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- OpenAlex
- W2098523432
- Cited by
- 179
- References
- 48
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 9 · 2023: 15 · 2022: 10 · 2021: 12 · 2020: 12 · 2019: 11 · 2018: 23 · 2017: 17 · 2016: 34 · 2015: 17 · 2014: 8 · 2013: 1
- FWCI
- 22.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Biostatistics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Johns Hopkins University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- false discovery rate, false positives, p-values, journal abstracts
- MeSH terms
- Algorithms, Computer Simulation, Data Interpretation, Statistical, False Positive Reactions, United Kingdom, Humans, Publications, Software, United States, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.