An estimate of the science-wise false discovery rate and application to the top medical literature
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The accuracy of published medical research is critical for scientists, physicians and patients who rely on these results. However, the fundamental belief in the medical literature was called into serious question by a paper suggesting that most published medical research is false. Here we adapt estimation methods from the genomics community to the problem of estimating the rate of false discoveries in the medical literature using reported $P$-values as the data. We then collect $P$-values from the abstracts of all 77 430 papers published in The Lancet, The Journal of the American Medical Association, The New England Journal of Medicine, The British Medical Journal, and The American Journal of Epidemiology between 2000 and 2010. Among these papers, we found 5322 reported $P$-values. We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, $P = 0.18$) or with respect to journal submissions (0.5% more FP per 100 submissions, $P = 0.12$). Statistical analysis must allow for false discoveries in order to make claims on the basis of noisy data. But our analysis suggests that the medical literature remains a reliable record of scientific progress.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Analyses based on the distribution of P values reported in the medical literature, for example, suggested a false-discovery rate of only 14% (9).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [9]
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_25 · Cited by 45,181 · cited 1×
“This is a similar problem encountered in the analysis of survival data, when patients are lost to follow-up (Kaplan and Meier, 1958).”
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501452 · Cited by 39,158
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Statistical significance for genomewide studies
Storey, J. D., & Tibshirani, R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1530509100 · Cited by 9,692 · cited 2×
“…similar statistical methods for estimating the rate of false discoveries in genomic (Kendziorski and others, 2003; Newton and others, 2001; Efron and Tibshirani, 2002; Storey and Tibshirani, 2003) or brainimaging studies (Genovese and others, 2002, where many hypotheses are tested simultaneously.”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Casella, G., & Berger, R. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,711
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002) · NeuroImage · doi:10.1006/nimg.2001.1037 · Cited by 4,923
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
The interaction of TIGIT with PVR and PVRL2 inhibits human NK cell cytotoxicity
Stanietsky, N., Simic, H., Arapovic, J., Toporik, A., Levy, O., Novik, A., Levine, Z., Beiman, M., Dassa, L., Achdout, H., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0903474106 · Cited by 3,524
Research works printed but not linked: 4
- 1989, Journal of Political Economy, 100, 1257
- 2011, STATISTICAL APPLICATIONS IN GENETICS AND MOLECULAR BIOLOGY, 10, 28
- 2012, PNAS, 334, 1226
- 2011, PROCEEDINGS OF THE WORLD CONGRESS ON ENGINEERING, 1, 1
Cited by
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
“We chose p 0.05 as the cutoff for our upper bin (following [3]), rather than p = 0.05 (see [46]) because we suspect that many authors do not regard p = 0.05 as significant.”
BackgroundMethodInfluentialDonoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695
Why, When and How to Adjust Your P Values?
Jafari, M., & Ansari‐Pour, N. (2019) · PubMed · doi:10.22074/cellj.2019.5992 · Cited by 468
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Evolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
“197 921 60 317 (736-3700) Clinical trial 52 639 16 018 (30-2960) Meta-analysis 1 396 965 17 125 (176-2245) Review”
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Kers, J. G., & Saccenti, E. (2022) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.796025 · Cited by 359
“p-Hacking is an illegitimate practice that promotes unreproducible results, polluting literature and adding to publication bias (Ioannidis, 2005; Jager and Leek, 2014; Raj et al., 2018).”
BackgroundPatil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
“It is also in line with estimates we have previously made about the rate of false discoveries in the medical literature ( Jager & Leek, 2014).”
Self-citationReproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
“Jager and Leek [2] argue in a similar direction and describe the p -curve as a mixture of a uniform distribution and a beta distribution.”
BackgroundInfluentialFinkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Reproducible research can still be wrong: Adopting a prevention approach
Leek, J. T., & Peng, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1421412111 · Cited by 193
“This problem is compounded by the fact that data sets and data analyses are becoming increasingly complex, the rate of submission to journals continues to increase (8), and the demands on statisticians to referee are increasing.”
BackgroundSelf-citationThe Prevalence of Marginally Significant Results in Psychology Over Time
Olsson-Collentine, A., van Assen, M. A. L. M., & Hartgerink, C. H. J. (2019) · Psychological Science · doi:10.1177/0956797619830326 · Cited by 178
The Perils of Balance Testing in Experimental Design: Messy Analyses of Clean Data
Mutz, D. C., Pemantle, R., & Pham, P. (2019) · The American Statistician · doi:10.1080/00031305.2017.1322143 · Cited by 195
“In the last 10 years the same theme has been repeated (Harvey, Liu, and Zhu 2014), questioned (Jager and Leek 2014), and elaborated (Moonesinghe, Khoury, and Janssens 2007), leading to 3000 citations for Ioannidis’ article alone. The crisis of confidence in experimental studies is of particular concern because randomi…”
BackgroundReplicability and Prediction: Lessons and Challenges from GWAS
Marigorta, U. M., Rodríguez, J. A., Gibson, G., & Navarro, A. (2018) · Trends in Genetics · doi:10.1016/j.tig.2018.03.005 · Cited by 185
Xu, X., Chen, Y., & Miao, J. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.6 · Cited by 169
Avoiding common pitfalls in machine learning omic data science
Teschendorff, A. E. (2019) · Nature Materials · doi:10.1038/s41563-018-0241-z · Cited by 132
Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Jarvis, M. F., & Williams, M. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2015.12.001 · Cited by 129
P-Values: Misunderstood and Misused
Vidgen, B., & Yasseri, T. (2016) · Frontiers in Physics · doi:10.3389/fphy.2016.00006 · Cited by 143
“This core finding is supported by Jager and Leek’s study on “significant” publications as well [13].”
BackgroundResultLeek, J. T., & Peng, R. D. (2015) · Science · doi:10.1126/science.aaa6146 · Cited by 107
Self-citationWeighted false discovery rate controlling procedures for clinical trials
Benjamini, Y., & Cohen, R. (2016) · Biostatistics · doi:10.1093/biostatistics/kxw030 · Cited by 117
Ash, J. R., Wognum, C., Rodríguez-Pérez, R., Aldeghi, M., Cheng, A. C., Clevert, D.-A., Engkvist, O., Fang, C., Price, D. J., Hughes‐Oliver, J. M., et al. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01609 · Cited by 130
Machine learning-integrated omics for the risk and safety assessment of nanomaterials
Ahmad, F., Mahmood, A., & Muhmood, T. (2021) · Biomaterials Science · doi:10.1039/d0bm01672a · Cited by 110
LaKind, J. S., Sobus, J. R., Goodman, M., Barr, D. B., Fürst, P., Albertini, R. J., Arbuckle, T. E., Schoeters, G., Tan, Y.-M., Teeguarden, J., et al. (2014) · Environment International · doi:10.1016/j.envint.2014.07.011 · Cited by 95
A direct approach to estimating false discovery rates conditional on covariates
Boca, S. M., & Leek, J. T. (2018) · PeerJ · doi:10.7717/peerj.6035 · Cited by 109
“, 2013), estimating science-wise false discovery rates (Jager and Leek, 2013), or improving power in high-throughput biological studies (Ignatiadis et al.”
Self-citationBishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90
Hargis, K. E., & Blalock, E. M. (2017) · Behavioural Brain Research · doi:10.1016/j.bbr.2016.05.007 · Cited by 90
Distributions of p -values smaller than .05 in psychology: what is going on?
Hartgerink, C. H. J., van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · PeerJ · doi:10.7717/peerj.1935 · Cited by 89
“…biased23 reporting of p-values in medical abstracts, but noted the variety of potential causes (e.g., publication bias,24 fraud, selective reporting).25 At the same time, other studies failed to find a bump of p-values below .05 (Jager and Leek, 2014;26 Krawczyk, 2015; Vermeulen et al., 2015).”
BackgroundMethodResultInfluentialThe quest for an optimal alpha
Miller, J. O., & Ulrich, R. (2019) · PLoS ONE · doi:10.1371/journal.pone.0208631 · Cited by 89
The Dark Side of Radiomics: On the Paramount Importance of Publishing Negative Results
Buvat, I., & Orlhac, F. (2019) · Journal of Nuclear Medicine · doi:10.2967/jnumed.119.235325 · Cited by 90
Elliott, G., Kudrin, N., & Wüthrich, K. (2022) · Econometrica · doi:10.3982/ecta18583 · Cited by 113
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
Reproducible Research: A Retrospective
Peng, R. D., & Hicks, S. C. (2021) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-012420-105110 · Cited by 64
Is Most Published Research Really False?
Leek, J. T., & Jager, L. R. (2016) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-060116-054104 · Cited by 90
Self-citationMatthes, J., Marquart, F., Naderer, B., Arendt, F., Schmuck, D., & Adam, K. (2015) · Communication Methods and Measures · doi:10.1080/19312458.2015.1096334 · Cited by 66
“…low statistical power, undisclosed flexibilities in the selection of items and cases, and confirmation biases in published research (e.g., Asendorpf et al., 2013; Bakker & Wicherts, 2014; Chambers et al., 2014; Humphreys et al., 2013; Jager & Leek, 2014; O’Boyle et al., 2014; Pigott et al., 2013).”
BackgroundResultEvidence-Based Medicine and Clinical Research: Both Are Needed, Neither Is Perfect
Szajewska, H. (2018) · Annals of Nutrition and Metabolism · doi:10.1159/000487375 · Cited by 83
Keller‐McNulty, S., Korkmaz, G., Orr, M., Schroeder, A. D., & Shipp, S. S. (2017) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-060116-054114 · Cited by 82
From DSM-IV to DSM-5 alcohol use disorder: An overview of epidemiological data
Bartoli, F., Carrà, G., Crocamo, C., & Clerici, M. (2014) · Addictive Behaviors · doi:10.1016/j.addbeh.2014.09.029 · Cited by 72
Cristea, I. A., & Ioannidis, J. P. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197440 · Cited by 61
Reconnecting p -Value and Posterior Probability Under One- and Two-Sided Tests
Shi, H., & Yin, G. (2020) · The American Statistician · doi:10.1080/00031305.2020.1717621 · Cited by 65
“Jager and Leek (2014) surveyed publications in high-profile medical journals and estimated the rate of false discoveries in the medical literature using reported p-values as the data, which led to a conclusion that the medical literature remains a reliable record of scientific progress.”
BackgroundPerneger, T., & Combescure, C. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.003 · Cited by 51
The limitations of observation studies for decision making regarding drugs efficacy and safety
Gueyffier, F., & Cucherat, M. (2018) · Therapies · doi:10.1016/j.therap.2018.11.001 · Cited by 56
“Moreover, the adjustment-variables set is rarely predefined, leaving the possibility of some p hacking [23, 24].”
Van Poucke, S., Thomeer, M., Heath, J., & Vukicevic, M. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5549 · Cited by 46
“Limitations of RCTs or suboptimally designed RCTs are at times overlooked or ignored [14].”
Emotion Regulation in Couples Affected by Female Sexual Interest/Arousal Disorder
Dubé, J. P., Corsini‐Munt, S., Muise, A., & Rosen, N. O. (2019) · Archives of Sexual Behavior · doi:10.1007/s10508-019-01465-4 · Cited by 52
Miller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 43
“The model can also be used to address the ongoing discussion of FP and FN rates considered in the introduction (e.g., Ioannidis, 2005b; Jager & Leek, 2014; Simmons et al., 2011).”
BackgroundMethodType A Behavior Pattern is not a Predictor of Premature Mortality
Šmigelskas, K., Žemaitienė, N., Julkunen, J., & Kauhanen, J. (2014) · International Journal of Behavioral Medicine · doi:10.1007/s12529-014-9435-1 · Cited by 40
Spiegelhalter, D. J. (2017) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12302 · Cited by 49
Burt, T., Button, K. S., Thom, H. H. Z., Noveck, R. J., & Munafo, M. R. (2017) · Clinical and Translational Science · doi:10.1111/cts.12478 · Cited by 46
Empowering phase II clinical trials to reduce phase III failures
Martini, D. D. (2019) · Pharmaceutical Statistics · doi:10.1002/pst.1980 · Cited by 40
Top ten errors of statistical analysis in observational studies for cancer research
Carmona‐Bayonas, A., Jiménez‐Fonseca, P., Fernández‐Somoano, A., Alvarez-Manceñido, F., Castañón, E., Custodio, A., de la Peña, F. A., Martín‐Payo, R., & Valiente, L. P. (2017) · Clinical & Translational Oncology · doi:10.1007/s12094-017-1817-9 · Cited by 36
The GRIMMER test: A method for testing the validity of reported measures of variability
Anaya, J. (2016) · PeerJ Preprints · doi:10.7287/peerj.preprints.2400v1 · Cited by 31
“It is said that we are in the midst of a reproducibility crisis Ioannidis (2005), although it should be noted that some believe these concerns are overblown (Patil et al., 2016; Jager and Leek, 2014).”
BackgroundKoçak, B., Bulut, E., Bayrak, O. N., Okumus, A. A., Altun, O., Arvas, Z. B., & Kavukoglu, I. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110830 · Cited by 34
Type M Error Might Explain Weisburd’s Paradox
Gelman, A., Skarðhamar, T., & Aaltonen, M. (2017) · Journal of Quantitative Criminology · doi:10.1007/s10940-017-9374-5 · Cited by 32
“Jager and Leek (2014) find 75–80% of p-values below 0.05 in several leading journals in medicine and public health, Gerber and Malhotra (2008a,b) examine hypothesis tests published in three leading sociology journals and two leading political science journals and find well over half to be…”
BackgroundInfluentialMultiplicity corrections in life sciences: challenges and consequences
Menyhárt, O., & Győrffy, B. (2025) · International Journal of Epidemiology · doi:10.1093/ije/dyaf098 · Cited by 31
Are most randomised trials in anaesthesia and critical care wrong? An analysis using Bayes’ theorem
Sidebotham, D. A. (2020) · Anaesthesia · doi:10.1111/anae.15029 · Cited by 33
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28
Limitations of empirical calibration of p‐values using observational data
Gruber, S., & Tchetgen, E. T. (2016) · Statistics in Medicine · doi:10.1002/sim.6936 · Cited by 30
“Although methods for quantifying the proportion of false findings remain under debate, with some estimates as low as 14% [3], there is agreement that multiple factors contribute to the problem.”
MethodAssessing robustness to worst case publication bias using a simple subset meta-analysis
Mathur, M. (2024) · British medical journal · doi:10.1136/bmj-2023-076851 · Cited by 23
P-hacking in meta-analyses: A formalization and new meta-analytic methods
Mathur, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 19
“In the Supporting Information (Section 1), we review the strengths and limitations of other existing methods for SWS. 20,29 In this article, we develop new analysis methods that advance methodologically upon these existing approaches for SAS or SWS.”
MethodA Theory of Statistical Inference for Ensuring the Robustness of Scientific Results
Coker, B., Rudin, C. D., & King, G. (2021) · Management Science · doi:10.1287/mnsc.2020.3818 · Cited by 42
“ching all Open Access papers in the PubMed database (~100,000 papers). This type of analysis has also been contested (Bishop et al., 2016), and not all meta-analyses have found evidence of p-hacking (Jager and Leek, 2014). Several types of solutions to p-hacking have been proposed: •We could require researchers to “pr…”
BackgroundDegen, P. M., & Medo, M. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011630 · Cited by 28
“Although the author’s claim that “most published research findings are false” has been the subject of considerable debate [18, 19], it indeed appears to be the case that certain fields such as preclinical cancer biology are struggling with a high prevalence of research with replication problems…”
de Ridder, J. (2022) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2022.02.003 · Cited by 26
Replicability in Brain Imaging
Kelly, R. E., & Hoptman, M. J. (2022) · Brain Sciences · doi:10.3390/brainsci12030397 · Cited by 24
Stock-specific sentiment and return predictability
Coqueret, G. (2020) · Quantitative Finance · doi:10.1080/14697688.2020.1736314 · Cited by 26
Publication Bias in Recent Empirical Accounting ResearchPreprint
Basu, S., & Park, H. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2379889 · Cited by 25
Goodman, S. N. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt035 · Cited by 27
“Stanford University, Stanford, CA 94305, USA steve.goodman@stanford.edu It is a pleasure to have the opportunity to comment on this contribution by Jager and Leek (2013), perhaps the first broad-based empirical attempt to examine the now 8-year-old claim that most conclusions in the c© The Author…”
Cesana, B. M. (2018) · Biomedical Journal of Scientific & Technical Research · doi:10.26717/bjstr.2018.06.001359 · Cited by 21
“However, the Jager and Leek’s paper has been furtherly criticized by six companion commentaries [18-23] in the same issue of Biostatistics with, not surprisingly, very different judgements and considerations.”
BackgroundQuality Control for Scientific Research: Addressing Reproducibility, Responsiveness, and Relevance
Hubbard, D. W., & Carriquiry, A. L. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543138 · Cited by 23
“In a recent publication, Jager and Leek (2014) dispute the notion that the reproducibility crisis in medical studies is as dire as it is in other scientific areas.”
BackgroundMa, Q., Li, Z., Li, W., Chen, Q., Liu, X., Feng, W., & Lei, J. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.111127 · Cited by 19
Statistical methods for replicability assessment
Hung, K. E., & Fithian, W. (2020) · The Annals of Applied Statistics · doi:10.1214/20-aoas1336 · Cited by 26
“Related error rates have also been estimated before: Jager and Leek (2013) have modeled the pvalue distributions under alternatives and the selection for statistical significance to estimate the FDR in the medical literature, accompanied by useful discussions from Gelman and O’Rourke (2013);…”
MethodIs There Evidence of P-Hacking in Imaging Research?
Rooprai, P., Islam, N., Salameh, J., Ebrahimzadeh, S., Kazi, A., Frank, R., Ramsay, T., Mathur, M. B., Absi, M., Khalil, A. A., et al. (2022) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221139418 · Cited by 18
“Similar to our results, Jager and Leek 27 found no evidence of p-hacking in a text-mining study using P -values from abstracts of fi ve medical journals.”
Resultp-Hacking in Experimental Audit Research
Khan, M. J., & Tronnes, P. C. (2018) · Behavioral Research in Accounting · doi:10.2308/bria-52183 · Cited by 16
Reproducibility and replicability of rodent phenotyping in preclinical studiesPreprint
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R. T., Golani, I., Gómez-Marín, À., Heller, R., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/079350 · Cited by 16
The distribution of probability values in medical abstracts: an observational study
Ginsel, B. L., Aggarwal, A., Xuan, W., & Harris, I. A. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1691-x · Cited by 15
“05 [15].”
Intuition is not evidence: Prescriptions for behavioral interventions from social psychology
Wilson, T. D., & Juarez, L. (2015) · Behavioral Science & Policy · doi:10.1353/bsp.2015.0006 · Cited by 17
Neurotechnological Convergence and “Big Data”: A Force-Multiplier Toward Advancing Neuroscience
DiEuliis, D., & Giordano, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-28422-4_6 · Cited by 14
Nelms, D. W., Vargas, H. D., Bedi, R. S., & Paruch, J. L. (2021) · Diseases of the Colon & Rectum · doi:10.1097/dcr.0000000000002146 · Cited by 15
Explorations in statistics: statistical facets of reproducibility
Curran‐Everett, D. (2016) · AJP Advances in Physiology Education · doi:10.1152/advan.00042.2016 · Cited by 14
“His paper spurred others to estimate theoretically (25) and empirically (28) the extent to which experimental results are reproducible.”
BackgroundSchimmack, U., & Bartos̆, F. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290084 · Cited by 17
“To summarize, our simulation confirms that Jager and Leek’s [19] method provides meaningful estimates of the false discovery risk and that the method is more likely to overestimate rather than underestimate the true false discovery rate.”
BackgroundMethodInfluentialMathur, M. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1667 · Cited by 15
Peng, R. D., & Parker, H. S. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040220-013917 · Cited by 15
“Recent work has focused on the quality and variability of data analyses published in various fields of study (e.g. 12, 30, 31, 46).”
BackgroundFaccioli, L. A. P., Cetin, Z., Kocas‐Kilicarslan, Z. N., Ortiz, K., Sun, Y., Hu, Z., Kurihara, T., Tafaleng, E. N., Florentino, R. M., Wang, Z., et al. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms241713406 · Cited by 11
Sports Metaresearch: An Emerging Discipline of Sport Science and Medicine
Warmenhoven, J., Menaspà, P., Borg, D. N., Vazire, S., White, N. M., Sainani, K. L., Nimphius, S., Coutts, A. J., & Impellizzeri, F. M. (2025) · Sports Medicine · doi:10.1007/s40279-025-02181-x · Cited by 11
“Additionally, Jager and Leek [67] have demonstrated the power of integrating text mining to facilitate metaresearch, through the use of a text extraction algorithm to estimate the rate of false discoveries in the medical literature, by using reported p -values as the data.”
Exploratory analyses: How to meaningfully interpret and report them
Roberts, A., Roche, M., & Sainani, K. L. (2023) · PM&R · doi:10.1002/pmrj.12980 · Cited by 5
Weighted mining of massive collections of P-values by convex optimization
Dobriban, E. (2017) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iax013 · Cited by 9
Promises and Pitfalls of High-Throughput Biological Assays
Finak, G., & Gottardo, R. (2016) · Methods in molecular biology · doi:10.1007/978-1-4939-3572-7_12 · Cited by 12
Statistical proof? The problem of irreproducibility
Holmes, S. (2017) · Bulletin of the American Mathematical Society · doi:10.1090/bull/1597 · Cited by 9
Is the replication crisis a base-rate fallacy?
Autzen, B. (2021) · Metamedicine · doi:10.1007/s11017-022-09561-8 · Cited by 9
False-positive and false-negative risks for individual multicentre trials in critical care
Sidebotham, D. A., & Barlow, C. J. (2022) · BJA Open · doi:10.1016/j.bjao.2022.100003 · Cited by 10
Ernst, E., & Smith, K. (2018) · Springer eBooks · doi:10.1007/978-3-319-69941-7_2 · Cited by 10
Estimating the replicability of highly cited clinical research (2004–2018)
da Costa, G. G., Neves, K., & Amaral, O. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307145 · Cited by 8
“Thus, the replicability panorama in high-impact clinical research may have changed during this period [20–22].”
BackgroundMartin, J. J., & Martin, D. (2021) · Kinesiology Review · doi:10.1123/kr.2020-0067 · Cited by 10
Creating falseness—How to establish statistical evidence of the untrue
Lytsy, P. (2017) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12823 · Cited by 9
Päll, T., Luidalepp, H., Tenson, T., & Maiväli, Ü. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002007 · Cited by 9
Cultural Genomics: Promises and Challenges
Chen, C., & Moyzis, R. K. (2018) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022117736526 · Cited by 9
“…from social psychology (Open Science Collaboration, 2015, but see Gilbert, King, Pettigrew, & Wilson, 2016) to biomedical research (Ioannidis, 2005; Ioannidis & Khoury, 2011; but see Jager & Leek, 2013), and from earth sciences (Santer, Wigley, & Taylor, 2011) to computer science (Peng, 2011).”
BackgroundAhmed, T. M., Zhu, Z., Yasrab, M., Blanco, A., Kawamoto, S., He, J. P., Fishman, E. K., Chu, L. C. H., & Javed, A. A. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-024-16064-4 · Cited by 7
“48 Within the realm of radiom-ics, this matter is further complicated by the added ambiguity of the “black box” nature of radiomics methods, which makes reproducibility and independent validation of results across different institutions with varying imaging protocols challenging.”
The Controversy behind the Controversies: Scientific Discourse in the Twenty-First Century
Fahnestock, J. (2025) · Rhetoric Society Quarterly · doi:10.1080/02773945.2025.2484162 · Cited by 6
How Credible Is the Credibility Revolution?
Lang, K. (2024) · Journal of Labor Economics · doi:10.1086/732772 · Cited by 10
Metadata
- Authors
- Jager, L. R., & Leek, J. T.
- Affiliations
- United States Naval Academy (US)
- Johns Hopkins University (US)
- Corresponding author
- Leek, J. T.
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-09-25
- Volume
- 15
- Issue
- 1
- Pages
- 1-12
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- OpenAlex
- W2098523432
- Cited by
- 179
- References
- 48
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 9 · 2023: 15 · 2022: 10 · 2021: 12 · 2020: 12 · 2019: 11 · 2018: 23 · 2017: 17 · 2016: 34 · 2015: 17 · 2014: 8 · 2013: 1
- FWCI
- 22.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Biostatistics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Johns Hopkins University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- false discovery rate, false positives, p-values, journal abstracts
- MeSH terms
- Algorithms, Computer Simulation, Data Interpretation, Statistical, False Positive Reactions, United Kingdom, Humans, Publications, Software, United States, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.