Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Jager and Leek have tried to estimate a false-discovery rate (FDR) in abstracts of articles published in five medical journals during 2000-2010. Their approach is flawed in sampling, calculations, and conclusions. It uses a tiny portion of select papers in highly select journals. Randomized controlled trials and systematic reviews (designs with the lowest anticipated false-positive rates) are 52% of the analyzed papers, while these designs account for only 4% in PubMed in the same period. The FDR calculations consider the entire published literature as equivalent to a single genomic experiment where all performed analyses are reported without selection or distortion. However, the data used are the P-values reported in the abstracts of published papers; these P-values are a highly distorted, highly select sample. Besides selective reporting biases, all other biases, in particular confounding in observational studies, are also ignored, while these are often the main drivers for high false-positive rates in the biomedical literature. A reproducibility check of the raw data shows that much of the data Jager and Leek used are either wrong or make no sense: most of the usable data were missed by their script, 94% of the abstracts that reported ≥2 P-values had high correlation/overlap between reported outcomes, and only a minority of P-values corresponded to relevant primary outcomes. The Jager and Leek paper exemplifies the dreadful combination of using automated scripts with wrong methods and unreliable data. Sadly, this combination is common in the medical literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 1×
“See Ioannidis (2008) for references on this issue; Siontis and others (2011) on inflated effects in major journals; and Pereira and others (2011) on inflation of large effects in >85 000 meta-analyses.”
Self-citationContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Self-citationNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 1×
“Empirical data have demonstrated that non-significant results are disappearing from the published literature and that least credible scientific fields publish the highest proportions of significant results (Fanelli, 2010a, 2010b, 2012).”
Boutron, I. (2010) · JAMA · doi:10.1001/jama.2010.651 · Cited by 842
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · cited 1×
“2.12 Selecting only the P 0.05 range deflates FDR estimates The attraction of passing the 0.05 mark is not absolute (Ioannidis and Trikalinos, 2007): P-value clustering extends in the range of P = 0.05 and possibly even higher (Kavvoura and others, 2008).”
InfluentialSelf-citation“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 1×
“Empirical data have demonstrated that non-significant results are disappearing from the published literature and that least credible scientific fields publish the highest proportions of significant results (Fanelli, 2010a, 2010b, 2012).”
Smith, G. D., Lawlor, D. A., Harbord, R., Timpson, N., Day, I., & Ebrahim, S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040352 · Cited by 569
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474
Research works printed but not linked: 11
- 2010, 5, 10271, 10.1371/journal.pone.0010271 · DOI · Paper page
- Hedges, 1996, Journal of Educational and Behavioral Statistics, 21, 299, 10.3102/10769986021004299 · doi:10.3102/10769986021004299 · Paper page
- 2005, PLOS MEDICINE, 2, 124, 10.1371/journal.pmed.0020124 · DOI · Paper page
- 2009, Science Translational Medicine, 1, 7
- 2007, PLOS MEDICINE, 4, 79, 10.1371/journal.pmed.0040079 · DOI · Paper page
- 2008, American Journal of Epidemiology, 168, 855, 10.1093/aje/kwn206 · DOI · Paper page
- 2011, JAMA, 308, 1676
- 2011, 6, 18362, 10.1371/journal.pone.0018362 · DOI · Paper page
- 2011, International Journal of Epidemiology, 40, 1280, 10.1093/ije/dyr095 · DOI · Paper page
- 1995, Psychometrika, 60, 419, 10.1007/BF02294384 · DOI · Paper page
- 2011, SIGNIFICANCE, 8, 116, 10.1111/j.1740-9713.2011.00506.x · DOI · Paper page
Cited by
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
Evolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
Self-citationp-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
Self-citationAn estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179
“By using web-based version control, we were able to correct a minor typo identified by a discussant and push the changes to the repository quickly (Ioannidis, 2013).”
InfluentialCurrent Research and Statistical Practices in Sport Science and a Need for Change
Bernards, J., Sato, K., Haff, G., & Bazyler, C. (2017) · Sports · doi:10.3390/sports5040087 · Cited by 120
Metadata
- Authors
- Ioannidis, J. P. A.
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-09-25
- Volume
- 15
- Issue
- 1
- Pages
- 28-36
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- OpenAlex
- W2128564430
- Cited by
- 60
- References
- 15
- FWCI
- 8.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false. https://app.cochat.ai/papers/10.1093/biostatistics/kxt036
- The paper
- Ioannidis, J. P. A. (2014). Discussion: Why "An estimate of the science-wise false discovery rate and application to the top medical literature" is false. Biostatistics. https://doi.org/10.1093/biostatistics/kxt036
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.