Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
The original paper by Ioannidis (2005) has done much to raise the awareness to the possible sources of biases in current scientific work. However, Ioannidis described what may be happening in the scientific literature if scientists use blindly the marginal 0.05 significance level rule throughout their work. From this possibility, he reached the alarming conclusion that most research findings are false. In the current work, Prof. Jager and Prof. Leek took it upon themselves to find what actually is happening in current medical research if we assume that the practice is to publish only the findings significant at the 0.05 level. Since Ioannidis was actually arguing about the false discovery rate (FDR) in research (without using this terminology), they utilized FDR methods that are being used in a single research project to study the phenomenon across studies. They further developed modeling tools to address the peculiarities of the data they encountered, such as rounding to specific digits, and they developed relevant text extraction algorithms, allowing them to collect data on the entire population of important medical papers. Finally, they produced the analysis in a reproducible way, which allows everyone to repeat, check, and modify their analysis (as we did). All in all, we congratulate them for shouldering the challenge in such a way. We raise three questions. (1) Does the |$\sim 14\pm 1{\%}$| science-wise FDR reflect the situation in medical research? (2) How can we improve the estimation? (3) Whatever the actual number is, if it is well above the perceived 5% FDR, how can it be brought under control? (1) Is it |$14\pm 1{\%}$|? Unfortunately, this is not clear. There are problems with some of the decisions made as to the statements collected for the analysis: (i) only “|$p <0.05$|” and typographical variations to it, but not “|$p \le 0.05$|”, (ii) only statements regarding significance expressed as “|$p=$|” and typographical variations to it, but not confidence intervals, and (iii) only statements from the Abstracts were analyzed. Each one of these decisions affects the estimate. With regard to decision (i): repeating their analysis (using their available and accessible tools) while moving the threshold slightly to “|$p\le 0.05$|” increased their science-wise FDR estimate to 20.5%, much beyond the reported |$\sim 14 \pm 1{\%}$|. With regard to decision (ii): selection of the significant findings at the 0.05 level is a very common practice even when reporting confidence intervals (CIs). It takes the form of reporting in the abstract the marginal 95% CIs that do not cover 0 (or do not cover 1 for odd ratios and relative risks). Although we did not quantify this phenomenon, many studies report only CIs and hence were entirely excluded from the analysis, with unknown effect on the estimate. More importantly, in some studies that report CIs and |$p$|-values, reported |$p$|-values tend to escort only the more decisive CIs. With regard to decision (iii): examining in depth a sample of 25 papers from those used in the original analysis, 5 from each journal, revealed that the number of significance statements in the body of the papers varied much, from 1 to 68 with a median of 24, while the median of abstract-only data in the same papers was 4, close to the value 2 reported for the entire database. The implication is that there is a strong selection-into-the-abstract effect. As further evidence, we observed that in 19 of the 25 papers that we studied in depth the smallest |$p$|-value in the paper was further selected into the abstract. One result of such a selection is that the distribution of the |$p$|-values of false discoveries in the abstracts that are smaller than 0.05 is stochastically smaller than uniform. The authors did study the implication of this problem on their estimate. From their Figure 5(b), which describes the effect of selecting the smallest of 20 uniformly distributed |$p$|-values into the abstract, one can approximate that the “true science-wise” FDR is about 30% when the estimate is 20%. While Jager and Leek discuss it as an extreme deviation from the assumptions, in view of the above we see it as a realistic or even optimistic scenario. So far we discussed the data being used. There are also some problems with the analysis. The authors treated appropriately the values of 0.04, 0.03, 0.02, and 0.01 as being the result of rounding. But then they avoided using the same approach to address the practice of “rounding on a log-base-10 scale”, even though only 7.8% of the |$p$|-values are at 0.01, while 36.9% are at 0.001, and 8.8% at 0.0001. More importantly, any |$p$|-value reported as “|$p\le x$|” was treated as left censored at |$x$|. This means that if |$p\le 0.04$| is reported, it is treated as |$0.00<p \le 0.04$|, and not as |$0.03
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters
Benjamini, Y., & Yekutieli, D. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001907 · Cited by 671
Self-citationSimultaneous and selective inference: Current successes and future challenges
Benjamini, Y. (2010) · Biometrical Journal · doi:10.1002/bimj.200900299 · Cited by 169 · cited 1×
“Some of the situations that lack methodologies are selective inference for primary and secondary endpoints (Cohen’s thesis, 2013), safety analysis (Benjamini, 2010), and heterogeneous treatment effect analysis,”
Self-citation
Research works printed but not linked: 3
- 2006, THE ANNALS OF STATISTICS, 34, 584, 10.1214/009053606000000074 · DOI · Paper page
- Fletcher, 2002, JAMA, 288, 366, 10.1001/jama.288.3.366 · DOI · Paper page
- 1989, 84, 608
Cited by
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
Anatomical Abnormalities in Autism?
Haar, S., Berman, S., Behrmann, M., & Dinstein, I. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhu242 · Cited by 292
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215
Self-citationAn estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179
“primary versus secondary) (Ioannidis, 2013; Gelman and O’Rourke, 2013; Goodman, 2013; Benjamini and Hechtlinger, 2013).”
InfluentialIrreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps
Jarvis, M. F., & Williams, M. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2015.12.001 · Cited by 129
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
Metadata
- Authors
- Benjamini, Y., & Hechtlinger, Y.
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-09-25
- Volume
- 15
- Issue
- 1
- Pages
- 13-16
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- OpenAlex
- W2118523264
- Cited by
- 54
- References
- 2
- FWCI
- 8.60
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.