CoChat

Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek

Open access
Benjamini, Y., Hechtlinger, Y. · 2014 · Biostatistics
Cited by 54

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

The original paper by Ioannidis (2005) has done much to raise the awareness to the possible sources of biases in current scientific work. However, Ioannidis described what may be happening in the scientific literature if scientists use blindly the marginal 0.05 significance level rule throughout their work. From this possibility, he reached the alarming conclusion that most research findings are false. In the current work, Prof. Jager and Prof. Leek took it upon themselves to find what actually is happening in current medical research if we assume that the practice is to publish only the findings significant at the 0.05 level. Since Ioannidis was actually arguing about the false discovery rate (FDR) in research (without using this terminology), they utilized FDR methods that are being used in a single research project to study the phenomenon across studies. They further developed modeling tools to address the peculiarities of the data they encountered, such as rounding to specific digits, and they developed relevant text extraction algorithms, allowing them to collect data on the entire population of important medical papers. Finally, they produced the analysis in a reproducible way, which allows everyone to repeat, check, and modify their analysis (as we did). All in all, we congratulate them for shouldering the challenge in such a way. We raise three questions. (1) Does the |$\sim 14\pm 1{\%}$| science-wise FDR reflect the situation in medical research? (2) How can we improve the estimation? (3) Whatever the actual number is, if it is well above the perceived 5% FDR, how can it be brought under control? (1) Is it |$14\pm 1{\%}$|? Unfortunately, this is not clear. There are problems with some of the decisions made as to the statements collected for the analysis: (i) only “|$p <0.05$|” and typographical variations to it, but not “|$p \le 0.05$|”, (ii) only statements regarding significance expressed as “|$p=$|” and typographical variations to it, but not confidence intervals, and (iii) only statements from the Abstracts were analyzed. Each one of these decisions affects the estimate. With regard to decision (i): repeating their analysis (using their available and accessible tools) while moving the threshold slightly to “|$p\le 0.05$|” increased their science-wise FDR estimate to 20.5%, much beyond the reported |$\sim 14 \pm 1{\%}$|⁠. With regard to decision (ii): selection of the significant findings at the 0.05 level is a very common practice even when reporting confidence intervals (CIs). It takes the form of reporting in the abstract the marginal 95% CIs that do not cover 0 (or do not cover 1 for odd ratios and relative risks). Although we did not quantify this phenomenon, many studies report only CIs and hence were entirely excluded from the analysis, with unknown effect on the estimate. More importantly, in some studies that report CIs and |$p$|-values, reported |$p$|-values tend to escort only the more decisive CIs. With regard to decision (iii): examining in depth a sample of 25 papers from those used in the original analysis, 5 from each journal, revealed that the number of significance statements in the body of the papers varied much, from 1 to 68 with a median of 24, while the median of abstract-only data in the same papers was 4, close to the value 2 reported for the entire database. The implication is that there is a strong selection-into-the-abstract effect. As further evidence, we observed that in 19 of the 25 papers that we studied in depth the smallest |$p$|-value in the paper was further selected into the abstract. One result of such a selection is that the distribution of the |$p$|-values of false discoveries in the abstracts that are smaller than 0.05 is stochastically smaller than uniform. The authors did study the implication of this problem on their estimate. From their Figure 5(b), which describes the effect of selecting the smallest of 20 uniformly distributed |$p$|-values into the abstract, one can approximate that the “true science-wise” FDR is about 30% when the estimate is 20%. While Jager and Leek discuss it as an extreme deviation from the assumptions, in view of the above we see it as a realistic or even optimistic scenario. So far we discussed the data being used. There are also some problems with the analysis. The authors treated appropriately the values of 0.04, 0.03, 0.02, and 0.01 as being the result of rounding. But then they avoided using the same approach to address the practice of “rounding on a log-base-10 scale”, even though only 7.8% of the |$p$|-values are at 0.01, while 36.9% are at 0.001, and 8.8% at 0.0001. More importantly, any |$p$|-value reported as “|$p\le x$|” was treated as left censored at |$x$|⁠. This means that if |$p\le 0.04$| is reported, it is treated as |$0.00<p \le 0.04$|⁠, and not as |$0.03

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/biostatistics/kxt032

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. 2006, THE ANNALS OF STATISTICS, 34, 584, 10.1214/009053606000000074 · DOI · Paper page
  2. Fletcher, 2002, JAMA, 288, 366, 10.1001/jama.288.3.366 · DOI · Paper page
  3. 1989, 84, 608

Cited by

See all 54

Metadata

Authors
Benjamini, Y., & Hechtlinger, Y.
Venue
Biostatistics
Published
2013-09-25
Volume
15
Issue
1
Pages
13-16
ISSN
  • 1465-4644 (ISSN-L)
  • 1468-4357
OpenAlex
W2118523264
Cited by
54
References
2
FWCI
8.60
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.