CoChat

Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values

Gelman, A., O'Rourke, K. · 2014 · Biostatistics
Cited by 48

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

There has been much unease in1 recent years about our current default system of evaluating and reporting experiments and observational studies. The profusion of dubious and unreplicated claims in subfields ranging from social psychology to brain imaging to medicine has led many observers including ourselves to feel that the scientific publication system is failing. Four flashpoints of this ongoing conversation have been the following: John Ioannidis’s 2005 article on most published findings being false, which resonated both with researchers within applied fields and with outsiders who felt whipsawed by various medical research claims that hit the newspapers but rarely seemed to pan out. Daryl Bem’s much-publicized paper about extra-sensory percerption, which was published in a leading psychology journal despite serious methodological flaws and a highly implausible scientific framework (Bem, 2011; Yarkoni, 2011; Francis, 2012). This case became even more notorious when a failed replication of Bem’s study was denied publication in that same journal on the grounds that the journal “does not publish replications” (French, 2012). Recent scandals about faked research by leading psychologists Mark Hauser and Diederik Stapel, along with reporting such as that on the blog Retraction Watch of low quality control and defensive, non-scientific attitudes among editors, and publishers of scientific journals (Marcus and Oransky, 20102013). The 2009 Duke clinical trial scandal where a critical data processing error (a row slip) was identified by an outside group at MD Anderson and when the Duke clinical trialists were informed of it, they did their best to dismiss, deny, and then obscure it rather than correct it. Following up, the MD Anderson group realized similar errors were all too prevalent in their own research institute and have since taken measures to help prevent and detect them (Baggerly and Coombes, 2009, 2012). In the article under discussion, Jager and Leek present a bold idea to measure the “science-wise false discovery rate” by taking the distribution of published p-values and then applying an estimation procedure used in genetic linkage studies to estimate the proportion of p-values coming from null and alternative distributions. We admire the authors’ energy and creativity, but we find their claims unbelievable (as well as implausibly precise, as, for example, their estimate with a margin of error of 1%) and based on unreasonable assumptions. Jager and Leek may well be correct in their larger point, that the medical literature is broadly correct. To answer such a question would require additional care in defining what it means for a study to be correct. The research paradigm of effects being either zero or non-zero is not, we believe, particularly helpful, for two reasons. First, we almost always care about the direction of an effect, not merely its existence. Second, the magnitude of any effects or comparisons are also important and, in fact, connect directly to concerns about replicability of scientific phenomena. Medical researchers are mostly studying real effects (setting aside certain wacky examples and desperate clinical research areas involving high mortalities). But there is a lot of variation. A new treatment will help in some cases and hurt in others. Also, studies are not perfect; there are various sorts of measurement error and selection bias that creep in, and hence even the occasionally truly zero effect will not be zero in statistical expectation (i.e. with a large enough study, effects will be found). For these reasons, we are not so concerned with Type 1 or Type 2 errors in the classical sense (and as considered by Jager and Leek), rather we worry about Type S errors (someone says an effect is positive when it is actually negative) and Type M errors (someone says an effect is large when it is actually small, or vice versa). For example, the notorious study of beauty and sex ratios described by Gelman and Weakliem (2009) had a Type M error: the claim was an 8% point difference in the probability of a girl (comparing the children of beautiful and non-beautiful parents), but we are sure any actual difference is 0.3% points or less, it could go in either direction, and there is no reason to suppose it persists over time. The point in that example is not that the true effect is or is not zero (thus making the original claim “false” or “true”) but rather that the study is non-informative. If it got the sign right, it is by luck, and in any case it is overestimating the magnitude of any difference by more than an order of magnitude. Our point about Type 1 errors is not primarily “semantics” or “philosophy”. The framework of the Jager and Leek paper under discussion is admirably clear—our problem is that we do not think it applies well to reality. We have a problem with the identification of scientific hypotheses as statistical “hypotheses” of the “|$\theta = 0$|” variety. We understand that the

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/biostatistics/kxt034

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 48

Metadata

Authors
Gelman, A., & O'Rourke, K.
Venue
Biostatistics
Published
2013-09-25
Volume
15
Issue
1
Pages
18-23
ISSN
  • 1465-4644 (ISSN-L)
  • 1468-4357
OpenAlex
W2135905026
Cited by
48
References
4
FWCI
8.25
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.