Statistical Significance and the Dichotomization of Evidence
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
In light of recent concerns about reproducibility and replicability, the ASA issued a Statement on Statistical Significance and p-values aimed at those who are not primarily statisticians. While the ASA Statement notes that statistical significance and p-values are “commonly misused and misinterpreted,” it does not discuss and document broader implications of these errors for the interpretation of evidence. In this article, we review research on how applied researchers who are not primarily statisticians misuse and misinterpret p-values in practice and how this can lead to errors in the interpretation of evidence. We also present new data showing, perhaps surprisingly, that researchers who are primarily statisticians are also prone to misuse and misinterpret p-values thus resulting in similar errors. In particular, we show that statisticians tend to interpret evidence dichotomously based on whether or not a p-value crosses the conventional 0.05 threshold for statistical significance. We discuss implications and offer recommendations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 5×
“Statisticians and others have been sounding the alarm about these matters for decades, to little avail” (Wasserstein and Lazar 2016).”
BackgroundMethodInfluentialCohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007) · Child Development · doi:10.1111/j.1467-8624.2007.00995.x · Cited by 3,784
The SAGE Handbook of Quantitative Methodology for the Social Sciences
Kaplan, D. (2004) · doi:10.4135/9781412986311 · Cited by 1,454
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 2×
“An example of dichotomous thinking is provided by Gelman and Stern (2006), who show applied researchers often fail to appreciate that “the difference between ‘significant’ and ‘not significant’ is not itself statistically significant.””
BackgroundMethodBem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · cited 2×
“…has been well-investigated by statisticians over the decades (Diaconis 1978; Diaconis and Graham 1981; Diaconis andMosteller 1989; Briggs 2006) and which has surfaced again recently (Bem 2011)—that there is strong evidence for the existence of psychic powers such as extrasensory perception.”
BackgroundGigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · cited 7×
“For example, Gigerenzer (2004) reported an example of research conducted on psychology professors, lecturers, teaching assistants, and students (see also Haller and Krauss (2002), Oakes (1986), and Gigerenzer, Krauss, and Vitouch (2004)).”
BackgroundMethodResultInfluentialErroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · cited 1×
“…and Schindler (2016) as well as Thorstenson, Pazda and Elliot (2015), which was retracted for this error after being discussed on the blog), while Nieuwenhuis, Forstmann, and Wagenmakers (2011) documented that it is rife in neuroscience, appearing in half of neuroscience papers in top journals…”
Background
Cited by
Moving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
Moving beyond P values: data analysis with estimation graphics
Ho, J., Tumkaya, T., Aryal, S., Choi, H., & Claridge-Chang, A. (2019) · Nature Methods · doi:10.1038/s41592-019-0470-3 · Cited by 1,758
Abandon Statistical Significance
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019) · The American Statistician · doi:10.1080/00031305.2018.1527253 · Cited by 791
Self-citationSensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
Greenland, S. (2019) · The American Statistician · doi:10.1080/00031305.2018.1529625 · Cited by 313
Metadata
- Authors
- McShane, B. B., & Gal, D.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2017-07-03
- Volume
- 112
- Issue
- 519
- Pages
- 885-895
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2765781497
- Cited by
- 166
- References
- 15
- FWCI
- 13.95
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical Significance and the Dichotomization of Evidence. https://app.cochat.ai/papers/10.1080/01621459.2017.1289846
- The paper
- McShane, B. B., & Gal, D. (2017). Statistical Significance and the Dichotomization of Evidence. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2017.1289846
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.