Needed: A Ban on the Significance Test
Abstract
The significance test as currently used is a disaster Whereas most researchers falsely believe that the significance test has an error rate of 5%, empirical studies show the average error rate across psychology is 60%–12 times higher than researchers think it to be The error rate for inference using the significance test is greater than the error rate using a coin toss to replace the empirical study The significance test has devastated the research review process Comprehensive reviews cite conflicting results on almost every issue Yet quantitatively accurate review of the same results shows that the apparent conflicts stem almost entirely from the high error rate for the significance test If 60% of studies falsely interpret their primary results, then reviewers who base their reviews on the interpreted study “findings” will have a 100% error rate in concluding that there is conflict between study results
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
We currently have no formal models for genetic code evolution.
“We currently have no such formal models for genetic code evolution [11,12,25,27].”
From On universal common ancestry, sequence similarity, and phylogenetic structure: the sins of P-values and the virtues of Bayesian evidence (2011) · cites this paper as [27]
Cites
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.6.848 · Cited by 1,712 · see citing works
Hunter, J. E., & Schmidt, F. L. (2000) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00156 · Cited by 833 · see citing works
Self-citation. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702 · see citing works
The scientific study of dreams: Neural networks, cognitive development, and content analysis.
Domhoff, G. W. (2003) · doi:10.1037/10463-000 · Cited by 355 · see citing works
An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection
Aguinis, H., & Vandenberg, R. J. (2014) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-031413-091231 · Cited by 304 · see citing works
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
Smith, R. A., Levine, T. R., Lachlan, K. A., & Fediuk, T. A. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00821.x · Cited by 77 · see citing works
Rindermann, H., & Ceci, S. J. (2018) · Journal for the Education of the Gifted · doi:10.1177/0162353218799481 · Cited by 68 · see citing works
Theobald, D. L. (2011) · Biology Direct · doi:10.1186/1745-6150-6-60 · Cited by 16 · see citing works
“Frequentist hypothesis tests (of both Fisher and Neyman-Pearson flavors) have been highly criticized and disparaged for many decades by professional statisticians [44-71].”
Background
- Authors
- Hunter, J. E.
- Affiliations
- Michigan State University (US)
- Corresponding author
- Hunter, John E.
- ORCID
- Hunter, John E.
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 1997-01
- Volume
- 8
- Issue
- 1
- Pages
- 3-7
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2101132953
- Cited by
- 314
- References
- 5
- Citations by year
- 2025: 3 · 2024: 3 · 2023: 7 · 2022: 5 · 2021: 4 · 2020: 11 · 2019: 10 · 2018: 12 · 2017: 12 · 2016: 9 · 2015: 9 · 2014: 17 · 2013: 8 · 2012: 7
- FWCI
- 28.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Science (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- significance testing, false positive rate, error rate, meta-analysis, statistical inference, psychology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
