Significance tests as sorcery: Science is empirical—significance tests are not
Abstract
Since the 1930s, many of our top methodologists have argued that significance tests are not conducive to science. Bakan (1966) believed that “everyone knows this” and that we slavishly lean on the crutch of significance testing because, if we didn’t, much of psychology would simply fall apart. If he was right, then significance testing is tantamount to psychology’s “dirty little secret.” This paper will revisit and summarize the arguments of those who have been trying to tell us—for more than 70 years—that p values are not empirical. If these arguments are sound, then the continuing popularity of significance tests in our peer-reviewed journals is at best embarrassing and at worst intellectually dishonest.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Instead of relying on significance testing, effect sizes with 95% confidence intervals were used to assess the path and indirect effects.
“Instead of relying on significance testing, effect sizes with 95% confidence intervals ( CIs ) were used to assess the path and indirect effects (Kline, 2013; Lambdin, 2012).”
From Examining the Prospective Associations Between Mindfulness Facets and Substance Use in Emerging Adulthood. (2021) · cites this paper as Lambdin, 2012
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Perception and the Representative Design of Psychological Experiments
Brunswik, E. (1956) · doi:10.1525/9780520350519 · Cited by 2,974 · see citing works
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2× · see citing works
“…Thomason, Cumming, Finch, & Leeman, 2004; Fisher, 1955, 1956; Gigerenzer, 1987, 1993, 2004; Gigerenzer et al., 1989; Gill, 1999; Granaas, 2002; Greenwald, 1975; Hubbard & Armstrong, 2006; Hubbard & Lindsay, 2008; Hubbard & Ryan, 2000; Jones & Tukey, 2000; Kirk, 1996, 2003; Lindsay, 1995;…”
BackgroundDo studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 3× · see citing works
“Because of this, perhaps the most important type of study that can presently be done in the social sciences is the meta-analysis of large and representative collections of studies (Schmidt, 1992).”
BackgroundThompson, B. (2002) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2002.tb00167.x · Cited by 398 · see citing works
Finch, S., Cumming, G., & Thomason, N. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401612001 · Cited by 52 · cited 1× · see citing works
“(p. 698) Nineteen years later, Finch, Cumming, and Thomason (2001) noted that little had changed in almost two decades.”
Background
Flagged first, then by citations
Cited by
Sorted by citations
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
Malignant side effects of null-hypothesis significance testing
Branch, M. (2014) · Theory & Psychology · doi:10.1177/0959354314525282 · Cited by 81 · see citing works
Single, A., Bilevicius, E., Johnson, E. A., & Keough, M. T. (2019) · Mindfulness · doi:10.1007/s12671-019-1092-7 · Cited by 16 · see citing works
Single, A., & Keough, M. T. (2021) · Alcohol and Alcoholism · doi:10.1093/alcalc/agaa112 · Cited by 10 · see citing works
- Authors
- Lambdin, C.
- Affiliations
- Intel (United States) (US)
- Corresponding author
- Lambdin, Charles
- Venue
- Theory & Psychology
- Publisher
- SAGE Publications
- Published
- 2012-01-25
- Volume
- 22
- Issue
- 1
- Pages
- 67-90
- ISSN
- 0959-3543 (ISSN-L)
- 1461-7447
- OpenAlex
- W2144804691
- Cited by
- 143
- References
- 12
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 7 · 2023: 10 · 2022: 8 · 2021: 7 · 2020: 6 · 2019: 10 · 2018: 18 · 2017: 17 · 2016: 14 · 2015: 14 · 2014: 16 · 2013: 7 · 2012: 2
- FWCI
- 44.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Theory & Psychology (publishedVersion)
- Topics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- significance testing, p-values, statistical inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
