Malignant side effects of null-hypothesis significance testing
The DOI matches its Crossref record; 88 of the 89 references it cites were found in a registry or an index.
Abstract
Six decades-worth of published information has shown irrefutably that null-hypothesis significance tests (NHSTs) provide no information about the reliability of research outcomes. Nevertheless, they are still the core of editorial decision-making in Psychology. Two reasons appear to contribute to the continuing practice. One, survey information suggests that a majority of psychological researchers incorrectly believe that p values provide information about reliability of results. Two, a position sometimes taken is that using them to make decisions has been essentially benign. The mistaken belief has been pointed out many times, so it is briefly covered because of the apparent persistence of the misunderstanding. The idea that NHSTs have been benign is challenged by seven “side-effects” that continue to retard effective development of psychological science. The article concludes with both a few suggestions about possible alternatives and a challenge to psychological researchers to develop new methods that actually assess the reliability of research findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also checked whether the relationships we found were artifacts of between-person differences rather than within-person associations.”
From Context and craving during stressful events in the daily lives of drug-dependent patients (2017) · cites this paper as Branch 2014
In the citing paper
“Even so, when we examined the raw data person by person (Branch 2014), we found that the relationship was not perfect: of the 118 pts who used more than one categorical label for severity across occasions, only 49% gave numerical ratings that increased as would be expected with severity category, for 39%, numerical ratings were flat or fluctuated across categories, and for 12%, numerical ratings”
From Context and craving during stressful events in the daily lives of drug-dependent patients (2017) · cites this paper as Branch 2014
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,499
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“The lack of replicability of statistically significant effects is increasingly being recognized as a clear negative effect of NHSTs in medical research (Ioannidis, 2005).”
BackgroundThe Behavior of Organisms: An Experimental Analysis
Jones, F. N., & Skinner, B. F. (1939) · The American Journal of Psychology · doi:10.2307/1416495 · Cited by 4,398
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Personality: A Psychological Interpretation
Jenkins, J. G., & Allport, G. W. (1938) · The American Journal of Psychology · doi:10.2307/1415717 · Cited by 3,214
Research works printed but not linked: 12
- Beaudette C. G., 2002, Excess heat: Why cold fusion research prevailed
- Berger J. O., 1988, American Scientist, 76, 159
- 10.1080/01621459.1942.10501760 · doi:10.1080/01621459.1942.10501760 · Paper page
- Cohen J., 1988, Statistical power analysis for the behavior sciences
- Committee on CMMP 2010, Solid State Sciences Committee, National Research Council, 2007, Condensed-matter and materials physics: The science of the world around us
- Ebbinghaus H., 1885, Über das Gedächtnis
- 10.1201/9780429258411 · doi:10.1201/9780429258411 · Paper page
- Haller S., 2002, Methods of Psychological Research, 7, 1
- Hays W. L., 1981, Statistics
- 10.1037/0003-066X.56.2.119 · doi:10.1037/0003-066x.56.2.119 · Paper page
- Oakes M., 1986, Statistical inference: A commentary for the social and behavioral sciences
- Piaget J., 1928, The child’s conception of the world
Cited by
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K. E., van Witteloostuijn, A., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 318
“The regression result does not prove or disprove a hypothesis, it does not provide evidence regarding the reliability of the research (Branch, 2014: 257), and it does not make statements about a population other than the sample.”
ResultThe earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
“This is one of the reasons why reliance on significance testing has been accused of systematically retarding the development of cumulative knowledge (Schmidt 1996; Branch 2014).”
BackgroundBettis, R. A., Ethiraj, S., Gambardella, A., Helfat, C., & Mitchell, W. (2016) · Strategic Management Journal · doi:10.1002/smj.2477 · Cited by 225
“P-values arising from NHTSs provide no information “regarding the reliability of the research” (Branch, 2014: 257, citing 13 other papers).”
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 114
“2 We hasten to make clear that this statistical tool is not bad per se, contrary to what might be thought based on the recent debates on the difficulties associated with p-values (e.g., Branch, 2014; Trafimow & Marks, 2015).”
Result60th Anniversary Essay: How Journals Could Improve Research Practices in Social Science
Starbuck, W. H. (2016) · Administrative Science Quarterly · doi:10.1177/0001839216629644 · Cited by 122
“Unfortunately, p-values based on sample data are unreliable evidence about the reproducibility of research findings because they vary quite a bit from sample to sample (Bedeian, Sturman, and Streiner, 2009; Cumming, 2013; Branch, 2014).”
Metadata
- Authors
- Branch, M.
- Affiliations
- University of Florida (US)
- Corresponding author
- Branch, Marc
- ORCID
- Branch, Marc
- Venue
- Theory & Psychology
- Publisher
- SAGE Publications
- Published
- 2014-04
- Volume
- 24
- Issue
- 2
- Pages
- 256-277
- ISSN
- 0959-3543 (ISSN-L)
- 1461-7447
- OpenAlex
- W2335213174
- Cited by
- 81
- References
- 89
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 2 · 2023: 9 · 2022: 7 · 2021: 7 · 2020: 5 · 2019: 7 · 2018: 12 · 2017: 12 · 2016: 9 · 2015: 5
- FWCI
- 12.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Theory & Psychology (publishedVersion)
- Funders
- U.S. Public Health Service
- National Institute on Drug Abuse
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- null hypothesis significance testing, p-values, editorial decision-making, research methodology critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.