Are there benefits from NHST?
Abstract
Krueger’s (January 2001) thesis can be summarized as follows: Induction is central to science (true). From a philosophical perspective, induction cannot be defended logically but can be defended pragmatically— because it leads to progress in science (true). Null hypothesis significance testing (NHST) is a good tool—perhaps the essential tool— for induction and inference from research data (false). NHST is like induction: It cannot be defended logically (true) but can be defended pragmatically (false). The pragmatic defense Krueger offered is the contention that NHST has “proven useful” (p. 24) and that it “rewards the pragmatic scientist” (p. 23; both false). Induction as used by Krueger (2001) is the same as hypothesis testing; it is undisputed that hypothesis testing is indispensable for science. Krueger’s position, then, reduces to the proposition that NHST is the best procedure—and perhaps the essential procedure— for testing hypotheses. This false argument has long been offered as a defense of significance testing (Schmidt & Hunter, 1997, pp. 42–44). In its strong form, the argument is that without significance testing, psychologists could not have a science because they would no longer be able to test hypotheses. The physical sciences, such as physics and chemistry, do not use NHST or statistical significance tests of any kind, yet these sciences test hypotheses and have done so for centuries. In fact, in contrast to many psychologists, most researchers in the physical sciences regard reliance on significance testing as unscientific (Schmidt, 1996). If the argument is that hypothesis testing requires the use of significance tests, then the logical implications are that physicists and chemists are not really testing hypotheses and that their research is not really scientific. How plausible is this? If the argument is that significance testing is the best method of testing hypotheses in science, then the logical implication is that the hypothesis-testing methods used in physics and chemistry are suboptimal and inferior to those based on NHST and typically used in psychology. Does anyone really believe this? Hence, although induction (hypothesis testing) is central to science, neither NHST nor any other form of statistical significance testing is required for hypothesis testing. Significance testing almost invariably retards the search for knowledge by producing false conclusions about research literature. The evidence is strong that the null hypothesis is almost always false in psychological research. For example, Lipsey and Wilson (1993) examined 302 meta-analyses of psychological interventions of all kinds in many areas of psychology. In only 2 of these meta-analyses were the effect sizes (ESs) zero or near zero (less than 1%). An examination of all published meta-analyses would produce a similar figure for psychology as a whole. If the null hypothesis is typically false, then Type I error is not important because it is impossible to make a Type I error when the null is false. What is important is Type II error: failing to detect the effect or relation that is there. One minus the Type II error rate is the statistical power of the study: the probability of detecting the effect or relation. The evidence is clear that the average level of statistical power in psychological research is between .40 and .60 (e.g., see Cohen, 1962, 1994; Sedlmeier & Gigerenzer, 1989). The operational decision rule used by researchers is “if it is significant, it is real; if it is not significant, it is zero” (Schmidt, 1996). Hence, the error rate in the typical psychological research literature is approximately 50%—that is, half of all studies reach false conclusions about the null hypothesis, a situation of maximal apparent conflict in the literature. As discussed by Schmidt (1996), this leads to one of two false conclusions about the meaning of the literature. The first is that the literature is so conflicting that nothing can be concluded. The second is that there are interactions or moderator variables that cause the effect to exist in some studies and to be nonexistent in others and that research should be directed at finding these moderator variables. Meta-analysis typically indicates that both of these conclusions are false—by revealing that the effect exists in all studies (Schmidt, 1996). Significance tests are a disastrous method for testing hypotheses, but a better method does exist: use of point estimates (ESs) and confidence intervals (CIs). First, unlike significance tests, CIs hold the real error rate to .05 (or whatever confidence level is set); there is no possibility of a higher error rate as with significance tests. In particular, the true error rate will never be 50% when the researcher thinks it is 5% (because the alpha level is set at .05). Second, almost all of the CIs from different studies overlap each other, correctly suggesting that the studies are not contradictory. Third, the CI clearly reveals
Cites (4)
- The earth is round (p < .05).Cohen, Jacob · 1994 · American Psychologist
- The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis.Lipsey, Mark W., Wilson, David B. · 1993 · American Psychologist
- The statistical power of abnormal-social psychological research: A review.Cohen, Jacob · 1962 · The Journal of Abnormal and Social Psychology
- Do studies of statistical power have an effect on the power of studies?Sedlmeier, Peter, Gigerenzer, Gerd · 1989 · Psychological Bulletin
Cited by (6)
- Conspiracy Beliefs, Rejection of Vaccination, and Support for hydroxychloroquine: A Conceptual Replication-Extension in the COVID-19 Pandemic ContextBertin, Paul, Nera, Kenzo, Delouvée, Sylvain · 2020 · Frontiers in Psychology
- TARGET ARTICLE: Does the Positive Psychology Movement Have Legs?Lazarus, Richard S. · 2003 · Psychological Inquiry
- The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful ArticlesO’Boyle, Ernest Hugh, Banks, George Christopher, Gonzalez-Mulé, Erik · 2017 · Journal of Management
- When can we Conclude that Treatments or Programs “Don’t Work”?Weisburd, David, Lum, Cynthia M., Yang, Sue-Ming · 2003 · The ANNALS of the American Academy of Political and Social Science
- Untitled work2006 · PubMed
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Are there benefits from NHST? https://app.cochat.ai/papers/10.1037//0003-066x.57.1.65
- The paper
- Schmidt, F., & Hunter, J. (2002). Are there benefits from NHST? American Psychologist. https://doi.org/10.1037//0003-066x.57.1.65
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.