Malignant side effects of null-hypothesis significance testing
The DOI matches its Crossref record; 88 of the 89 references it cites were found in a registry or an index.
Abstract
Six decades-worth of published information has shown irrefutably that null-hypothesis significance tests (NHSTs) provide no information about the reliability of research outcomes. Nevertheless, they are still the core of editorial decision-making in Psychology. Two reasons appear to contribute to the continuing practice. One, survey information suggests that a majority of psychological researchers incorrectly believe that p values provide information about reliability of results. Two, a position sometimes taken is that using them to make decisions has been essentially benign. The mistaken belief has been pointed out many times, so it is briefly covered because of the apparent persistence of the misunderstanding. The idea that NHSTs have been benign is challenged by seven “side-effects” that continue to retard effective development of psychological science. The article concludes with both a few suggestions about possible alternatives and a challenge to psychological researchers to develop new methods that actually assess the reliability of research findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also checked whether the relationships we found were artifacts of between-person differences rather than within-person associations.”
From Context and craving during stressful events in the daily lives of drug-dependent patients (2017) · cites this paper as Branch 2014
In the citing paper
“Even so, when we examined the raw data person by person (Branch 2014), we found that the relationship was not perfect: of the 118 pts who used more than one categorical label for severity across occasions, only 49% gave numerical ratings that increased as would be expected with severity category, for 39%, numerical ratings were flat or fluctuated across categories, and for 12%, numerical ratings”
From Context and craving during stressful events in the daily lives of drug-dependent patients (2017) · cites this paper as Branch 2014
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
David, F. N., & Tukey, J. W. (1977) · Biometrics · doi:10.2307/2529486 · Cited by 12,708
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,499
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“The lack of replicability of statistically significant effects is increasingly being recognized as a clear negative effect of NHSTs in medical research (Ioannidis, 2005).”
BackgroundThe Behavior of Organisms: An Experimental Analysis
Jones, F. N., & Skinner, B. F. (1939) · The American Journal of Psychology · doi:10.2307/1416495 · Cited by 4,398
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Personality: A Psychological Interpretation
Jenkins, J. G., & Allport, G. W. (1938) · The American Journal of Psychology · doi:10.2307/1415717 · Cited by 3,214
Child's Conception of the World: Selected Works vol 1
Piaget, J. (2013) · Routledge eBooks · doi:10.4324/9781315006215 · Cited by 2,675
Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex.
(1930) · Archives of Neurology And Psychiatry · doi:10.1001/archneurpsyc.1930.02220110256018 · Cited by 2,673
(2010) · Encyclopedia of Curriculum Studies · doi:10.4135/9781412958806.n175 · Cited by 2,556
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,947
Conditioned Reflexes: an Investigation of the Physiological Activity of the Cerebral Cortex
DENNY-BROWN, D. (1928) · Nature · doi:10.1038/121662a0 · Cited by 1,836
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Tactics of Scientific Research
Sidman, M. (1960) · Cited by 1,638
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00051.x · Cited by 1,629
Experimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,621
Personality: A Psychological Interpretation
Allport, G. W. (1962) · Cited by 1,578
On the relative motion of the Earth and the luminiferous ether
Michelson, A. A., & Morley, E. W. (1887) · American Journal of Science · doi:10.2475/ajs.s3-34.203.333 · Cited by 1,507
Cleveland, W. S. (1985) · Medical Entomology and Zoology · Cited by 1,425
Helping Doctors and Patients Make Sense of Health Statistics
Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007) · Psychological Science in the Public Interest · doi:10.1111/j.1539-6053.2008.00033.x · Cited by 1,336
Risk factors for suicide in psychiatric outpatients: A 20-year prospective study.
Brown, G. K., Beck, A. T., Steer, R. A., & Grisham, J. R. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.3.371 · Cited by 1,317
Experimenter Effects in Behavioral Research.
Titchener, J. L. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00300050109027 · Cited by 1,194
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,185
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Personality: A Psychological Interpretation.
Chapman, D., & Allport, G. W. (1938) · Sociometry · doi:10.2307/2785590 · Cited by 1,085
Graphs in Statistical Analysis
Anscombe, F. J. (1973) · The American Statistician · doi:10.2307/2682899 · Cited by 1,048
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2×
“…probabilistic information about the reliability of research findings (e.g., Bakan, 1966; Carver, 1978; Cohen, 1994; Falk & Greenbaum, 1995; Gelman, Carlin, Stern, & Rubin, 1995; Gigerenzer, 1993; Greenwald, 1975; Kline, 2004; Lambdin, 2012; Nickerson, 2000; Oakes, 1986; Sohn, 1998; Thompson, 1999).”
BackgroundGraphs in Statistical Analysis
Anscombe, F. J. (1973) · The American Statistician · doi:10.1080/00031305.1973.10478966 · Cited by 976
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
The test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 911
Why Summaries of Research on Psychological Theories are Often Uninterpretable
Meehl, P. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.195 · Cited by 909
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 830
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.2.115 · Cited by 783
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775
The Philosophy of Multiple Comparisons
Tukey, J. W. (1991) · Statistical Science · doi:10.1214/ss/1177011945 · Cited by 773
The cult of statistical significance: how the standard error costs us jobs, justice, and lives
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0372 · Cited by 713
Things I have learned (so far).
Cohen, J. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-028 · Cited by 553
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
A Handbook for data analysis in the behavioral sciences: methodological issues
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0602 · Cited by 543
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 542
Kent, D. M., & Hayward, R. A. (2007) · JAMA · doi:10.1001/jama.298.10.1209 · Cited by 505
Russek, E., & Hays, W. L. (1982) · Journal of the American Statistical Association · doi:10.2307/2287811 · Cited by 460
Strategies and tactics of human behavioral research
Johnston, J. M., Pennypacker, H. S., & Deitz, S. M. (1981) · The Behavior Analyst · doi:10.1007/bf03391853 · Cited by 448
Methods in Psychological ResearchPreprint
Chow, S. L. (2002) · CogPrints (University of Southampton) · Cited by 439
Statistical analysis and the illusion of objectivity
(1988) · Infectious Diseases Newsletter · doi:10.1016/0278-2316(88)90057-6 · Cited by 408
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384 · cited 1×
“That was the advice of Isaac Newton, perhaps anticipating research on experimenter bias (Rosenthal, 1966; Rosenthal & Fode, 1963).”
BackgroundPsychology Will Be a Much Better Science When We Change the Way We Analyze Data
Loftus, G. R. (1996) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep11512376 · Cited by 361
AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms
Thompson, B. (1996) · Educational Researcher · doi:10.2307/1176337 · Cited by 339
Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking
Motulsky, H. J. (2017) · Cited by 334
(1995) · Choice Reviews Online · doi:10.5860/choice.32-6272 · Cited by 304
The Behavior of the P-Value When the Alternative Hypothesis is True
Hung, H. M. J., O'Neill, R. T., Bauer, P., & Kohne, K. (1997) · Biometrics · doi:10.2307/2533093 · Cited by 281
Significance Tests Die Hard: The Amazing Persistence of a Probabilistic Misconception
Falk, R., & Greenbaum, C. W. (1995) · Theory & Psychology · doi:10.1177/0959354395051004 · Cited by 279 · cited 4×
“Therefore it is unlikely I met an American (cf. Cohen, 1994; Falk & Greenbaum, 1995).”
BackgroundMethodInfluentialMisinterpretations of Significance: A Problem Students Share with Their Teachers?
Haller, H., & Krauss, S. (2002) · IPN - Institute for Science Education at the University of Kiel, Germany · doi:10.23668/psycharchives.12769 · Cited by 250
Thompson, B. (1996) · Educational Researcher · doi:10.3102/0013189x025002026 · Cited by 211
The Superego, the Ego, and the Id in Statistical Reasoning
Gigerenzer, G. (2002) · Adaptive Thinking · doi:10.1093/acprof:oso/9780195153729.003.0013 · Cited by 203
Single-participant research design: Bringing science to managed care.
Morgan, D. L., & Morgan, R. K. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.119 · Cited by 188
The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives
Thompson, B. (1993) · The Journal of Experimental Education · doi:10.1080/00220973.1993.10806596 · Cited by 183
Smithson, M. (2003) · Sage eBooks · doi:10.4135/9781412983761 · Cited by 179
Tests of Significance Considered as Evidence
Berkson, J. (1942) · Journal of the American Statistical Association · doi:10.2307/2279000 · Cited by 174
Significance tests as sorcery: Science is empirical—significance tests are not
Lambdin, C. (2012) · Theory & Psychology · doi:10.1177/0959354311429854 · Cited by 143
APA Handbook of Behavior Analysis
Guha, M. (2013) · Reference Reviews · doi:10.1108/rr-05-2013-0106 · Cited by 138
Strategies and tactics of behavioral research, 3rd ed.
Johnston, J. M., & Pennypacker, H. S. (2009) · Cited by 138
On the Surprising Longevity of Flogged Horses: Why There Is a Case for the Significance Test
Abelson, R. P. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00536.x · Cited by 137
Thompson, B. (1994) · Journal of Personality · doi:10.1111/j.1467-6494.1994.tb00289.x · Cited by 129
Confidence intervals and replication: Where will the next mean fall?
Cumming, G., & Maillardet, R. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.3.217 · Cited by 122
What is the probability of replicating a statistically significant effect?
Miller, J. O. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.617 · Cited by 101
Bad science: the short life and weird times of cold fusion
Taubes, G. (1993) · Choice Reviews Online · doi:10.5860/choice.31-2086 · Cited by 89
Odds are, it's wrong: Science fails to face the shortcomings of statistics
Siegfried, T. (2010) · Science News · doi:10.1002/scin.5591770721 · Cited by 84
When Averages Hide Individual Differences in Clinical Trials
Kent, D. M., & Hayward, R. A. (2007) · American Scientist · doi:10.1511/2007.63.60 · Cited by 69
Mittag, K. C., & Thompson, B. (2000) · Educational Researcher · doi:10.3102/0013189x029004014 · Cited by 67
Santi, A., & Wasserman, E. A. (2008) · PsycEXTRA Dataset · doi:10.1037/e552312012-001 · Cited by 58
Perils of Evidence-Based Medicine
Williams, B. A. (2010) · Perspectives in biology and medicine · doi:10.1353/pbm.0.0132 · Cited by 57
Holikatti, P., & Grover, S. (2010) · British Journal of Psychiatry · doi:10.1192/bjp.196.5.415 · Cited by 51
Mittag, K. C., & Thompson, B. (2000) · Educational Researcher · doi:10.2307/1176454 · Cited by 50
Statistical Significance Tests, Effect Size Reporting and the Vain Pursuit of Pseudo-Objectivity
Thompson, B. (1999) · Theory & Psychology · doi:10.1177/095935439992007 · Cited by 50
Brain is Related to Behavior (p < .05)
Zakzanis, K. K. (1998) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.20.3.419.821 · Cited by 47
Statistical Significance and Replicability: Why the Former Does not Presage the Latter
Sohn, D. (1998) · Theory & Psychology · doi:10.1177/0959354398083001 · Cited by 46
Generality and generalization of research findings.
Branch, M. N., & Pennypacker, H. S. (2012) · American Psychological Association eBooks · doi:10.1037/13937-007 · Cited by 41
Self-citationStatistical Hypothesis Testing in Biology: A Contradiction in Terms
Jones, D., & Matloff, N. (1986) · Journal of Economic Entomology · doi:10.1093/jee/79.5.1156 · Cited by 34
Large-scale randomised trials – a misguided approach to clinical research
Penston, J. G. (2004) · Medical Hypotheses · doi:10.1016/j.mehy.2004.09.006 · Cited by 27
Overcoming the Inverse Probability Fallacy: A Comparison of Two Teaching Interventions
Kalinowski, P., Fidler, F. M., & Cumming, G. (2008) · Methodology · doi:10.1027/1614-2241.4.4.152 · Cited by 18
Research works printed but not linked: 12
- Beaudette C. G., 2002, Excess heat: Why cold fusion research prevailed
- Berger J. O., 1988, American Scientist, 76, 159
- 10.1080/01621459.1942.10501760 · doi:10.1080/01621459.1942.10501760 · Paper page
- Cohen J., 1988, Statistical power analysis for the behavior sciences
- Committee on CMMP 2010, Solid State Sciences Committee, National Research Council, 2007, Condensed-matter and materials physics: The science of the world around us
- Ebbinghaus H., 1885, Über das Gedächtnis
- 10.1201/9780429258411 · doi:10.1201/9780429258411 · Paper page
- Haller S., 2002, Methods of Psychological Research, 7, 1
- Hays W. L., 1981, Statistics
- 10.1037/0003-066X.56.2.119 · doi:10.1037/0003-066x.56.2.119 · Paper page
- Oakes M., 1986, Statistical inference: A commentary for the social and behavioral sciences
- Piaget J., 1928, The child’s conception of the world
Cited by
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K. E., van Witteloostuijn, A., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 318
“The regression result does not prove or disprove a hypothesis, it does not provide evidence regarding the reliability of the research (Branch, 2014: 257), and it does not make statements about a population other than the sample.”
ResultThe earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
“This is one of the reasons why reliance on significance testing has been accused of systematically retarding the development of cumulative knowledge (Schmidt 1996; Branch 2014).”
BackgroundBettis, R. A., Ethiraj, S., Gambardella, A., Helfat, C., & Mitchell, W. (2016) · Strategic Management Journal · doi:10.1002/smj.2477 · Cited by 225
“P-values arising from NHTSs provide no information “regarding the reliability of the research” (Branch, 2014: 257, citing 13 other papers).”
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 114
“2 We hasten to make clear that this statistical tool is not bad per se, contrary to what might be thought based on the recent debates on the difficulties associated with p-values (e.g., Branch, 2014; Trafimow & Marks, 2015).”
Result60th Anniversary Essay: How Journals Could Improve Research Practices in Social Science
Starbuck, W. H. (2016) · Administrative Science Quarterly · doi:10.1177/0001839216629644 · Cited by 122
“Unfortunately, p-values based on sample data are unreliable evidence about the reproducibility of research findings because they vary quite a bit from sample to sample (Bedeian, Sturman, and Streiner, 2009; Cumming, 2013; Branch, 2014).”
Metadata
- Authors
- Branch, M.
- Affiliations
- University of Florida (US)
- Corresponding author
- Branch, Marc
- ORCID
- Branch, Marc
- Venue
- Theory & Psychology
- Publisher
- SAGE Publications
- Published
- 2014-04
- Volume
- 24
- Issue
- 2
- Pages
- 256-277
- ISSN
- 0959-3543 (ISSN-L)
- 1461-7447
- OpenAlex
- W2335213174
- Cited by
- 81
- References
- 89
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 2 · 2023: 9 · 2022: 7 · 2021: 7 · 2020: 5 · 2019: 7 · 2018: 12 · 2017: 12 · 2016: 9 · 2015: 5
- FWCI
- 12.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Theory & Psychology (publishedVersion)
- Funders
- U.S. Public Health Service
- National Institute on Drug Abuse
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- null hypothesis significance testing, p-values, editorial decision-making, research methodology critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.