The importance of proving the null
Open accessAbstract
Null hypotheses are simple, precise, and theoretically important. Conventional statistical analysis cannot support them; Bayesian analysis can. The challenge in a Bayesian analysis is to formulate a suitably vague alternative, because the vaguer the alternative is (the more it spreads out the unit mass of prior probability), the more the null is favored. A general solution is a sensitivity analysis: Compute the odds for or against the null as a function of the limit(s) on the vagueness of the alternative. If the odds on the null approach 1 from above as the hypothesized maximum size of the possible effect approaches 0, then the data favor the null over any vaguer alternative to it. The simple computations and the intuitive graphic representation of the analysis are illustrated by the analysis of diverse examples from the current literature. They pose 3 common experimental questions: (a) Are 2 means the same? (b) Is performance at chance? (c) Are factors additive?
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Bayesian approach provides an opportunity to analytically determine whether the null hypothesis is more likely than the alternative(s) given the observed data.
“An advantage of the Bayesian approach is that it provides an opportunity to analytically determine whether the null hypothesis is more likely than the alternative(s) given the observed data (e.g., Raftery, 1995; Gallistel, 2009).”
From Cognitive Mapping Without Vision: Comparing Wayfinding Performance After Learning From Digital Touchscreen-Based Multimodal Maps vs. Embossed Tactile Overlays (2020) · cites this paper as Gallistel, 2009
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · see citing works
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
A theory of attention: Variations in the associability of stimuli with reinforcement
Mackintosh, N. J. (1975) · Psychological Review · doi:10.1037/h0076778 · Cited by 2,733 · see citing works
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
DiCiccio, T., Kass, R., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 2,029 · see citing works
Gallistel, C. R., & Gibbon, J. (2000) · Psychological Review · doi:10.1037/0033-295x.107.2.289 · Cited by 1,122 · see citing works
Self-citationInformation Processing in Animals
(2014) · Psychology Press eBooks · doi:10.4324/9781315798820 · Cited by 902 · see citing works
Autoshaping and Conditioning Theory
Perkins, C. C., Locurto, C. W., Terrace, H. S., & Gibbon, J. (1982) · The American Journal of Psychology · doi:10.2307/1422674 · Cited by 864 · see citing works
The Automaticity of Visual Statistical Learning.
Turk‐Browne, N. B., Junge, J. A., & Scholl, B. J. (2005) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.134.4.552 · Cited by 817 · see citing works
Langendoen, D. T. (1997) · Language · doi:10.2307/417339 · Cited by 756 · see citing works
The problem of inference from curves based on group data.
Estes, W. K. (1956) · Psychological Bulletin · doi:10.1037/h0045156 · Cited by 677 · see citing works
An overview of robust Bayesian analysis
Berger, J. O., Moreno, E., Pericchi, L. R., Bayarri, M. J., Bernardo, J. M., Cano, J. A., de la Horra, J., Martín, J., Insua, D. R., Betrò, B., et al. (1994) · Test · doi:10.1007/bf02562676 · Cited by 591 · see citing works
The learning curve: implications of a quantitative analysis.
Gallistel, C., Fairhurst, S. P., & Balsam, P. (2004) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.0404965101 · Cited by 587 · see citing works
Self-citationMeyer, D. E., & Kornblum, S. (1993) · Cited by 523 · see citing works
Applying Occam’s razor in modeling cognition: A Bayesian approach
Myung, I. J., & Pitt, M. A. (1997) · Psychonomic Bulletin & Review · doi:10.3758/bf03210778 · Cited by 409 · see citing works
Temporal maps and informativeness in associative learning
Balsam, P. D., & Gallistel, C. R. (2009) · Trends in Neurosciences · doi:10.1016/j.tins.2008.10.004 · Cited by 264 · see citing works
Self-citationThree case studies in the Bayesian analysis of cognitive models
Lee, M. D. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.1.1 · Cited by 176 · see citing works
Weight of Evidence, Corroboration, Explanatory Power, Information and the Utility of Experiments
Good, I. J. (1960) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1960.tb00378.x · Cited by 154 · see citing works
Discovering mental processing stages: The method of additive factors.
Sternberg, S. (1998) · Cited by 144 · see citing works
Estes, W. K., & Maddox, W. T. (2005) · Psychonomic Bulletin & Review · doi:10.3758/bf03193784 · Cited by 141 · see citing works
The Meaning of Additive Reaction-Time Effects: Tests of Three Alternatives
Roberts, S., & Sternberg, S. (1993) · The MIT Press eBooks · doi:10.7551/mitpress/1477.003.0038 · Cited by 129 · see citing works
Effect of unconditioned stimulus magnitude on the emergence of conditioned responding.
Morris, R., & Bouton, M. (2006) · Journal of experimental psychology. Animal behavior processes · doi:10.1037/0097-7403.32.4.371 · Cited by 64 · see citing works
Rosenbaum, D. Α., Halloran, E. S., & Cohen, R. G. (2006) · Psychonomic Bulletin & Review · doi:10.3758/bf03194019 · Cited by 63 · see citing works
Methods, models, and conceptual issues
Scarborough, D. L., & Sternberg, S. (1998) · Medical Entomology and Zoology · Cited by 60 · see citing works
AUTOSHAPED HEAD POKING IN THE MOUSE: A QUANTITATIVE ANALYSIS OF THE LEARNING CURVE
Papachristos, E. B., & Gallistel, C. R. (2006) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2006.71-05 · Cited by 54 · see citing works
Self-citationIs the number of trials a primary determinant of conditioned responding?
Gottlieb, D. A. (2008) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.34.2.185 · Cited by 47 · see citing works
Replicability, Confidence, and Priors
Killeen, P. R. (2005) · Psychological Science · doi:10.1111/j.1467-9280.2005.01653.x · Cited by 46 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Jeffreys H 1961 Theory of probability 3rd Oxford, England Oxford University Press
- Kass RE Raftery AE 1995 Bayes factors Journal of the American Statistical Association 90 773 795
Cited by
Sorted by citations
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
Default Bayes factors for ANOVA designs
Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.08.001 · Cited by 1,800 · see citing works
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288 · see citing works
What are the odds? A practical guide to computing and reporting Bayes factors
Jarosz, A. F., & Wiley, J. (2014) · The Journal of Problem Solving · doi:10.7771/1932-6246.1167 · Cited by 1,101 · see citing works
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 <i>t</i> Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
A tutorial on a practical Bayesian alternative to null-hypothesis significance testing
Masson, M. E. J. (2011) · Behavior Research Methods · doi:10.3758/s13428-010-0049-5 · Cited by 752 · see citing works
Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence
Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0660-4 · Cited by 654 · see citing works
Doing Bayesian Data Analysis: A Tutorial with R and BUGS
Barry, J. (2011) · Europe’s Journal of Psychology · doi:10.5964/ejop.v7i4.163 · Cited by 563 · see citing works
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492 · see citing works
Default Bayes Factors for Model Selection in Regression
Rouder, J. N., & Morey, R. D. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.734737 · Cited by 477 · see citing works
Beyond good and evil: The time-course of neural activity elicited by specific picture content.
Weinberg, A., & Hajcak, G. (2010) · Emotion · doi:10.1037/a0020242 · Cited by 425 · see citing works
Kragel, P. A., Kano, M., Van Oudenhove, L., Ly, H. G., Dupont, P., Rubio, A., Delon-Martin, C., Bonaz, B. L., Manuck, S. B., Gianaros, P. J., et al. (2018) · Nature Neuroscience · doi:10.1038/s41593-017-0051-7 · Cited by 272 · see citing works
Representation of aversive prediction errors in the human periaqueductal gray
Roy, M., Shohamy, D., Daw, N., Jepma, M., Wimmer, G. E., & Wager, T. D. (2014) · Nature Neuroscience · doi:10.1038/nn.3832 · Cited by 246 · see citing works
A Bayes factor meta-analysis of Bem’s ESP claim
Rouder, J. N., & Morey, R. D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0088-7 · Cited by 194 · see citing works
Papies, E. K., Pronk, T. M., Keesman, M., & Barsalou, L. W. (2015) · Journal of Personality and Social Psychology · doi:10.1037/a0038032 · Cited by 197 · see citing works
Distinct opioid circuits determine the palatability and the desirability of rewarding events
Wassum, K. M., Ostlund, S. B., Maidment, N. T., & Balleine, B. W. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905874106 · Cited by 182 · see citing works
Bayesian inference for psychology, part IV: parameter estimation and Bayes factors
Rouder, J. N., Haaf, J. M., & Vandekerckhove, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1420-7 · Cited by 102 · see citing works
Pantelis, P. C., Byrge, L., Tyszka, J. M., Adolphs, R., & Kennedy, D. P. (2015) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsv021 · Cited by 97 · see citing works
Giudice, N. A., Guenther, B. A., Jensen, N. A., & Haase, K. N. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.00087 · Cited by 60 · see citing works
“An advantage of the Bayesian approach is that it provides an opportunity to analytically determine whether the null hypothesis is more likely than the alternative(s) given the observed data (e.g., Raftery, 1995; Gallistel, 2009).”
Animal models of episodic memory
Crystal, J. D. (2018) · Comparative Cognition & Behavior Reviews · doi:10.3819/ccbr.2018.130012 · Cited by 59 · see citing works
Wilson, A. G., & Crystal, J. D. (2012) · Animal Cognition · doi:10.1007/s10071-011-0459-5 · Cited by 50 · see citing works
Rouder, J. N., Morey, R. D., & Province, J. M. (2013) · Psychological Bulletin · doi:10.1037/a0029008 · Cited by 48 · see citing works
Disruption of endogenous opioid activity during instrumental learning enhances habit acquisition
Wassum, K. M., Cely, I. C., Maidment, N. T., & Balleine, B. W. (2009) · Neuroscience · doi:10.1016/j.neuroscience.2009.06.071 · Cited by 46 · see citing works
Tidwell, J. W., Dougherty, M. R., Chrabaszcz, J. R., Thomas, R. P., & Mendoza, J. L. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0560-7 · Cited by 45 · see citing works
- Authors
- Gallistel, C. R.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Corresponding author
- Gallistel, C. R.
- ORCID
- Gallistel, C. R.
- Venue
- Psychological Review
- Published
- 2009
- Volume
- 116
- Issue
- 2
- Pages
- 439-453
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- PMC
- PMC2859953
- OpenAlex
- W2104128511
- Cited by
- 392
- References
- 30
- Citations by year
- 2026: 10 · 2025: 7 · 2024: 15 · 2023: 6 · 2022: 13 · 2021: 21 · 2020: 15 · 2019: 27 · 2018: 35 · 2017: 42 · 2016: 39 · 2015: 29 · 2014: 27 · 2013: 32 · 2012: 23
- FWCI
- 19.78
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Psychological Review (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (R01MH077027)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- null hypothesis testing, Bayesian hypothesis testing, sensitivity analysis, prior probability, equivalence testing, chance performance, additive effects
- MeSH terms
- Attention, Bayes Theorem, Conditioning, Classical, Humans, Learning, Psychomotor Performance, Likelihood Functions, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
