Improving inferences about null effects with Bayes factors and equivalence tests.
Open accessAbstract
Researchers often conclude an effect is absent when a null-hypothesis significance test yields a nonsignificant p value. However, it is neither logically nor statistically correct to conclude an effect is absent when a hypothesis test is not significant. We present two methods to evaluate the presence or absence of effects: Equivalence testing (based on frequentist statistics) and Bayes factors (based on Bayesian statistics). In four examples from the gerontology literature, we illustrate different ways to specify alternative models that can be used to reject the presence of a meaningful or predicted effect in hypothesis tests. We provide detailed explanations of how to calculate, report, and interpret Bayes factors and equivalence tests. We also discuss how to design informative studies that can provide support for a null model or for the absence of a meaningful effect. The conceptual differences between Bayes factors and equivalence tests are discussed, and we also note when and why they might lead to similar or different inferences in practice. It is important that researchers are able to falsify predictions or can quantify the support for predicted null effects. Bayes factors and equivalence tests provide useful statistical tools to improve inferences about null effects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A Bayesian mixed model ANOVA on the whole sample revealed a Bayes factor (BF10) of 0.14.
“The analysis revealed a Bayes factor (BF10) of 0.14, providing weak support for a possible significant interaction effect (Lakens et al., 2020).”
From Retrieval Practice Is Effective Regardless of Self-Reported Need for Cognition - Behavioral and Brain Imaging Evidence (2022) · cites this paper as Lakens et al., 2020
In “Testing Predictions Using Bayes Factors and Equivalence Tests” (page unavailable), the paper says Bayes factors between 0 and 1 favor the null over the alternative. Its “Bayes Factors” section interprets a similar value, 0.13, as strong evidence for the null—not weak support for an interaction effect. The cited paper does not report the citing authors’ mixed-model ANOVA or its BF10 of 0.14. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
Jaeschke, R., Singer, J., & Guyatt, G. H. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90005-6 · Cited by 4,982 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Dworkin, R. H., Turk, D. C., Wyrwich, K. W., Beaton, D., Cleeland, C. S., Farrar, J. T., Haythornthwaite, J. A., Jensen, M. P., Kerns, R. D., Ader, D. N., et al. (2008) · The Journal of Pain · doi:10.1016/j.jpain.2007.09.005 · Cited by 3,471 · see citing works
When to use the <scp>B</scp> onferroni correction
Armstrong, R. A. (2014) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12131 · Cited by 3,031 · see citing works
Schuirmann, D. J. (1987) · Journal of Pharmacokinetics and Biopharmaceutics · doi:10.1007/bf01068419 · Cited by 2,696 · see citing works
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161 · cited 7× · see citing works
“We provide reproducible R scripts for all of the examples as well as instructions for how to calculate Bayes factors and equivalence tests from summary statistics or raw data in online calculators (Dienes, 2008; Rouder, Speckman, Sun, Morey, & Iverson, 2009), simple spreadsheets (Lakens, 2017), the new statistical poi…”
BackgroundMethodInfluentialSelf-citationUsing Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
Self-citationStatistics notes: Absence of evidence is not evidence of absence
Altman, D. G., & Bland, J. M. (1995) · BMJ · doi:10.1136/bmj.311.7003.485 · Cited by 1,946 · see citing works
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 2× · see citing works
“However, when building on effect sizes from the published literature, publication bias and “follow-up bias” (Albers & Lakens, 2018) often lead to inflated effect sizes.”
BackgroundSelf-citationWhy Psychologists Should by Default Use Welch’s t-test Instead of Student’s t-test
Delacre, M., Lakens, D., & Leys, C. (2017) · International Review of Social Psychology · doi:10.5334/irsp.82 · Cited by 1,037 · see citing works
Self-citationKelly, A.-M. (2001) · Emergency Medicine Journal · doi:10.1136/emj.18.3.205 · Cited by 969 · see citing works
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · see citing works
Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.12.001 · Cited by 740 · see citing works
Rejecting or Accepting Parameter Values in Bayesian Estimation
Kruschke, J. K. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771304 · Cited by 734 · cited 2× · see citing works
“The TOST approach is equivalent to examining whether a 90% CI (for alpha = 0.05) around the effect falls between ΔL and ΔU, and concluding equivalence if the 90% CI does not contain either equivalence bound (Westlake, 1972; for a related Bayesian procedure, see Kruschke, 2011, 2018).”
BackgroundMethodInfluentialBayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 732 · see citing works
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619 · see citing works
Using significance tests to evaluate equivalence between two experimental groups.
Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.113.3.553 · Cited by 568 · see citing works
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 498 · see citing works
Speed-of-Processing and Driving Simulator Training Result in Improved Driving Performance
Roenker, D. L., Cissell, G. M., Ball, K. K., Wadley, V. G., & Edwards, J. D. (2003) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.45.2.218.27241 · Cited by 469 · see citing works
Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference
Dienes, Z. P. (2008) · Cited by 418 · see citing works
Self-citationAlbers, C. J., & Lakens, D. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.09.004 · Cited by 366 · see citing works
Self-citationHow Bayes factors change scientific practice
Dienes, Z. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.10.003 · Cited by 364 · see citing works
Self-citationFour reasons to prefer Bayesian analyses over significance testing
Dienes, Z. P., & McLatchie, N. M. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1266-z · Cited by 355 · see citing works
Self-citationThe philosophy of Bayes factors and the quantification of statistical evidence
Morey, R. D., Romeijn, J., & Rouder, J. N. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.001 · Cited by 307 · see citing works
Use of Confidence Intervals in Analysis of Comparative Bioavailability Trials
Westlake, W. J. (1972) · Journal of Pharmaceutical Sciences · doi:10.1002/jps.2600610845 · Cited by 286 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Distract or reappraise? Age-related differences in emotion-regulation choice.
Scheibe, S., Sheppes, G., & Staudinger, U. M. (2015) · Emotion · doi:10.1037/a0039246 · Cited by 190 · see citing works
Prevalence and Relevance of Pain in Older Persons
Gibson, S. J., & Lussier, D. (2012) · Pain Medicine · doi:10.1111/j.1526-4637.2012.01349.x · Cited by 173 · see citing works
Groot, A. D. D., & Spiekerman, J. A. A. (1969) · De Gruyter eBooks · doi:10.1515/9783112313121 · Cited by 169 · see citing works
Predictors of ageing-related decline across multiple cognitive functions
Ritchie, S. J., Tucker-Drob, E. M., Cox, S. R., Corley, J., Dykiert, D., Redmond, P., Pattie, A., Taylor, A. M., Sibbett, R., Starr, J. M., et al. (2016) · Intelligence · doi:10.1016/j.intell.2016.08.007 · Cited by 147 · see citing works
Meyners, M. (2012) · Food Quality and Preference · doi:10.1016/j.foodqual.2012.05.003 · Cited by 130 · cited 3× · see citing works
“For a more extensive introduction to the TOST procedure, see Meyners (2012), Rogers and colleagues (1993), or Lakens (2017).”
BackgroundMethodReward-Enhanced Memory in Younger and Older Adults
Spaniol, J., Schain, C., & Bowen, H. J. (2013) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbt044 · Cited by 120 · see citing works
The Relation of Ego Integrity and Despair to Personality Traits and Mental Health
Westerhof, G. J., Bohlmeijer, E. T., & McAdams, D. P. (2015) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbv062 · Cited by 76 · see citing works
Scientific Reporting and Personal Probabilities: Student's Hypothesis
Dickey, J. (1973) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1973.tb00959.x · Cited by 65 · see citing works
Martins, B. S., Sheppes, G., Gross, J., & Mather, M. (2016) · The journals of gerontology. Series B, Psychological sciences and social sciences · doi:10.1093/geronb/gbw028 · Cited by 55 · see citing works
Long-term Associations Between Physical Frailty and Performance in Specific Cognitive Domains
Bunce, D. J., Batterham, P. J., & Mackinnon, A. J. (2017) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbx177 · Cited by 52 · see citing works
Shega, J. W., Tiedt, A. D., Grant, K., & Dale, W. (2014) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbu101 · Cited by 39 · see citing works
Why optional stopping is a problem for BayesiansPreprint
de Heide, R., & Grünwald, P. (2017) · arXiv (Cornell University) · Cited by 15 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [20]20. Jeffreys, H. (1939). Theory of probability. Oxford: Clarendon.
- [27]27. Lenth, R. V. (2007). Post hoc power: Tables and commentary. Iowa City: Department of Statistics and Actuarial Science, University of Iowa. Retrieved from https://pdfs.semanticscholar.org/fbfb/ cab4b59e54c6a3ed39ba3656f35ef86c5ee3.pdf
Cited by
Sorted by citations
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178 · see citing works
Blake, K. R., & Gangestad, S. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220913363 · Cited by 159 · see citing works
Assessing expectancy and suggestibility in a trial of escitalopram v. psilocybin for depression
Szigeti, B., Weiss, B., Rosas, F. E., Erritzoe, D., Nutt, D., & Carhart-Harris, R. (2024) · Psychological Medicine · doi:10.1017/s0033291723003653 · Cited by 89 · see citing works
Pennington, C. R., Monk, R. L., Heim, D., Rose, A. K., Gough, T., Clarke, R., Knibb, G., Patel, R., Rai, P., Ravat, H., et al. (2023) · Psychology of Addictive Behaviors · doi:10.1037/adb0000919 · Cited by 22 · see citing works
Wiklund-Hörnqvist, C., Stillesjö, S., Andersson, M., Jonsson, B., & Nyberg, L. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.797395 · Cited by 12 · see citing works
- Authors
- Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z.
- Affiliations
- Eindhoven University of Technology (NL)
- Lancaster University (GB)
- University of Sussex (GB)
- Corresponding author
- Lakens, Daniël
- Venue
- The Journals of Gerontology: Series B
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-06-06
- Volume
- 75
- Issue
- 1
- Pages
- 45-57
- ISSN
- 1079-5014 (ISSN-L)
- 1758-5368
- OpenAlex
- W2803536098
- Cited by
- 279
- References
- 42
- Citations by year
- 2026: 22 · 2025: 27 · 2024: 29 · 2023: 34 · 2022: 46 · 2021: 48 · 2020: 36 · 2019: 27 · 2018: 10
- FWCI
- 25.10
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- The Journals of Gerontology Series B (publishedVersion)
- PubMed (publishedVersion)
- TU/e Research Portal (publishedVersion, other-oa)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Sussex Research Online (University of Sussex) (submittedVersion)
- TU/e Research Portal (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Figshare (submittedVersion, cc-by)
- TU/e Research Portal (submittedVersion, other-oa)
- Funders
- Netherlands Organization for Scientific Research (452-17-013)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- Bayes factor, equivalence testing, null effect, null hypothesis significance testing, alternative models, research design, gerontological research, Bayesian inference
- MeSH terms
- Emotional Regulation, Adult, Aged, Aging, Bayes Theorem, Data Interpretation, Statistical, Geriatrics, Humans, Male, Memory, Personality, Psychology, Research Design, Models, Statistical, Biomedical Research, Chronic Pain
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
