Indices of Effect Existence and Significance in the Bayesian Framework
Open accessPublished version of a preprint
Abstract
Turmoil has engulfed psychological science. Causes and consequences of the reproducibility crisis are in dispute. With the hope of addressing some of its aspects, Bayesian methods are gaining increasing attention in psychological science. Some of their advantages, as opposed to the frequentist framework, are the ability to describe parameters in probabilistic terms and explicitly incorporate prior knowledge about them into the model. These issues are crucial in particular regarding the current debate about statistical significance. Bayesian methods are not necessarily the only remedy against incorrect interpretations or wrong conclusions, but there is an increasing agreement that they are one of the keys to avoid such fallacies. Nevertheless, its flexible nature is its power and weakness, for there is no agreement about what indices of “significance” should be computed or reported. This lack of a consensual index or guidelines, such as the frequentist p-value, further contributes to the unnecessary opacity that many non-familiar readers perceive in Bayesian statistics. Thus, this study describes and compares several Bayesian indices, provide intuitive visual representation of their “behavior” in relationship with common sources of variance such as sample size, magnitude of effects and also frequentist significance. The results contribute to the development of an intuitive understanding of the values that researchers report, allowing to draw sensible recommendations for Bayesian statistics description, critical for the standardization of scientific reporting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The percentage in region of practical equivalence (ROPE) indexes the magnitude of an effect where the ROPE is the range of effect size values considered to be practically equivalent to the null.
“The percentage in ROPE indexes the magnitude of an effect where the ROPE is the range of effect size values considered to be practically equivalent to the null [302, 305].”
From Infraslow closed-loop brain training for anxiety and depression (ISAD): a protocol for a randomized, double-blind, sham-controlled pilot trial in adult females with internalizing disorders (2022) · cites this paper as [305]
The Introduction (page unavailable) describes the ROPE as a range of effect values too small to be practically relevant, and Materials and Methods defines the percentage as the portion of an interval or posterior distribution within that range. But the Discussion calls that percentage a continuous index of significance, not an index of effect magnitude. (LLM, conf=0.86)
Experimental — accuracy not yet measuredFor pd interpretation, suggested reference values are: ≤95% uncertain, >95% possibly existing, >97% likely existing, >99% probably existing, >99.9% certainly existing.
“For pd interpretation, the following reference values have been suggested:≤95% = uncertain>95% = possibly existing>97% = likely existing>99% = probably existing>99.9% = certainly existing”
From Infraslow closed-loop brain training for anxiety and depression (ISAD): a protocol for a randomized, double-blind, sham-controlled pilot trial in adult females with internalizing disorders (2022) · cites this paper as [305]
The Reporting Guidelines (page unavailable) suggest 95%, 97%, and 99% as possible pd reference points. They do not provide the claimed verbal categories or a 99.9% threshold, and say that defining interpretation heuristics is beyond the paper’s scope. (LLM, conf=0.90)
Experimental — accuracy not yet measuredFor percentage in ROPE interpretation, suggested reference values are: >99% negligible, >97.5% probably negligible, ≤97.5 and ≥2.5% undecided, <2.5% probably significant, <1% significant.
“For percentage in ROPE interpretation, the following reference values have been suggested:>99% = negligible>97.5% = probably negligible≤97.5 and ≥2.5% = undecided<2.5% = probably significant<1% = significant”
From Infraslow closed-loop brain training for anxiety and depression (ISAD): a protocol for a randomized, double-blind, sham-controlled pilot trial in adult females with internalizing disorders (2022) · cites this paper as [305]
The Discussion (no page supplied) supports interpreting the percentage in ROPE as an index of significance and allows cutoffs if categorization is needed. It gives 5% and 95% as examples, however, and does not suggest the claimed five-category labels or their 99%, 97.5%, 2.5%, and 1% thresholds. The Reporting Guidelines say that defining interpretation heuristics is beyond the paper’s scope. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe probability of direction (pd) is an index of the existence of an effect and is represented by the certainty (50-100%) in the direction, positive or negative.
“The pd is an index of the existence of an effect and is represented by the certainty (50-100%) in the direction, positive or negative [302, 305].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
<i>Stan</i> : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
The ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1× · see citing works
“The p-value represents the probability that for a given statistical model, when the null hypothesis is true, the effect would be greater than or equal to the observed coefficient (Wasserstein and Lazar, 2016).”
BackgroundSontrop, J. M., Dixon, S. N., Garg, A. X., Jiménez, I. B., Dohein, O., Huang, S. S., & Clark, W. F. (2013) · American Journal of Nephrology · doi:10.1159/000350377 · Cited by 5,903 · see citing works
Using Effect Size—or Why the P Value Is Not Enough
Sullivan, G. M., & Feinn, R. (2012) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-12-00156.1 · Cited by 5,553 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983 · see citing works
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772 · see citing works
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Statistical Methods for Research Workers.
I., L., & Fisher, R. A. (1926) · Journal Of The Royal Statistical Society · doi:10.2307/2341488 · Cited by 2,177 · see citing works
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161 · cited 2× · see citing works
“…the null hypothesis from the classic point-null hypothesis to a range of values considered negligible or too small to be of any practical relevance (the Region of Practical Equivalence – ROPE; Kruschke, 2014; Lakens, 2017; Lakens et al., 2018), usually spread equally around 0 (e.g., [−0.1; 0.1]).”
BackgroundInfluentialConfidence intervals rather than P values: estimation rather than hypothesis testing.
Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,014 · see citing works
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,934 · see citing works
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · see citing works
(2014) · Thammasat University Digital Collections · Cited by 1,732 · see citing works
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,651 · see citing works
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · see citing works
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536 · see citing works
Makowski, D., Ben-Shachar, M., & Lüdecke, D. (2019) · Journal of Open Source Software · doi:10.21105/joss.01541 · Cited by 1,483 · see citing works
Self-citationHewson, P. (2014) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2014.12096_1.x · Cited by 1,368 · see citing works
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288 · see citing works
What are the odds? A practical guide to computing and reporting Bayes factors
Jarosz, A. F., & Wiley, J. (2014) · The Journal of Problem Solving · doi:10.7771/1932-6246.1167 · Cited by 1,101 · see citing works
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761 · see citing works
Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.12.001 · Cited by 740 · see citing works
Statistical Rethinking: A Bayesian Course with Examples in R and Stan
McElreath, R. (2018) · doi:10.1201/9781315372495 · Cited by 726 · see citing works
<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · cited 1× · see citing works
“It is interesting to note that this perspective unites significance testing with the focus on effect size (involving a discrete separation between at least two categories: negligible and non-negligible), which finds an echo in recent statistical recommendations (Ellis and Steyn, 2003; Sullivan and Feinn, 2012; Simonso…”
BackgroundBayes factor approaches for testing interval null hypotheses.
Morey, R. D., & Rouder, J. N. (2011) · Psychological Methods · doi:10.1037/a0024377 · Cited by 584 · see citing works
Kruschke, J., Aguinis, H., & Joo, H. (2012) · Organizational Research Methods · doi:10.1177/1094428112457829 · Cited by 526 · see citing works
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 498 · see citing works
Default Bayes Factors for Model Selection in Regression
Rouder, J. N., & Morey, R. D. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.734737 · Cited by 477 · see citing works
Ly, A., Verhagen, J., & Wagenmakers, E. (2015) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.06.004 · Cited by 383 · see citing works
Four reasons to prefer Bayesian analyses over significance testing
Dienes, Z. P., & McLatchie, N. M. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1266-z · Cited by 355 · see citing works
What to believe: Bayesian methods for data analysis
Kruschke, J. K. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.001 · Cited by 342 · see citing works
Bayesian Benefits for the Pragmatic Researcher
Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016) · Current Directions in Psychological Science · doi:10.1177/0963721416643289 · Cited by 334 · see citing works
Halsey, L. (2019) · Biology Letters · doi:10.1098/rsbl.2019.0174 · Cited by 327 · see citing works
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · cited 2× · see citing works
“The Bayesian framework is quickly gaining popularity among psychologists and neuroscientists (Andrews and Baguley, 2013), for reasons such as flexibility, better accuracy in noisy data and small samples, less proneness to type I errors, the possibility of introducing prior knowledge into the analysis and the intuitive…”
BackgroundMethodEllis, S., & Steyn, H. S. (2003) · Bestuursdinamika · Cited by 254 · see citing works
Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think
Fidler, F. M., Thomason, N. R., Cumming, G., Finch, S. J., & Leeman, J. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502008.x · Cited by 244 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Prior sensitivity in theory testing: An apologia for the Bayes factor
Vanpaemel, W. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.07.003 · Cited by 165 · see citing works
insight: A Unified Interface to Access Information from Model Objects in R
Lüdecke, D., Waggoner, P., & Makowski, D. (2019) · The Journal of Open Source Software · doi:10.21105/joss.01412 · Cited by 154 · see citing works
Self-citationGelman, A. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167217729162 · Cited by 140 · see citing works
The Need for Bayesian Hypothesis Testing in Psychological Science
Wagenmakers, E., Verhagen, J., Ly, A., Matzke, D., Steingroever, H., Rouder, J. N., & Morey, R. D. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch8 · Cited by 122 · see citing works
Szűcs, D., & Ioannidis, J. P. A. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/071530 · Cited by 112 · see citing works
Bayesian inference for psychology, part IV: parameter estimation and Bayes factors
Rouder, J. N., Haaf, J. M., & Vandekerckhove, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1420-7 · Cited by 102 · see citing works
Prior approval: The growth of Bayesian methods in psychology
Andrews, M., & Baguley, T. S. (2013) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12004 · Cited by 85 · see citing works
Who Should Be Afraid of the Jeffreys-Lindley Paradox?
Spanos, A. (2013) · Philosophy of Science · doi:10.1086/668875 · Cited by 73 · see citing works
On the Jeffreys-Lindley Paradox
Robert, C. P. (2014) · Philosophy of Science · doi:10.1086/675729 · Cited by 65 · see citing works
Heck, D. W. (2018) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12150 · Cited by 54 · see citing works
The expected demise of the Bayes factor
Robert, C. P. (2015) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.08.002 · Cited by 47 · see citing works
Reform of statistical inference in psychology: The case ofMemory & Cognition
Finch, S. J., Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., Anderson, J. L., & Goodman, O. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195577 · Cited by 39 · see citing works
Bayesian inference and testing any hypothesis you can specifyPreprint
Etz, A. J., Haaf, J. M., Rouder, J. N., & Vandekerckhove, J. (2018) · doi:10.31234/osf.io/wmf3r · Cited by 16 · see citing works
Alternative Bayes factors: Sample size determination and discriminatory power assessment
Santis, F. D. (2007) · Test · doi:10.1007/s11749-006-0017-7 · Cited by 13 · see citing works
Mills, J. A., & Parent, O. (2013) · Springer eBooks · doi:10.1007/978-3-642-23430-9_89 · Cited by 9 · see citing works
The use of p-values in applied research: Interpretation and new trends
Marasini, D., Quatto, P., & Ripamonti, E. (2016) · Statistica · doi:10.6092/issn.1973-2201/6439 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Green, B., & Chen, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479562 · Cited by 69 · see citing works
Catelan, D., Biggeri, A., Russo, F., Gregori, D., Pitter, G., Da Re, F., Fletcher, T., & Canova, C. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18052734 · Cited by 47 · see citing works
Zeifman, R. J., Kettner, H., Ross, S., Weiss, B., Mithoefer, M. C., Mithoefer, A. T., & Wagner, A. C. (2024) · European Journal of Psychotraumatology · doi:10.1080/20008066.2023.2297536 · Cited by 37 · see citing works
Schwartzman, J. M., Roth, M. C., Paterson, A. V., Jacobs, A. X., & Williams, Z. J. (2024) · Autism · doi:10.1177/13623613231213543 · Cited by 24 · see citing works
“…the probability of direction ( P d , i.e., the posterior probability that the true parameter value has the same sign as its point estimate; Makowski et al., 2019) was computed for all parameters, with values of P d > 0.975 indicating “statistical significance” at a level comparable to…”
MethodInfluentialWilliams, Z. J., Suzman, E., & Woynaroski, T. G. (2021) · American Journal of Audiology · doi:10.1044/2021_aja-20-00209 · Cited by 16 · see citing works
Perez, T. M., Glue, P., Adhia, D. B., Navid, M. S., Zeng, J., Dillingham, P., Smith, M., Niazi, I. K., Young, C. K., & De Ridder, D. (2022) · Trials · doi:10.1186/s13063-022-06863-z · Cited by 8 · see citing works
“• The percentage in ROPE indexes the magnitude of an effect where the ROPE is the range of effect size values considered to be practically equivalent to the null [302, 305].”
BackgroundMild traumatic brain injury alters function in the dorsolateral prefrontal cortex: a TMS-EEG study
Moore, E., Webb, E., Sasaki, R., Rogasch, N. C., Foo, N., Semmler, J. G., & Opie, G. M. (2025) · doi:10.1101/2025.05.22.655676 · Cited by 3 · see citing works
“Custom contrasts of main effects and interactions were generated using the emmeans package (Lenth 2025), with effect existence and significance idenitified using the probability of direction (pd; Makowski et al. 2019) and region of practical equivalence, respectively.”
Delayed goal-directed processing underlies inhibitory control challenges in adult ADHD
Osborne, J. B., Sellers, J., Zhang, H., Wang, S., Liberatore, S., Sripada, C., Shah, P., & Jonides, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-42307-3 · Cited by 2 · see citing works
Butler, R. M., Christian, C., Girard, J. M., Vanzhula, I. A., & Levinson, C. A. (2024) · Behaviour Research and Therapy · doi:10.1016/j.brat.2024.104577 · Cited by 1 · see citing works
- Authors
- Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D.
- Affiliations
- Nanyang Technological University (SG)
- University Medical Center Hamburg-Eppendorf (DE)
- Corresponding author
- Makowski, Dominique
- Chen, S. H. Annabel
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2019-12-10
- Volume
- 10
- Pages
- 2767-2767
- ISSN
- 1664-1078 (ISSN-L)
- PMC
- PMC6914840
- OpenAlex
- W2996229661
- Cited by
- 983
- References
- 60
- Citations by year
- 2026: 188 · 2025: 187 · 2024: 182 · 2023: 146 · 2022: 136 · 2021: 99 · 2020: 41 · 2019: 2 · 2018: 1 · 2017: 1
- FWCI
- 54.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Psychology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- Bayesian analysis, statistical significance, replication crisis, p-values, effect size, sample size, prior knowledge, scientific reporting standards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
