The DOI matches its Crossref record; 59 of the 60 references it cites were found in a registry or an index.
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Open accessAbstract
Researchers are often confused about what can be inferred from significance tests. One problem occurs when people apply Bayesian intuitions to significance testing—two approaches that must be firmly separated. This article presents some common situations in which the approaches come to different conclusions; you can see where your intuitions initially lie. The situations include multiple testing, deciding when to stop running participants, and when a theory was thought of relative to finding out results. The interpretation of nonsignificant results has also been persistently problematic in a way that Bayesian inference can clarify. The Bayesian and orthodox approaches are placed in the context of different notions of rationality, and I accuse myself and others as having been irrational in the way we have been using statistics on a key notion of rationality. The reader is shown how to apply Bayesian inference in practice, using free online software, to allow more coherent inferences from data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [56]
Citations
Cites
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,600
Probability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 3×
“For a p value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect (Rouder et al., 2009).”
BackgroundMethodVannieuwenhoven, N. (2020) · McGill-Queen's University Press eBooks · doi:10.2307/j.ctvw1d7dg.9 · Cited by 3,370
The Methodology of Scientific Research Programmes: Philosophical Papers
Lakatos, I. (1978) · doi:10.1017/cbo9780511621123 · Cited by 3,045
Alexander, P. F. (1963) · Philosophical Books · doi:10.1111/j.1468-0149.1963.tb00798.x · Cited by 2,859
Bargh, J. A., Chen, M., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.71.2.230 · Cited by 2,736
Bargh, J. A., Chen, M. I.-C., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.71.2.230 · Cited by 2,646
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“For this reason, sooner or later, you are guaranteed to get a significant result if you run subjects long enough and stop when you get the p value you want (e.g., Wagenmakers, 2007).”
BackgroundHARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Methodology of Scientific Research Programmes
(1998) · Edward Elgar Publishing eBooks · doi:10.4337/9781781954249.00075 · Cited by 1,817
Statistical Methods in Medical Research
Armitage, P., Berry, G., & Matthews, J. N. S. (2002) · Wiley eBooks · doi:10.1002/9780470773666 · Cited by 1,638
Probability, Frequency and Reasonable Expectation
Cox, R. T. (1946) · American Journal of Physics · doi:10.1119/1.1990764 · Cited by 1,330
Error and the Growth of Experimental Knowledge
Mayo, D. G. (1996) · doi:10.7208/chicago/9780226511993.001.0001 · Cited by 1,325
Scientific reasoning: the Bayesian approach
(1990) · Choice Reviews Online · doi:10.5860/choice.27-2696 · Cited by 1,247
Sivia, D. S. (2006) · doi:10.1093/oso/9780198568315.001.0001 · Cited by 1,003
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962 · cited 5×
“For a couple of other ways of using Bayes factors, see Rouder et al. (2009) and Wetzels et al. (2011) for a suggested ‘‘default’’ Bayes factor to be used on any data where the null hypothesis is compared with a default theory (namely, the theory that effects may occur in either direction, scaled to a large standardize…”
BackgroundMethodInfluentialStatistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Gigerenzer, G. (2004) · The Journal of Socio-Economics · doi:10.1016/j.socec.2004.09.033 · Cited by 890 · cited 1×
“…camps was momentarily won in terms of what users of statistics did: Users followed the approach systematized by Jerzy Neyman and Egon Pearson (at least this approach defined norms; in practice, researchers often followed the somewhat different advice of Ronald Fisher; see e.g., Gigerenzer, 2004).”
MethodStatistical Evidence: A Likelihood Paradigm
Reeves, J. H., & Royall, R. M. (1998) · Journal of the American Statistical Association · doi:10.2307/2669869 · Cited by 816
Repeated Significance Tests on Accumulating Data
Armitage, P., McPherson, C. K., & Rowe, B. C. (1969) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343787 · Cited by 772
The cult of statistical significance: how the standard error costs us jobs, justice, and lives
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0372 · Cited by 716
What If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 675
Probability, Statistics and Truth.
N., E., & Mises, R. V. (1939) · The Journal of Philosophy · doi:10.2307/2018923 · Cited by 639
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619 · cited 1×
“(For arguments for the frequent relevance of raw rather than standardized effect sizes, see Baguley, 2009; Ziliak & McCloskey, 2008.)”
BackgroundStatistical Evidence: A Likelihood Paradigm
Vieland, V. J., & Hodge, S. E. (1998) · The American Journal of Human Genetics · doi:10.1086/301913 · Cited by 616 · cited 10×
“For this, credibility or likelihood intervals can be used (see Dienes, 2008; Kruschke, 2010b, 2011; Royall, 1997).”
BackgroundMethodInfluentialDoing Bayesian Data Analysis: A Tutorial with R and BUGS
Barry, J. (2011) · Europe’s Journal of Psychology · doi:10.5964/ejop.v7i4.163 · Cited by 563 · cited 8×
“(Note that I will not discuss confidence intervals and credibility intervals, the Bayesian equivalent to a confidence interval, in this article; see Dienes, 2008, and Kruschke 2010b, 2011, for detailed discussion and calculations.)”
BackgroundMethodInfluentialProbability, Statistics and Truth.
Good, I. J., Mises, R. V., & Geiringer, H. P. (1958) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343367 · Cited by 519
Statistical Methods in Medical Research, 4th edn
Purdie, D. (2003) · Internal Medicine Journal · doi:10.1046/j.1445-5994.2002.00319.x · Cited by 493
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 488 · cited 10×
“(Note that I will not discuss confidence intervals and credibility intervals, the Bayesian equivalent to a confidence interval, in this article; see Dienes, 2008, and Kruschke 2010b, 2011, for detailed discussion and calculations.)”
BackgroundMethodInfluentialUnderstanding Psychology as a Science: An Introduction to Scientific and Statistical Inference
Dienes, Z. P. (2008) · Cited by 418
Self-citationStatistical Inference: A Commentary for the Social and Behavioural Sciences
Oakes, M. W. (1986) · Medical Entomology and Zoology · Cited by 388
Could Fisher, Jeffreys and Neyman Have Agreed on Testing?
Berger, J. O. (2003) · Statistical Science · doi:10.1214/ss/1056397485 · Cited by 348
Kruschke, J. K. (2010) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.72 · Cited by 343
What to believe: Bayesian methods for data analysis
Kruschke, J. K. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.05.001 · Cited by 342 · cited 9×
“(Note that I will not discuss confidence intervals and credibility intervals, the Bayesian equivalent to a confidence interval, in this article; see Dienes, 2008, and Kruschke 2010b, 2011, for detailed discussion and calculations.)”
BackgroundMethodInfluentialOn the Foundations of Statistical Inference
Birnbaum, A. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_32 · Cited by 326
A Bayesian perspective on the bonferroni adjustment
Westfall, P. (1997) · Biometrika · doi:10.1093/biomet/84.2.419 · Cited by 316
Journal of the American Statistical Association
Jeroen, V., Version, P. R., Publication, L. T., Jeroen, D., & Vermunt, K. (2007) · Encyclopedia of Measurement and Statistics · doi:10.4135/9781412952644.n234 · Cited by 303
Scientific Reasoning: The Bayesian Approach
Shafer, G. R. (2007) · Journal of the American Statistical Association · doi:10.1198/jasa.2007.s176 · Cited by 269
Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think
Fidler, F. M., Thomason, N. R., Cumming, G., Finch, S. J., & Leeman, J. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502008.x · Cited by 244
Lakatos, I., Motterlini, M., & Feyerabend, P. (1999) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226467030.001.0001 · Cited by 189
On the Foundations of Statistical Inference: Discussion
Savage, L. J., Barnard, G., Cornfield, J., Bross, I., Box, G. E. P., Good, I. J., Lindley, D. V., Clunies-Ross, C. W., Pratt, J. W., Levene, H., et al. (1962) · Journal of the American Statistical Association · doi:10.2307/2281641 · Cited by 185
Bayesian Evaluation of Informative Hypotheses
Klugkist, I. G., van Wesel, F., van Well, S., Kolk, A. M. M., Hoijtink, H. J. A., Klugkist, I. G., & Boelen, P. A. (2008) · Springer eBooks · doi:10.1007/978-0-387-09612-4 · Cited by 168
Critical Rationalism: A Restatement and Defence
Miller, D. W. (1994) · Medical Entomology and Zoology · Cited by 163
Howard, G., Maxwell, S., & Fleming, K. J. (2000) · Psychological methods · doi:10.1037/1082-989x.5.3.315 · Cited by 141
Scandinavian Journal of Psychology*
Davis, D. R. (1961) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.1961.tb10343.x · Cited by 137
The Nature of Scientific Evidence
Taper, M. L., & Lele, S. R. (2004) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226789583.001.0001 · Cited by 133
Bayesian Evaluation of Informative Hypotheses
Hoijtink, H. (2013) · PsycEXTRA Dataset · doi:10.1037/e606342013-001 · Cited by 109
Mussweiler, T. (2005) · Psychological Science · doi:10.1111/j.1467-9280.2005.01659.x · Cited by 92 · cited 1×
“Wetzels et al. (2011) consider an experiment by Mussweiler (2006), who assessed whether unobtrusively inducing people to move in a portly manner would (in comparison with a control group) make subjects view an ambiguous target person as more overweight.”
BackgroundCoulson, M. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00026 · Cited by 77
Detecting chance: A solution to the null sensitivity problem in subliminal priming
Rouder, J. N., Morey, R. D., Speckman, P. L., & Pratte, M. S. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03196808 · Cited by 67
Halpern, J. Y. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.644 · Cited by 50
Baguley, T. S. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711009x481027 · Cited by 30
Likelihood and its Evidential Framework
Blume, J. D. (2011) · Elsevier eBooks · doi:10.1016/b978-0-444-51862-0.50014-9 · Cited by 23
An Error-Statistical Philosophy of Evidence
Mayo, D. G. (2004) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226789583.003.0004 · Cited by 20 · cited 1×
“By ignoring the issue of multiple testing, you have found strong evidence in favor of a child being the reincarnation just because you tested many children’’ (see e.g., Mayo, 1996, 2004, for the need for correction for multiple testing as an argument against Bayes).”
BackgroundTen Statisticians and Their Impacts for Psychologists
Wright, D. B. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01167.x · Cited by 15 · cited 19×
“(Note that I will not discuss confidence intervals and credibility intervals, the Bayesian equivalent to a confidence interval, in this article; see Dienes, 2008, and Kruschke 2010b, 2011, for detailed discussion and calculations.)”
BackgroundInfluentialAdvances in Cognitive Science: Learning, Evolution and Social Action
Miranda, L. Á. P., & Madariaga, A. I. (2010) · Cited by 4
Measuring the conscious status of knowledge
Dienes, Z. P. (2010) · Figshare · Cited by 2
Self-citationIversen, G. R., Klockars, A. J., Sax, G., Rosenthal, R., & Rosnow, R. L. (1987) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2069981 · Cited by 1
Research works printed but not linked: 2
- Blume J. D., Handbook of the philosophy of statistics
- Jeffreys H., 1961, The theory of probability
Cited by
Sorted by citations
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
“…with orthodox statistics, unless pre-specified as possible by one’s stopping rule (Armitage et al., 1969); by contrast, with Bayes, one can always collect more participants until the data are sensitive enough, that is, B < 1/3 or B > 3; see e.g., Berger and Wolpert (1988), Dienes (2008, 2011).”
BackgroundInfluentialSelf-citationBayesian inference for psychology. Part II: Example applications with JASP
Wagenmakers, E., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q. F., Dropmann, D., boutin, B., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1323-7 · Cited by 1,788
“…(i.e., Bayesian model averaging in regression, Clyde, 2016), the informative model comparison approach (e.g., Gu, Mulder, Decovi´c, & Hoijtink, 2014; Gu, 2016; Mulder, 2014, 2016), and a more flexible and subjective prior specification approach (e.g., Dienes, 2011, 2014, 2016; Gronau et al., 2017).”
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,767
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
“Dienes (2008, 2011) provided another analytical solution, with a corresponding online calculator.”
BackgroundMethod(2013) · Cited by 1,351
Metadata
- Authors
- Dienes, Z.
- Affiliations
- University of Sussex (GB)
- Corresponding author
- Dienes, Zoltan
- ORCID
- Dienes, Zoltan
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2011-05
- Volume
- 6
- Issue
- 3
- Pages
- 274-290
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2055754299
- Cited by
- 938
- References
- 60
- Citations by year
- 2026: 25 · 2025: 35 · 2024: 39 · 2023: 46 · 2022: 64 · 2021: 99 · 2020: 87 · 2019: 84 · 2018: 98 · 2017: 82 · 2016: 76 · 2015: 68 · 2014: 48 · 2013: 42 · 2012: 33
- FWCI
- 6.30
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- Bayesian inference, significance testing, null hypothesis significance testing, multiple comparisons, stopping rules
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
