The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Abstract
Statistical inference in psychology has traditionally relied heavily on p-value significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes. The second is to replace inference with Bayesian measures of evidence, such as the Bayes factor. The authors provide a practical comparison of p values, effect sizes, and default Bayes factors as measures of statistical evidence, using 855 recently published t tests in psychology. The comparison yields two main results. First, although p values and default Bayes factors almost always agree about what hypothesis is better supported by the data, the measures often disagree about the strength of this support; for 70% of the data sets for which the p value falls between .01 and .05, the default Bayes factor indicates that the evidence is only anecdotal. Second, effect sizes can provide additional evidence to p values and default Bayes factors. The authors conclude that the Bayesian approach is comparatively prudent, preventing researchers from overestimating the evidence in favor of an effect.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Individually, studies that find statistically significant effects may not actually provide much evidence for or against the null (see Wetzels et al., 2011).”
From Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null (2016) · cites this paper as Wetzels et al., 2011
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,368
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Both methods help prevent researchers from overestimating the strength of their findings and help keep the field from incorporating ambiguous findings as if these were real and reliable (Ioannidis, 2005).”
BackgroundProbability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,654
Research works printed but not linked: 9
- 10.1037/1082-989x.1.4.379 · doi:10.1037/1082-989x.1.4.379 · Paper page
- Good I.J., 1985, Bayesian statistics, 2, 249
- 10.1037/0003-066x.52.1.15 · doi:10.1037/0003-066x.52.1.15 · Paper page
- Jeffreys H. (1961). Theory of probability. Oxford, UK: Oxford University Press.
- Lee M.D., Journal of Mathematical Psychology
- Myung I.J., 2000, Journal of Mathematical Psychology, 44
- 10.1037/0033-295x.109.3.472 · doi:10.1037/0033-295x.109.3.472 · Paper page
- 10.1037/0003-066x.45.6.775 · DOI · Paper page
- 10.1037/0022-0663.74.2.166 · doi:10.1037/0022-0663.74.2.166 · Paper page
Cited by
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
Default Bayes factors for ANOVA designs
Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.08.001 · Cited by 1,800
Self-citationBayesian inference for psychology. Part II: Example applications with JASP
Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q. F., Dropmann, D., Boutin, B., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1323-7 · Cited by 1,715
Self-citationBayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Self-citationBayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Metadata
- Authors
- Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J.
- Affiliations
- University of Amsterdam (NL)
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- University of California, Irvine (US)
- University of Missouri (US)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2011-05
- Volume
- 6
- Issue
- 3
- Pages
- 291-298
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2170660620
- Semantic Scholar
- 18744e6c84087ccc20bc0f6db28020bc48c81b4a
- Cited by
- 945
- References
- 76
- Citations by year
- 2026: 40 · 2025: 63 · 2024: 71 · 2023: 82 · 2022: 82 · 2021: 122 · 2020: 75 · 2019: 82 · 2018: 65 · 2017: 79 · 2016: 46 · 2015: 41 · 2014: 33 · 2013: 23 · 2012: 27
- FWCI
- 4.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- statistical evidence, p-values, effect size, t-test, Bayesian inference, significance testing, quality of evidence, anecdotal evidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests. https://app.cochat.ai/papers/10.1177/1745691611406923
- The paper
- Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests. Perspectives on Psychological Science. https://doi.org/10.1177/1745691611406923
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.