The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
Abstract
Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“for example, following standard methods for their fields, researchers have observed a dead Atlantic salmon exhibiting neural responses to emotional stimuli [55] and university students apparently demonstrating ‘pre-cognitive’ abilities to predict the outcome of a random number generator [56].”
From The natural selection of bad science (2016) · cites this paper as [56]
Citations
Cites
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,269
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,575
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
DiCiccio, T. J., Kass, R. E., Raftery, A. E., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,977
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
JEFFREYS, H. (1922) · Nature · doi:10.1038/109132a0 · Cited by 1,217
Mixtures of g Priors for Bayesian Variable Selection
Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001337 · Cited by 1,098
The Intrinsic Bayes Factor for Model Selection and Prediction
Berger, J. O., & Pericchi, L. R. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476668 · Cited by 1,007
Cited by
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Self-citation(2013) · Cited by 1,351
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Judd, C. M., Westfall, J., & Kenny, D. A. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028347 · Cited by 1,236
Metadata
- Authors
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Journal of Personality and Social Psychology
- Published
- 2011-03
- Volume
- 100
- Issue
- 3
- Pages
- 426-432
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2169244873
- Cited by
- 977
- References
- 41
- Citations by year
- 2026: 35 · 2025: 83 · 2024: 65 · 2023: 72 · 2022: 75 · 2021: 91 · 2020: 85 · 2019: 71 · 2018: 89 · 2017: 69 · 2016: 54 · 2015: 46 · 2014: 46 · 2013: 44 · 2012: 36
- FWCI
- 62.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Personality and Social Psychology (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Dutch Organization for Scientific Research
- Topics
- Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- extrasensory perception, precognition, exploratory data analysis, statistical evidence, null hypothesis, confirmatory research, data analysis methods
- MeSH terms
- Bayes Theorem, Cognition, Data Interpretation, Statistical, Humans, Parapsychology, Psychology, Guidelines as Topic, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). https://app.cochat.ai/papers/10.1037/a0022790
- The paper
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology. https://doi.org/10.1037/a0022790
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.