Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“for example, following standard methods for their fields, researchers have observed a dead Atlantic salmon exhibiting neural responses to emotional stimuli [55] and university students apparently demonstrating ‘pre-cognitive’ abilities to predict the outcome of a random number generator [56].”
From The natural selection of bad science (2016) · cites this paper as [56]
Citations
Cites
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,269
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,575
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
Diciccio, T. J., Kass, R. E., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,943
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
JEFFREYS, H. (1922) · Nature · doi:10.1038/109132a0 · Cited by 1,217
Mixtures of g Priors for Bayesian Variable Selection
Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001337 · Cited by 1,089
The Intrinsic Bayes Factor for Model Selection and Prediction
Berger, J. O., & Pericchi, L. R. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476668 · Cited by 1,007
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Self-citationBem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893
Statistical Evidence: A Likelihood Paradigm
Reeves, J. H., & Royall, R. M. (1998) · Journal of the American Statistical Association · doi:10.2307/2669869 · Cited by 814
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 775
Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.12.001 · Cited by 740
Self-citationFractional Bayes Factors for Model Comparison
O'Hagan, A. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02017.x · Cited by 671
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664
Bayesian Methods: A Social and Behavioral Sciences Approach
Gill, J. (2002) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420057478 · Cited by 579
Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem
Scott, J. G., & Berger, J. O. (2010) · The Annals of Statistics · doi:10.1214/10-aos792 · Cited by 573
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548
Escaping affect: how motivated emotion regulation creates insensitivity to mass suffering.
Cameron, C. D., & Payne, B. K. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021643 · Cited by 477
Bayesian statistical methods for genetic association studies
Stephens, M., & Balding, D. J. (2009) · Nature Reviews Genetics · doi:10.1038/nrg2615 · Cited by 468
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
Subjective Bayesian Analysis: Principles and Practice
Goldstein, M. (2006) · Bayesian Analysis · doi:10.1214/06-ba116 · Cited by 214
Statistical Problems in ESP Research
Diaconis, P. (1978) · Science · doi:10.1126/science.663642 · Cited by 198
Writing the Empirical Journal Article
Bem, D. J. (2021) · Psychology Press eBooks · doi:10.4324/9781315808314-10 · Cited by 162
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143
Self-citationStorm, L., Tressoldi, P. E., & Di Risio, L. (2010) · Psychological Bulletin · doi:10.1037/a0019457 · Cited by 140
Replication and Meta-Analysis in Parapsychology
Utts, J. M. (1991) · Statistical Science · doi:10.1214/ss/1177011577 · Cited by 136
Price, G. (1955) · Science · doi:10.1126/science.122.3165.359 · Cited by 113
Bernardo, J. M., DeGroot, M. H., Lindley, D. V., & Smith, A. F. M. (1987) · Biometrics · doi:10.2307/2531979 · Cited by 110
Critical Thinking in Psychology
Sternberg, R. J. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804632.018 · Cited by 106
Gönen, M., Johnson, W. O., Lu, Y., & Westfall, P. H. (2005) · The American Statistician · doi:10.1198/000313005x55233 · Cited by 104
Experimenter Effects and the Remote Detection of Staring
Wiseman, R. J., & Schlitz, M. J. (2017) · Routledge eBooks · doi:10.4324/9781315247366-22 · Cited by 78
A Bayesian Perspective on Hypothesis Testing: A Comment on Killeen (2005)
Wagenmakers, E., & Grünwald, P. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01757.x · Cited by 65
Self-citationBem, D. J. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511807862.002 · Cited by 47
Bayesian Analysis of Random Event Generator Data
Jefferys, W. H. (1990) · Cited by 42
Why Is PSI So Elusive? A Review and Proposed Model
Kennedy, J. E. (2001) · Cited by 32
Guide to Publishing in Psychology Journals
(2018) · Cambridge University Press eBooks · doi:10.1017/9781108304443 · Cited by 31
The Compleat Academic: A Career Guide
Darley, J. M., Zanna, M. P., & Roediger, H. L., III (2009) · Medical Entomology and Zoology · Cited by 27
Evaluating Parapsychological Claims
Hyman, R. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804632.014 · Cited by 7
Cited by
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Self-citationIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Self-citation(2013) · Cited by 1,351
Metadata
- Authors
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Journal of Personality and Social Psychology
- Published
- 2011-03
- Volume
- 100
- Issue
- 3
- Pages
- 426-432
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2169244873
- Cited by
- 977
- References
- 41
- Citations by year
- 2026: 35 · 2025: 83 · 2024: 65 · 2023: 72 · 2022: 75 · 2021: 91 · 2020: 85 · 2019: 71 · 2018: 89 · 2017: 69 · 2016: 54 · 2015: 46 · 2014: 46 · 2013: 44 · 2012: 36
- FWCI
- 62.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Personality and Social Psychology (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Dutch Organization for Scientific Research
- Topics
- Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- extrasensory perception, precognition, exploratory data analysis, statistical evidence, null hypothesis, confirmatory research, data analysis methods
- MeSH terms
- Bayes Theorem, Cognition, Data Interpretation, Statistical, Humans, Parapsychology, Psychology, Guidelines as Topic, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.