Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“for example, following standard methods for their fields, researchers have observed a dead Atlantic salmon exhibiting neural responses to emotional stimuli [55] and university students apparently demonstrating ‘pre-cognitive’ abilities to predict the outcome of a random number generator [56].”
From The natural selection of bad science (2016) · cites this paper as [56]
Citations
Cites
An introduction to probability theory and its applications
(1958) · Journal of the Franklin Institute · doi:10.1016/0016-0032(58)90688-4 · Cited by 29,269
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,654
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
Diciccio, T. J., Kass, R. E., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,943
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
JEFFREYS, H. (1922) · Nature · doi:10.1038/109132a0 · Cited by 1,217
Mixtures of g Priors for Bayesian Variable Selection
Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001337 · Cited by 1,089
The Intrinsic Bayes Factor for Model Selection and Prediction
Berger, J. O., & Pericchi, L. R. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476668 · Cited by 1,007
Cited by
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Self-citationIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Self-citation(2013) · Cited by 1,351
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Judd, C. M., Westfall, J., & Kenny, D. A. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028347 · Cited by 1,236
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Self-citationStatistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Self-citationThe natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
A Gentle Introduction to Bayesian Analysis: Applications to Developmental Research
van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & van Aken, M. A. G. (2014) · Child Development · doi:10.1111/cdev.12169 · Cited by 779
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Unconscious influences on decision making: A critical review
Newell, B. R., & Shanks, D. R. (2014) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12003214 · Cited by 652
The fallacy of placing confidence in confidence intervals
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0947-8 · Cited by 577
Self-citationFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Kruschke, J. K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406925 · Cited by 468
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Simmons, J., Nelson, L., & Simonsohn, U. (2011) · PsycEXTRA Dataset · doi:10.1037/e519702015-014 · Cited by 454
On the Interpretation and Use of Mediation: Multiple Perspectives on Mediation Analysis
Agler, R., & De Boeck, P. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01984 · Cited by 400
Theory Construction Methodology: A Practical Framework for Building Theories in Psychology
Borsboom, D., van der Maas, H. L. J., Dalege, J., Kievit, R. A., & Haig, B. D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620969647 · Cited by 382
Self-citationStatistical Inference as Severe Testing: How to Get Beyond the Statistics Wars
Mayo, D. G. (2018) · doi:10.1017/9781107286184 · Cited by 376
A systematic review of Bayesian articles in psychology: The last 25 years.
van de Schoot, R., Winter, S. D., Ryan, O., Zondervan-Zwijnenburg, M., & Depaoli, S. (2017) · Psychological Methods · doi:10.1037/met0000100 · Cited by 374
Bayesian analysis of factorial designs.
Rouder, J. N., Morey, R. D., Verhagen, J., Swagman, A. R., & Wagenmakers, E.-J. (2017) · Psychological Methods · doi:10.1037/met0000057 · Cited by 343
Self-citationAcceptance of immersive head-mounted virtual reality in older adults
Huygelier, H., Schraepen, B., van Ee, R., Vanden Abeele, V., & Gillebert, C. R. (2019) · Scientific Reports · doi:10.1038/s41598-019-41200-6 · Cited by 299
Bayesian tests to quantify the result of a replication attempt.
Verhagen, J., & Wagenmakers, E.-J. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0036731 · Cited by 291
Self-citationWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Giner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
Model Comparison and the Principle of Parsimony
Vandekerckhove, J., Matzke, D., & Wagenmakers, E.-J. (2015) · Oxford Handbooks Online · doi:10.1093/oxfordhb/9780199957996.013.14 · Cited by 269
Self-citationA peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
A Large Scale Test of the Effect of Social Class on Prosocial Behavior
Korndörfer, M., Egloff, B., & Schmukle, S. C. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133193 · Cited by 255
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205
A Bayes factor meta-analysis of Bem’s ESP claim
Rouder, J. N., & Morey, R. D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0088-7 · Cited by 194
The Effects of Generative AI on Design Fixation and Divergent Thinking
Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q., & Velloso, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642919 · Cited by 320
Frank, M. C., & Saxe, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460686 · Cited by 175
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses
Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.12.007 · Cited by 126
Peterson, D. (2016) · doi:10.1177/2378023115625071 · Cited by 111
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105
Giofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0175583 · Cited by 98
Golay, P., Reverte, I., Rossier, J., Favez, N., & Lecerf, T. (2013) · Psychological Assessment · doi:10.1037/a0030676 · Cited by 94
The Choreography of Group Affiliation
von Zimmermann, J., Vicary, S., Sperling, M., Orgs, G., & Richardson, D. C. (2018) · Topics in Cognitive Science · doi:10.1111/tops.12320 · Cited by 73
Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science
Veldkamp, C. L. S., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114876 · Cited by 69
Objective Facebook behaviour: Differences between problematic and non-problematic users
Marino, C., Finos, L., Vieno, A., Lenzi, M., & Spada, M. M. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.04.015 · Cited by 73
Leichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
Age-related increases in reaction time result from slower preparation, not delayed initiation
Hardwick, R. M., Forrence, A. D., Costello, M. G., Zackowski, K., & Haith, A. M. (2022) · Journal of Neurophysiology · doi:10.1152/jn.00072.2022 · Cited by 66
Ruiz, F. J., Flórez, C. L., García-Martín, M. B., Monroy-Cifuentes, A., Barreto-Montero, K., García-Beltrán, D. M., Riaño-Hernández, D., Sierra, M. A., Suárez-Falcón, J. C., Cardona-Betancourt, V., et al. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.04.004 · Cited by 65
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
The Effect of Spatial Smoothing on Representational Similarity in a Simple Motor Paradigm
Hendriks, M. H. A., Daniels, N., Pegado, F., & Op de Beeck, H. P. (2017) · Frontiers in Neurology · doi:10.3389/fneur.2017.00222 · Cited by 55
Rouder, J. N., Morey, R. D., & Province, J. M. (2013) · Psychological Bulletin · doi:10.1037/a0029008 · Cited by 48
Measuring narrative engagement: The heart tells the story
Richardson, D. C., Griffin, N. K., Zaki, L., Stephenson, A., Yan, J., Curry, T., Noble, R., Hogan, J., Skipper, J. I., & Devlin, J. T. (2018) · doi:10.1101/351148 · Cited by 39
Golay, P., Morandi, S., Silva, B., Devas, C., & Bonsack, C. (2019) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2019.101512 · Cited by 34
Storm, L., Sherwood, S. J., Roe, C. A., Tressoldi, P. E., Rock, A. J., & Di Risio, L. (2017) · International Journal of Dream Research · doi:10.11588/ijodr.2017.2.34888 · Cited by 29
Ortega, A., & Navarrete, G. (2017) · Bayesian Inference · doi:10.5772/intechopen.70230 · Cited by 31
Skumlien, M., Mokrysz, C., Freeman, T. P., Wall, M. B., Bloomfield, M., Lees, R., Borissova, A., Petrilli, K., Carson, J., Coughlan, T., et al. (2022) · Neuropsychopharmacology · doi:10.1038/s41386-022-01316-2 · Cited by 27
Skumlien, M., Mokrysz, C., Freeman, T. P., Valton, V., Wall, M. B., Bloomfield, M., Lees, R., Borissova, A., Petrilli, K., Giugliano, M., et al. (2023) · International Journal of Neuropsychopharmacology · doi:10.1093/ijnp/pyac056 · Cited by 20
Storm, L., Tressoldi, P. E., & Utts, J. (2013) · Psychological Bulletin · doi:10.1037/a0029506 · Cited by 20
Salazar, D. M., Ruiz, F. J., Ramírez, E. S., & Cardona-Betancourt, V. (2020) · The Psychological Record · doi:10.1007/s40732-019-00362-5 · Cited by 19
Open Minded: Searching for Truth about the Unconscious Mind
Newell, B. R., & Shanks, D. R. (2023) · doi:10.7551/mitpress/14922.001.0001 · Cited by 17
Williams, Z. J., Suzman, E., & Woynaroski, T. G. (2021) · American Journal of Audiology · doi:10.1044/2021_aja-20-00209 · Cited by 16
Kuiling, J. M. E., Klaassen, F., & Hagenaars, M. A. (2019) · Memory · doi:10.1080/09658211.2018.1564331 · Cited by 14
Chen, L., Benjamin, R., Lai, A., & Heine, S. (2022) · doi:10.31234/osf.io/kuhy6 · Cited by 14
Follow‐up on the U.S. Central Intelligence Agency's (CIA) remote viewing experiments☆
Escolà‐Gascón, Á., Houran, J., Dagnall, N., Drinkwater, K., & Denovan, A. (2023) · Brain and Behavior · doi:10.1002/brb3.3026 · Cited by 6
Chen, L., Benjamin, R., Guo, Y., Lai, A., & Heine, S. J. (2023) · Research Square · doi:10.21203/rs.3.rs-1254756/v1 · Cited by 5
Preregistration of Studies with Existing Data
Mertens, G., & Krypotos, A.-M. (2022) · Integrity of Scientific Research · doi:10.1007/978-3-030-99680-2_36
Metadata
- Authors
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Journal of Personality and Social Psychology
- Published
- 2011-03
- Volume
- 100
- Issue
- 3
- Pages
- 426-432
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2169244873
- Cited by
- 977
- References
- 41
- Citations by year
- 2026: 35 · 2025: 83 · 2024: 65 · 2023: 72 · 2022: 75 · 2021: 91 · 2020: 85 · 2019: 71 · 2018: 89 · 2017: 69 · 2016: 54 · 2015: 46 · 2014: 46 · 2013: 44 · 2012: 36
- FWCI
- 62.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Personality and Social Psychology (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Dutch Organization for Scientific Research
- Topics
- Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- extrasensory perception, precognition, exploratory data analysis, statistical evidence, null hypothesis, confirmatory research, data analysis methods
- MeSH terms
- Bayes Theorem, Cognition, Data Interpretation, Statistical, Humans, Parapsychology, Psychology, Guidelines as Topic, Behavioral Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.