A Bayes factor meta-analysis of recent extrasensory perception experiments: Comment on Storm, Tressoldi, and Di Risio (2010).
Abstract
Psi phenomena, such as mental telepathy, precognition, and clairvoyance, have garnered much recent attention. We reassess the evidence for psi effects from Storm, Tressoldi, and Di Risio's (2010) meta-analysis. Our analysis differs from Storm et al.'s in that we rely on Bayes factors, a Bayesian approach for stating the evidence from data for competing theoretical positions. In contrast to more conventional analyses, inference by Bayes factors allows the analyst to state evidence for the no-psi-effect null as well as for a psi-effect alternative. We find that the evidence from Storm et al.'s presented data set favors the existence of psi by a factor of about 6 billion to 1, which is noteworthy even for a skeptical reader. Much of this effect, however, may reflect difficulties in randomization: Studies with computerized randomization have smaller psi effects than those with manual randomization. When the manually randomized studies are excluded and omitted studies included, the Bayes factor evidence is at most 330 to 1, a greatly attenuated value. We argue that this value is unpersuasive in the context of psi because there is no plausible mechanism and because there are almost certainly omitted replication failures.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors' analysis approach closely followed the methods used by Rouder and Morey (2011) and Rouder et al. (2013).
“Our analysis approach closely followed the methods used by Rouder and Morey (2011) and Rouder et al. (2013) in their meta-analyses of psi (i.e., extrasensory perception).”
From Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null (2016) · cites this paper as Rouder et al. (2013)
Cites
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Self-citationA practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
Numerical recipes in C (2nd ed.): the art of scientific computing
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1992) · Cambridge University Press eBooks · Cited by 1,156 · see citing works
Nedelman, J. R. (2005) · Computational Statistics · doi:10.1007/bf02741321 · Cited by 1,087 · see citing works
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 <i>t</i> Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
Self-citationBem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · see citing works
Statistical Analysis and the Illusion of Objectivity
Berger, J., & Berry, D. (1988) · Infectious Diseases Newsletter · doi:10.1016/0278-2316(88)90057-6 · Cited by 448 · see citing works
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392 · see citing works
Memoir on the Probability of the Causes of Events
Laplace, P. S. (1986) · Statistical Science · doi:10.1214/ss/1177013621 · Cited by 199 · see citing works
A Bayes factor meta-analysis of Bem’s ESP claim
Rouder, J. N., & Morey, R. D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0088-7 · Cited by 194 · see citing works
Self-citationBayes factors: Prior sensitivity and model generalizability
Liu, C. C., & Aitkin, M. A. (2008) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.03.002 · Cited by 163 · see citing works
Storm, L., Tressoldi, P. E., & Di Risio, L. (2010) · Psychological Bulletin · doi:10.1037/a0019457 · Cited by 140 · see citing works
Givens, G. H., Smith, D. D., & Tweedie, R. L. (1997) · Statistical Science · doi:10.1214/ss/1030037958 · Cited by 127 · see citing works
Kass, R. E. (1993) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348679 · Cited by 120 · see citing works
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105 · see citing works
Tressoldi, P. E. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00117 · Cited by 69 · see citing works
A Joint Communiqué: The Psi Ganzfeld Controversy
Hyman, R., & Honorton, C. (2019) · Journal of Parapsychology · doi:10.30891/jopar.2018s.01.09 · Cited by 64 · see citing works
Meta-analysis that conceals more than it reveals: Comment on Storm et al. (2010).
Hyman, R. (2010) · Psychological Bulletin · doi:10.1037/a0019676 · Cited by 31 · see citing works
Anomalous Cognition in Hypnagogic State with OBE Induction: An Experimental Study
Prete, G. D., & Tressoldi, P. E. (2005) · Journal of Parapsychology · Cited by 16 · see citing works
ESP under Hypnosis: The Role of Induction Instructions and Personality Characteristics
Tressoldi, P. E., & Prete, G. D. (2007) · Padua Research Archive (University of Padova) · Cited by 13 · see citing works
Paranormal effects using sighted and vision-impaired participants in a quasi-Ganzfeld task
Storm, L., & Thalbourne, M. A. (2001) · Cited by 11 · see citing works
Remote viewing by committee: RV using a multiple agent/multiple percipient design
Storm, L. (2003) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 10 · see citing works
A remote viewing pilot study using a ganzfeld induction procedure
Roe, C. A., & Flint, S. W. (2007) · Cited by 10 · see citing works
Sender and receiver creativity scores as predictors of performance at a Ganzfeld ESP task.
Roe, C. A., McKenzie, E. A., & Ali, A. N. (2001) · Cited by 10 · see citing works
Dream Gesp and Consensus Vote: A Replication
Dalton, K., Utts, J. M., Novotny, G., Sickafoose, L., Burrone, J., & Phillips, C. S. (2000) · Journal of Parapsychology · Cited by 9 · see citing works
ESP under the Ganzfeld, in Contrast with the Induction of Relaxation as a Psi-Conducive State
Parra, A. E., & Villanueva, J. (2006) · Australian journal of parapsychology · Cited by 9 · see citing works
Storm, L., & Barrett-Woodbridge, M. (2007) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 7 · see citing works
The Psi Phenomena: A Bayesian Approach to the Ganzfeld Procedure
Lau, M. Y.-K. (2022) · Figshare · doi:10.7274/1c18df67p4q · Cited by 6 · see citing works
Parra, A. E., & Villanueva, J. (2004) · Australian journal of parapsychology · Cited by 5 · see citing works
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288 · see citing works
Rejecting or Accepting Parameter Values in Bayesian Estimation
Kruschke, J. K. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771304 · Cited by 734 · see citing works
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492 · see citing works
Self-citationDefault Bayes Factors for Model Selection in Regression
Rouder, J. N., & Morey, R. D. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.734737 · Cited by 477 · see citing works
Self-citationBayesian Analysis Reporting Guidelines
Kruschke, J. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01177-7 · Cited by 462 · see citing works
The experimental evidence for parapsychological phenomena: A review.
Cardeña, E. (2018) · American Psychologist · doi:10.1037/amp0000236 · Cited by 221 · see citing works
Dougherty, M. R., Hamovitz, T., & Tidwell, J. W. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0865-9 · Cited by 106 · see citing works
Is There a Free Lunch in Inference?
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. (2016) · Topics in Cognitive Science · doi:10.1111/tops.12214 · Cited by 104 · see citing works
Self-citationVasilev, M. R., & Angele, B. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1147-x · Cited by 84 · see citing works
Auditory Distraction During Reading: A Bayesian Meta-Analysis of a Continuing Controversy
Vasilev, M. R., Kirkby, J. A., & Angele, B. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617747398 · Cited by 85 · see citing works
Kruschke, J. K., & Liddell, T. M. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2606016 · Cited by 57 · see citing works
Bayesian Inference and Testing Any Hypothesis You Can Specify
Etz, A. J., Haaf, J. M., Rouder, J. N., & Vandekerckhove, J. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918773087 · Cited by 44 · see citing works
Self-citationConstraining multiple controls on planktic foraminifera Mg/Ca
Holland, K., Branson, O., Haynes, L., Hönisch, B., Allen, K., Russell, A. D., Fehrenbacher, J., Spero, H., & Eggins, S. (2020) · Geochimica et Cosmochimica Acta · doi:10.1016/j.gca.2020.01.015 · Cited by 33 · see citing works
Rejecting or accepting parameter values in Bayesian estimationPreprint
Kruschke, J. K. (2018) · doi:10.31219/osf.io/s5vdy · Cited by 33 · see citing works
Storm, L., Sherwood, S. J., Roe, C. A., Tressoldi, P. E., Rock, A. J., & Di Risio, L. (2017) · International Journal of Dream Research · doi:10.11588/ijodr.2017.2.34888 · Cited by 29 · see citing works
Baptista, J., Derakhshani, M., Tressoldi, P. E., Cardeña, E., Palmer, J. A., & Eds, D. M.-C. (2014) · Cited by 29 · see citing works
Meta-Analysis of Free-Response Studies 2009-2018: Assessing the Noise-Reduction Model Ten Years OnPreprint
Storm, L., & Tressoldi, P. E. (2020) · doi:10.31234/osf.io/3d7at · Cited by 29 · see citing works
Kruschke, J. K., & Liddell, T. M. (2016) · doi:10.31219/osf.io/ksfyr · Cited by 28 · see citing works
Wulff, J. N., & Taylor, L. (2023) · Strategic Organization · doi:10.1177/14761270231214429 · Cited by 20 · see citing works
Storm, L., Tressoldi, P. E., & Utts, J. (2013) · Psychological Bulletin · doi:10.1037/a0029506 · Cited by 12 · see citing works
Investor Intuition Promotes Gender Equality in Access to Reward-Based Crowdfunding
Fellnhofer, K., & Deng, Y. (2023) · Entrepreneurship Theory and Practice · doi:10.1177/10422587231192150 · Cited by 17 · see citing works
Attentional fluctuations and the temporal organization of memory
Jayakumar, M., Balusu, C., & Aly, M. (2022) · Cognition · doi:10.1016/j.cognition.2023.105408 · Cited by 17 · see citing works
Vasilev, M. R., Yates, M. J., & Slattery, T. J. (2019) · Journal of Cognition · doi:10.5334/joc.87 · Cited by 16 · see citing works
Tressoldi, P. E., & Storm, L. (2021) · Journal of Anomalous Experience and Cognition · doi:10.31156/jaex.23206 · Cited by 19 · see citing works
Adaptive scaling of reward in episodic memory: a replication study
Mason, A., Ludwig, C. J. H., & Farrell, S. (2016) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2016.1233439 · Cited by 11 · see citing works
Tressoldi, P. E., & Storm, L. (2021) · F1000Research · doi:10.12688/f1000research.51746.1 · Cited by 13 · see citing works
A Skeptical Eye on PsiPreprint
Wagenmakers, E., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. J. (2020) · doi:10.31234/osf.io/syq2h · Cited by 10 · see citing works
More is better: Relative prevalence of multiple targets affects search accuracy
Cheng, P., & Rich, A. N. (2018) · Journal of Vision · doi:10.1167/18.4.2 · Cited by 9 · see citing works
Tressoldi, P. E., & Storm, L. (2021) · F1000Research · doi:10.12688/f1000research.24868.3 · Cited by 7 · see citing works
Bayesian inference and testing any hypothesis you can specifyPreprint
Etz, A. J., Haaf, J. M., Rouder, J. N., & Vandekerckhove, J. (2018) · doi:10.31234/osf.io/wmf3r · Cited by 16 · see citing works
Self-citation'Fringe Science'—A Tautology, Not Pariah
Houran, J., & Bauer, H. H. (2022) · Journal of Scientific Exploration · doi:10.31275/20222527 · Cited by 7 · see citing works
Tressoldi, P. E., & Storm, L. (2024) · F1000Research · doi:10.12688/f1000research.51746.3 · Cited by 5 · see citing works
Williams, B. (2019) · Journal of Scientific Exploration · doi:10.31275/2019/1667 · Cited by 6 · see citing works
Tressoldi, P. E., & Storm, L. (2023) · F1000Research · doi:10.12688/f1000research.51746.2 · Cited by 4 · see citing works
Tressoldi, P. E., & Storm, L. (2020) · F1000Research · doi:10.12688/f1000research.24868.1 · Cited by 3 · see citing works
Anomalous Perception: An Umbrella Review of the Meta-Analytic EvidencePreprint
Tressoldi, P. E., & Storm, L. (2020) · doi:10.31234/osf.io/5gwqb · Cited by 3 · see citing works
Disentangling Mechanisms from Causes: And the Effects on Science
Protzko, J. (2016) · Foundations of Science · doi:10.1007/s10699-016-9511-x · Cited by 2 · see citing works
Questionable evidence for prefrontal cortex as an alleged psi inhibitor
Pilacinski, A., Klaes, C., Friedman, J., & Wiesing, M. (2024) · Cortex · doi:10.1016/j.cortex.2024.01.001 · Cited by 2 · see citing works
Beyond Ancient and Modern Superstitions
Radin, D. I. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190905538.013.31 · Cited by 1 · see citing works
A Four Element-Themed Self-Selecting Mobile Application for ESP Testing
Laythe, B. R. (2023) · Journal of Scientific Exploration · doi:10.31275/20222789 · Cited by 1 · see citing works
The Bayesian New Statistics (Public)Preprint
Kruschke, J. K., & Liddell, T. M. (2017) · doi:10.31234/osf.io/wntsa · Cited by 1 · see citing works
Using bootstrapping to account for sampling variation and large study bias in meta-analysisPreprint
Schmitz, J., Packheiser, J., Παπαδάτου-Παστού, Μ., Ocklenburg, S., & McManus, I. C. (2022) · doi:10.31234/osf.io/4rcn7 · Cited by 1 · see citing works
JSE 36:2 Summer 2022 Whole Issue PDF
Erickson, K. E. (2022) · Journal of Scientific Exploration · doi:10.31275/20222713
Erickson, K. E. (2019) · Journal of Scientific Exploration · doi:10.31275/2019.1729
Tressoldi, P. E., & Storm, L. (2020) · F1000Research · doi:10.12688/f1000research.24868.2
Tressoldi, P. E., & Storm, L. (2024) · F1000Research · doi:10.12688/f1000research.51746.4 · Cited by 1 · see citing works
The Bayesian New Statistics (Public)Preprint
Kruschke, J. K., & Liddell, T. M. (2017) · doi:10.31234/osf.io/wntsa_v1
Glaubke, R., Sikes, E. L., Umling, N., Allen, K. A., & Schmidt, M. W. (2025) · Palaeogeography Palaeoclimatology Palaeoecology · doi:10.1016/j.palaeo.2025.113190
Bayesian inference and testing any hypothesis you can specifyPreprint
Haaf, J. M., Rouder, J. N., Vandekerckhove, J., & Etz, A. (2020) · OSF Preprints (OSF Preprints)
Self-citation
- Authors
- Rouder, J. N., Morey, R. D., & Province, J. M.
- Affiliations
- University of Missouri System (US)
- Venue
- Psychological Bulletin
- Published
- 2013
- Volume
- 139
- Issue
- 1
- Pages
- 241-247
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2016461963
- Cited by
- 48
- References
- 32
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 4 · 2020: 7 · 2019: 3 · 2018: 7 · 2017: 3 · 2016: 5 · 2015: 2 · 2014: 2 · 2013: 1 · 2012: 1
- FWCI
- 0.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion)
- Funders
- National Science Foundation (SES-1024080)
- Topics
- Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
- Social Sciences › Arts and Humanities › Philosophy › Religious Studies and Spiritual Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- extrasensory perception, Bayes factor, meta-analysis critique, randomization, replication failure, telepathy, precognition, clairvoyance
- MeSH terms
- Acoustic Stimulation, Humans, Models, Psychological, Parapsychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
