A Bayes factor meta-analysis of Bem’s ESP claim
Open accessAbstract
In recent years, statisticians and psychologists have provided the critique that p-values do not capture the evidence afforded by data and are, consequently, ill suited for analysis in scientific endeavors. The issue is particular salient in the assessment of the recent evidence provided for ESP by Bem (2011) in the mainstream Journal of Personality and Social Psychology. Wagenmakers, Wetzels, Borsboom, and van der Maas (Journal of Personality and Social Psychology, 100, 426-432, 2011) have provided an alternative Bayes factor assessment of Bem's data, but their assessment was limited to examining each experiment in isolation. We show here that the variant of the Bayes factor employed by Wagenmakers et al. is inappropriate for making assessments across multiple experiments, and cannot be used to gain an accurate assessment of the total evidence in Bem's data. We develop a meta-analytic Bayes factor that describes how researchers should update their prior beliefs about the odds of hypotheses in light of data across several experiments. We find that the evidence that people can feel the future with neutral and erotic stimuli to be slight, with Bayes factors of 3.23 and 1.57, respectively. There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors' analysis approach closely followed the methods used by Rouder and Morey (2011) and Rouder et al. (2013).
“Our analysis approach closely followed the methods used by Rouder and Morey (2011) and Rouder et al. (2013) in their meta-analyses of psi (i.e., extrasensory perception).”
From Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null (2016) · cites this paper as Rouder and Morey (2011)
The shortcomings of null hypothesis significance testing are well documented.
“The limitations of NHST methods are well documented and need not be rehashed here in their entirety”
From Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null (2016) · cites this paper as Rouder and Morey (2011)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Continuous Univariate Distributions.
Omori, Y., Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995) · Journal of the American Statistical Association · doi:10.2307/2291547 · Cited by 9,085 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Self-citationA practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · see citing works
Exposure and affect: Overview and meta-analysis of research, 1968–1987.
Bornstein, R. F. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.106.2.265 · Cited by 1,970 · cited 1× · see citing works
“In the vast majority of conventional mereexposure effect studies, participants prefer previously viewed stimuli (Bornstein, 1989).”
BackgroundBayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · see citing works
Mixtures of g Priors for Bayesian Variable Selection
Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001337 · Cited by 1,308 · see citing works
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Lindley, D. V. (1957) · Biometrika · doi:10.1093/biomet/44.1-2.187 · Cited by 946 · see citing works
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · see citing works
Continuous Univariate Distributions.Vol. 1
Kocherlakota, S., Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995) · Biometrics · doi:10.2307/2533292 · Cited by 861 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 789 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.2307/2289131 · Cited by 548 · see citing works
Journal of the Royal Statistical Society.
(1923) · The Economic Journal · doi:10.1093/ej/33.132.593 · Cited by 535 · see citing works
Posterior odds ratios for selected regression hypotheses
Zellner, A., & Siow, A. (1980) · Trabajos de Estadistica y de Investigacion Operativa · doi:10.1007/bf02888369 · Cited by 490 · see citing works
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392 · see citing works
Storm, L., Tressoldi, P. E., & Di Risio, L. (2010) · Psychological Bulletin · doi:10.1037/a0019457 · Cited by 140 · see citing works
Kass, R. E. (1993) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348679 · Cited by 120 · see citing works
Gönen, M., Johnson, W. O., Lu, Y., & Westfall, P. H. (2005) · The American Statistician · doi:10.1198/000313005x55233 · Cited by 100 · see citing works
Affective habituation: Subliminal exposure to extreme stimuli decreases their extremity.
Dijksterhuis, A., & Smith, P. K. (2002) · Emotion · doi:10.1037/1528-3542.2.3.203 · Cited by 99 · see citing works
Extending conventional priors for testing general hypotheses in linear models
Bayarri, M. J., & García‐Donato, G. (2007) · Biometrika · doi:10.1093/biomet/asm014 · Cited by 77 · see citing works
THE STRENGTH OF EVIDENCE VERSUS THE POWER OF BELIEF: ARE WE ALL BAYESIANS?
Utts, J. M., Norris, M., Suess, E. A., & Johnson, W. O. (2010) · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Jeffreys, H. (1961). Theory of probability (3rd ed.). New York: Oxford University Press.
- Wagenmakers, E.-J. (2007). A practical solution to the pervasive problem of p values. Psychonomic Bulletin &Review, 14, 779- 804.
- Zellner, A. (1986). On assessing prior distirbutions and Bayesian regression analysis with g-prior distribution. In P. K. Goel & A. Zellner (Eds.), Bayesian inference and decision techniques: Essays in honour of Bruno de Finetti (pp. 233-243). Amsterdam: North Holland.
Cited by
Sorted by citations
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · see citing works
Default Bayes factors for ANOVA designs
Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.08.001 · Cited by 1,800 · see citing works
Self-citationWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288 · see citing works
Optional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492 · see citing works
Self-citationDefault Bayes Factors for Model Selection in Regression
Rouder, J. N., & Morey, R. D. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.734737 · Cited by 477 · see citing works
Self-citationReal-Time Strategy Game Training: Emergence of a Cognitive Flexibility Trait
Glass, B. D., Maddox, W. T., & Love, B. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070350 · Cited by 227 · see citing works
Stomach-brain synchrony reveals a novel, delayed-connectivity resting-state network in humans
Rebollo, I., Devauchelle, A.-D., Béranger, B., & Tallon-Baudry, C. (2018) · eLife · doi:10.7554/elife.33321 · Cited by 194 · see citing works
Dougherty, M. R., Hamovitz, T., & Tidwell, J. W. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0865-9 · Cited by 106 · see citing works
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105 · see citing works
The Attentional Blink Reveals the Probabilistic Nature of Discrete Conscious Perception
Asplund, C. L., Fougnie, D., Zughni, S., Martin, J. W., & Marois, R. (2014) · Psychological Science · doi:10.1177/0956797613513810 · Cited by 92 · see citing works
Gamelike features might not improve data
Hawkins, G. E., Rae, B., Nesbitt, K. V., & Brown, S. D. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0264-3 · Cited by 61 · see citing works
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61 · see citing works
Giromini, L., Ales, F., de Campora, G., Zennaro, A., & Pignolo, C. (2017) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-017-9611-0 · Cited by 56 · see citing works
Rebollo, I., & Tallon-Baudry, C. (2022) · The Journal of Neuroscience · doi:10.1523/jneurosci.1285-21.2021 · Cited by 46 · see citing works
Rouder, J. N., Morey, R. D., & Province, J. M. (2013) · Psychological Bulletin · doi:10.1037/a0029008 · Cited by 48 · see citing works
Self-citationStorm, L., Sherwood, S. J., Roe, C. A., Tressoldi, P. E., Rock, A. J., & Di Risio, L. (2017) · International Journal of Dream Research · doi:10.11588/ijodr.2017.2.34888 · Cited by 29 · see citing works
- Authors
- Rouder, J. N., & Morey, R. D.
- Affiliations
- University of Missouri (US)
- University of Groningen (NL)
- Corresponding author
- Rouder, Jeffrey N.
- Venue
- Psychonomic Bulletin & Review
- Published
- 2011-05-15
- Volume
- 18
- Issue
- 4
- Pages
- 682-689
- ISSN
- 1069-9384 (ISSN-L)
- 1531-5320
- OpenAlex
- W2155061836
- Cited by
- 194
- References
- 22
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 7 · 2023: 6 · 2022: 12 · 2021: 21 · 2020: 17 · 2019: 13 · 2018: 18 · 2017: 17 · 2016: 22 · 2015: 13 · 2014: 14 · 2013: 11 · 2012: 10
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Psychonomic Bulletin & Review (publishedVersion)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, other-oa)
- ORCA Online Research @Cardiff (Cardiff University) (submittedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Paranormal Experiences and Beliefs
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- extrasensory perception, Bayes factor, prior updating, erotic stimuli, emotional valence, skepticism
- MeSH terms
- Bayes Theorem, Data Interpretation, Statistical, Humans, Parapsychology, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
