Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Large-scale collaborative projects recently demonstrated that several key findings from the social-science literature could not be replicated successfully. Here, we assess the extent to which a finding’s replication success relates to its intuitive plausibility. Each of 27 high-profile social-science findings was evaluated by 233 people without a Ph.D. in psychology. Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%. We discuss the prediction patterns and apply signal detection theory to disentangle detection ability from response bias. Our study suggests that laypeople’s predictions contain useful information for assessing the probability that a given finding will be replicated successfully.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“At the level of individual studies, problems concern low statistical power (Button et al., 2013; Ioannidis, 2005) and questionable research practices, such as data-driven flexibility in statistical analysis (i.”
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Bayesian Cognitive Modeling: A Practical Course
Lee, M. D., & Wagenmakers, E.-J. (2014) · doi:10.1017/cbo9781139087759 · Cited by 1,795
Self-citationGoogle Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
Sparrow, B., Liu, J., & Wegner, D. M. (2011) · Science · doi:10.1126/science.1207745 · Cited by 1,762
Reading literary fiction improves theory of mind
Kidd, D. C., & Castano, E. (2013) · Science · doi:10.1126/science.1239918 · Cited by 1,642
Some Consequences of Having Too Little
Shah, A. K., Mullainathan, S., & Shafir, E. (2012) · Science · doi:10.1126/science.1222426 · Cited by 1,567 · cited 1×
“Finally, most participants correctly predicted that the effect found in the study in the bottom panel (i.e., Shah, Mullainathan, & Shafir, 2012) would fail to be replicated.”
Research works printed but not linked: 37
- 10.1177/0956797611434537 · DOI · Paper page
- 10.1126/science.1224313 · DOI · Paper page
- 10.1126/science.1211180 · DOI · Paper page
- 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 · DOI · Paper page
- 10.1016/j.cortex.2012.12.016 · DOI · Paper page
- 10.1002/bdm.586 · DOI · Paper page
- 10.1038/nature12774 · DOI · Paper page
- 10.1016/j.jesp.2016.01.008 · DOI
- 10.1126/science.1221936 · DOI · Paper page
- Etz A., 2019, Poster presented at the annual meeting of the Association for Psychological Science
- 10.1016/j.joep.2018.10.009 · DOI · Paper page
- 10.1016/j.jmp.2013.02.003 · DOI
- 10.1016/j.obhdp.2006.10.001 · DOI · Paper page
- 10.1126/science.1253932 · DOI · Paper page
- Green D. M., 1966, Signal detection theory and psychophysics
- 10.1017/S0140525X18000675 · DOI
- 10.1038/nature13530 · DOI · Paper page
- 10.1371/journal.pmed.0020124 · DOI · Paper page
- Jeffreys H., 1939, Theory of probability
- Jeffreys H., 1961, Theory of probability
- 10.2307/2291091 · doi:10.2307/2291091 · Paper page
- 10.1111/j.1467-9574.2005.00279.x · DOI
- 10.1126/science.1190792 · DOI · Paper page
- 10.1126/science.1186799 · DOI · Paper page
- 10.1126/science.1195701 · DOI · Paper page
- 10.1177/2515245918797607 · DOI · Paper page
- 10.1038/nature15392 · DOI · Paper page
- 10.1037/e519702015-014 · DOI · Paper page
- 10.1037/h0058700 · DOI · Paper page
- Tversky A., 1978, Cognition and Categorization, 1, 79
- van Doorn J., Ly A., Marsman M., Wagenmakers E.J. (2017). Bayesian latent-normal inference for the rank sum test, the signed rank test, and Spearman’s ρ. arXiv. Retrieved from https://arxiv.org/abs/1712.06941 · arXiv
- 10.1037/a0036731 · DOI · Paper page
- 10.1037/a0022790 · DOI · Paper page
- 10.1177/2515245918767122 · DOI
- 10.1126/science.1250830 · DOI · Paper page
- 10.1038/nclimate2093 · DOI · Paper page
- 10.1126/science.1130726 · DOI · Paper page
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
“Though many prediction markets are composed of researchers or students with research training, even lay people seem to perform better than chance in predicting replicability 100 .”
BackgroundEstimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13
“This finding might be expected, given that the replicability of findings in biomedical science is probably less intuitive than that of psychology studies, for which prediction accuracy is above chance even among laypeople (Hoogeveen et al., 2020).”
Metadata
- Authors
- Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E.-J.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2020-08-21
- Volume
- 3
- Issue
- 3
- Pages
- 267-285
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W3080884520
- Cited by
- 64
- References
- 30
- FWCI
- 11.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (Talent grant (406-17-568), Vici grant (016.Vici.170.083))
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully. https://app.cochat.ai/papers/10.1177/2515245920919667
- The paper
- Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E.-J. (2020). Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920919667
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.