The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4×
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 2
- Z Maniadis, F Tufano, JA List, One swallow doesn’t make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104, 277–290 (2014).
- R Hanson, Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds EM Patashnik, AS Gerber (Brookings Institution Press, Washington, DC), pp. 151–173 (2006).
Cited by
Sorted by citations
Items 101–200 of 326
Improving Social Science: Lessons from the Open Science Movement
Engzell, P., & Rohrer, J. M. (2020) · PS Political Science & Politics · doi:10.1017/s1049096520000967 · Cited by 23
In situ measurement of cesium-137 contamination in fruits from the northern Marshall Islands
Topping, C. E. W., Abella, M. K. I. L., Berkowitz, M. E., Molina, M. R., Nikolic-Hughes, I., Hughes, E. W., & Ruderman, M. A. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903481116 · Cited by 22
Dimant, E., Clemente, E. G., Pieper, D. J., Dreber, A., Gelfand, M. J., Hallsworth, M., Holzwarth, A., & Tantia, P. (2021) · Scientific Reports · doi:10.1038/s41598-022-10524-1 · Cited by 22
Self-citationThe power of prediction markets
Mann, A. (2016) · Nature · doi:10.1038/538308a · Cited by 22
Heyard, R., Pawel, S., Frese, J., Voelkl, B., Würbel, H., McCann, S., Held, L., Wever, K. E., Hartmann, H., Townsin, L., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242076 · Cited by 21
Is preclinical research in cancer biology reproducible enough?
Kane, P. B., & Kimmelman, J. (2021) · eLife · doi:10.7554/elife.67527 · Cited by 20
LaPlante, D. A. (2019) · International Gambling Studies · doi:10.1080/14459795.2019.1672768 · Cited by 21
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 20
When averaging goes wrong: The case for mixture model estimation in psychological science.
Moreau, D., & Corballis, M. C. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000504 · Cited by 20
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22193 · Cited by 20
A modified sequential probability ratio test
Pramanik, S., Johnson, V. E., & Bhattacharya, A. (2021) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2021.102505 · Cited by 19
Preregistered Replication of the Auditory Deviant Effect: A Robust Benchmark Finding
Bell, R., Mieth, L., Röer, J. P., Troche, S. J., & Buchner, A. (2019) · Journal of Cognition · doi:10.5334/joc.64 · Cited by 19
Advancing the Cognitive Science of Religion through Replication and Open Science
Hoogeveen, S., & van Elk, M. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.39039 · Cited by 21
Bayes Factor Design Analysis: Planning for Compelling EvidencePreprint
Schhnbrodt, F. D., & Wagenmakers, E. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2722435 · Cited by 19
Prediction-market innovations can improve climate-risk forecasts
Roulston, M. S., Kaplan, T. R., Day, B., & Kaivanto, K. (2022) · Nature Climate Change · doi:10.1038/s41558-022-01467-6 · Cited by 17
Subjective Confidence as a Monitor of the Replicability of the Response
Koriat, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231224387 · Cited by 22
Stability of Experimental Results: Forecasts and Evidence
DellaVigna, S., & Pope, D. G. (2019) · American Economic Journal Microeconomics · doi:10.1257/mic.20200129 · Cited by 17
Coville, A., & Vivalt, E. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5nsh3 · Cited by 17
The replication crisis has led to positive structural, procedural, and community changesPreprint
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M. M., Breznau, N., Robertson, O. M., Kalandadze, T., et al. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/r6cvx · Cited by 18
Examining the replicability of online experiments selected by a decision market
Holzmeister, F., Johannesson, M., Camerer, C. F., Chen, Y., Ho, T., Hoogeveen, S., Huber, J., Imai, N., Imai, T., Jin, L. J., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02062-9 · Cited by 20
Self-citationOptimizing Research Output: How Can Psychological Research Methods Be Improved?
Miller, J. O., & Ulrich, R. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-094927 · Cited by 20
Benjamin, D. M., Hey, S. P., MacPherson, A., Hachem, Y., Smith, K. S., Zhang, S. X., Wong, S., Dolter, S., Mandel, D. R., & Kimmelman, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262862 · Cited by 16
How failure to falsify in high-volume science contributes to the replication crisis
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2022) · eLife · doi:10.7554/elife.78830 · Cited by 18
Francioli, S. P., Shakeri, A., & North, M. S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311009121 · Cited by 15
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15
How (not) to measure replication
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00377-2 · Cited by 16
Understanding of researcher behavior is required to improve data reliability
Wass, M. N., Ray, L., & Michaelis, M. (2019) · GigaScience · doi:10.1093/gigascience/giz017 · Cited by 19
Wu, J. F., Nivargi, R., Lanka, S. S. T., Menon, A. M., Modukuri, S. A., Nakshatri, N. S., Wei, X., Wang, Z., Caverlee, J., Rajtmajer, S. M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04580 · Cited by 22
Improving the reproducibility of findings by updating research methodology
Klein, J. (2021) · Quality & Quantity · doi:10.1007/s11135-021-01196-6 · Cited by 19
Boyce, V., Mathur, M. B., & Frank, M. C. (2023) · Royal Society Open Science · doi:10.1098/rsos.231240 · Cited by 14
Neves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., de Abreu, M. B., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · Memórias do Instituto Oswaldo Cruz · doi:10.1590/0074-02760200328 · Cited by 11
The Role of Open Science Practices in Scaling Evidence-Based Prevention Programs
Supplee, L. H., Ammerman, R. T., DUGGAN, A. K., List, J. A., & Suskind, D. L. (2021) · Prevention Science · doi:10.1007/s11121-021-01322-8 · Cited by 13
Coles‐Black, J., Ong, S., Teh, J., Kearns, P., Ischia, J. J., Bolton, D. M., & Lawrentschuk, N. (2022) · Journal of Robotic Surgery · doi:10.1007/s11701-022-01401-0 · Cited by 13
Journal-based replication of experiments: An application to “Being Chosen to Lead”
Allan Drazen, Y., Dreber, A., Ozbay, E. Y., & Snowberg, E. (2021) · Journal of Public Economics · doi:10.1016/j.jpubeco.2021.104482 · Cited by 14
Self-citationThe Informational Content of Prices When Policy Makers React to Financial Markets
Siemroth, C. (2018) · Journal of Economic Theory · doi:10.1016/j.jet.2018.11.002 · Cited by 13
How Do We Choose Our Giants? Perceptions of Replicability in Psychological Science
Alister, M., Vickers-Jones, R., Sewell, D. K., & Ballard, T. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211018199 · Cited by 15
Generation Next: Experimentation with AI
Charness, G., Jabarian, B., & List, J. A. (2023) · National Bureau of Economic Research · doi:10.3386/w31679 · Cited by 15
Information markets and nonmarkets
Bergemann, D., & Ottaviani, M. (2021) · Handbook of Industrial Organization · doi:10.1016/bs.hesind.2021.11.008 · Cited by 12
Butera, L., Grossman, P. J., Houser, D. E., List, J. A., & Villeval, M. C. (2020) · National Bureau of Economic Research · doi:10.3386/w26801 · Cited by 12
Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicabilityPreprint
Ebersole, C. R., Nosek, B. A., Kidwell, M. C., Buttrick, N., Baranski, E. N., Chartier, C. R., Mathur, M. B., IJzerman, H., Lazarević, L. B., Corker, K. S., et al. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/jmnsq · Cited by 12
Self-citationCalnan, M., Kirchin, S., Roberts, D. L., Wass, M. N., & Michaelis, M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.107043 · Cited by 11
Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Marcoci, A., Wilkinson, D. P., Vercammen, A., Wintle, B. C., Abatayo, A. L., Baskin, E., Berkman, H., Buchanan, E. M., Capitán, S., Capitán, T., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01961-1 · Cited by 14
Self-citationStock Markets, Market Crashes, and Market Bubbles
Andraszewicz, S. (2020) · Springer eBooks · doi:10.1007/978-3-030-45500-2_10 · Cited by 11
Viganola, D., Buckles, G., Chen, Y., Diego‐Rosell, P., Johannesson, M., Nosek, B. A., Pfeiffer, T., Siegel, A. C., & Dreber, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.181308 · Cited by 11
Self-citationBrown, M. I., Grossenbacher, M. A., Martín‐Raugh, M. P., Kochert, J. F., & Prewett, M. S. (2021) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12353 · Cited by 11
Efficient alternatives for Bayesian hypothesis tests in psychology.
Pramanik, S., & Johnson, V. E. (2022) · Psychological Methods · doi:10.1037/met0000482 · Cited by 11
The Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · National Bureau of Economic Research · doi:10.3386/w25451 · Cited by 12
Participating in a climate prediction market increases concern about global warming
Cerf, M., Matz, S., & MacIver, M. A. (2023) · Nature Climate Change · doi:10.1038/s41558-023-01679-4 · Cited by 11
Bickel, D. R. (2020) · The American Statistician · doi:10.1080/00031305.2020.1816214 · Cited by 10
Intergenerational transmission of risk attitudes in Burkina Faso
Sepahvand, M. H., & Shahbazian, R. (2020) · Empirical Economics · doi:10.1007/s00181-020-01857-9 · Cited by 10
Prediction Markets for Science: Is the Cure Worse than the Disease?
Thicke, M. (2017) · Social Epistemology · doi:10.1080/02691728.2017.1346720 · Cited by 9
Axén, G., & Cortis, D. (2020) · Risks · doi:10.3390/risks8030088 · Cited by 9
Prospects for a transparency revolution in the field of business history
Smith, A. D. A., & Umemura, M. (2018) · Business History · doi:10.1080/00076791.2018.1439019 · Cited by 9
Forecasting the publication and citation outcomes of COVID-19 preprints
Gordon, M., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Tran, L., Twardy, C. R., et al. (2022) · Royal Society Open Science · doi:10.1098/rsos.220440 · Cited by 9
Self-citationEliciting group judgements about replicability: a technical implementation of the IDEA Protocol
Pearson, R., Fraser, H. S., Bush, M., Mody, F., Widjaja, I., Head, A., Wilkinson, D. P., Sinnott, R., Wintle, B. C., Burgman, M. A., et al. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.055 · Cited by 8
Replication Markets: Results, Lessons, Challenges and Opportunities in AI ReplicationPreprint
Liu, Y., Gordon, M., Wang, J., Bishop, M., Chen, Y., Pfeiffer, T., Twardy, C. R., & Viganola, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04543 · Cited by 8
Self-citationThe Reliability of Replications: A Study in Computational ReproductionsPreprint
Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Akdenız, E., Álvarez-Benjumea, A., Andersen, H. K., Auer, D., et al. (2021) · doi:10.31235/osf.io/j7qta · Cited by 8
Responsible Science MattersPreprint
Faff, R. (2021) · Social Science Research Network · doi:10.2139/ssrn.3880341 · Cited by 2
Fanelli, D., Tan, P. B., Amaral, O. B., & Neves, K. (2025) · Royal Society Open Science · doi:10.1098/rsos.241446 · Cited by 7
Hidden Indicators of Collective Intelligence in Crowdfunding
Horvát, E.-Á., Dambanemuya, H. K., Uparna, J. S., & Uzzi, B. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583414 · Cited by 7
Chu, J., Voelkel, J. G., Stagnaro, M. N., Kang, S., Druckman, J. N., Rand, D. G., & Willer, R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2307008121 · Cited by 7
Priestley, D. R., Staph, J., Koneru, S., Rajtmajer, S. M., Cwiek, A. P., Vervoordt, S. M., & Hillary, F. G. (2022) · Brain Communications · doi:10.1093/braincomms/fcac322 · Cited by 7
Brandon, A., & List, J. A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521417112 · Cited by 7
Improving decisions with market information: an experiment on corporate prediction markets
Dianat, A., & Siemroth, C. (2020) · Experimental Economics · doi:10.1007/s10683-020-09654-y · Cited by 7
Generation Next: Experimentation with AIPreprint
Charness, G., Jabarian, B., & List, J. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4576689 · Cited by 7
The reliability of replications: a study in computational reproductions
Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Álvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.241038 · Cited by 6
Rostgaard, K. (2023) · Scientific Reports · doi:10.1038/s41598-023-31838-8 · Cited by 7
Price formation in field prediction markets: The wisdom in the crowd
Bossaerts, F., Yadav, N., Bossaerts, P. L., Nash, C., Todd, T., Rudolf, T., Hutchins, R., Ponsonby, A., & Mattingly, K. (2024) · Journal of Financial Markets · doi:10.1016/j.finmar.2023.100881 · Cited by 6
5. Improving the Reliability of Judgements
Wintle, B. C., Mukherjee, N., Hemming, V., Canessa, S., & McBride, M. F. (2022) · Open Book Publishers · doi:10.11647/obp.0321.05 · Cited by 6
Does the left-digit bias affect prices in financial markets?
Heraud, F., & Page, L. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.10.012 · Cited by 6
Misplaced Intuitions in Interventions to Reduce Attractiveness-Based Discrimination
Axt, J. R., Yang, J., & Deshpande, H. (2022) · Personality and Social Psychology Bulletin · doi:10.1177/01461672221074748 · Cited by 10
Accuracy-Risk Trade-Off Due to Social Learning in Crowd-Sourced Financial Predictions
Adjodah, D., Leng, Y., Chong, S. K., Krafft, P. M., Moro, E., & Pentland, A. (2021) · Entropy · doi:10.3390/e23070801 · Cited by 8
Editorial: Replicability in Cognitive Science
Strickland, B., & Cruz, H. D. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-021-00531-y · Cited by 6
Generation Next: Experimentation with AIPreprint
Charness, G., Jabarian, B., & List, J. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574623 · Cited by 6
Whose Preferences Matter for Redistribution? Cross-Country Evidence
Maréchal, M. A., Cohn, A., Yusof, J., & Fisman, R. J. (2024) · Journal of Political Economy Microeconomics · doi:10.1086/732238 · Cited by 5
Elicitation of expectations using Colonel Blotto
Peeters, R. J. A. P., & Wolk, L. (2018) · Experimental Economics · doi:10.1007/s10683-018-9596-x · Cited by 5
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.01 · Cited by 5
Reliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
Network Signatures of Success: Emulating Expert and Crowd Assessment in Science, Art, and Technology
Zakhlebin, I., & Horvát, E.-Á. (2017) · Studies in computational intelligence · doi:10.1007/978-3-319-72150-7_36 · Cited by 5
Arvan, M., Bright, L. K., & Heesen, R. (2022) · The British Journal for the Philosophy of Science · doi:10.1086/719117 · Cited by 5
Design and analysis of a synthetic prediction market using dynamic convex sets
Nakshatri, N. S., Menon, A. M., Giles, C. L., Rajtmajer, S. M., & Griffin, C. H. (2021) · Results in Control and Optimization · doi:10.1016/j.rico.2021.100052 · Cited by 5
Agricultural Productivity in Burkina Faso: The Role of Gender and Risk AttitudesPreprint
Sepahvand, M. H. (2022) · RePEc: Research Papers in Economics · Cited by 5
Pandemic preparation without romance: insights from public choice
Tabarrok, A. T. (2025) · Public Choice · doi:10.1007/s11127-025-01277-2 · Cited by 5
A Train Wreck by Any Other Name
Harris, C., Rohrer, D., & Pashler, H. (2021) · Psychological Inquiry · doi:10.1080/1047840x.2021.1889317 · Cited by 4
11. Creating a Culture of Evidence Use
Frick, W. F., Cooke, S. J., Dickson, I., Jackson, H. A., Keller, K. E., la Luz, A. D., Parks, D., McPherson, T., Simkins, A. T., Tanshi, I., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.11 · Cited by 4
Replication for Quantitative Research
Jeremy Freese, A., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
The endogenous formation of common pool resource coalitions
Chávez, C. A., Murphy, J. J., Quezada, F. J., & Stranlund, J. K. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.04.028 · Cited by 4
Quantitative Macroeconomics: Lessons Learned from Fourteen Replications
Kirkby, R. (2022) · Computational Economics · doi:10.1007/s10614-022-10234-w · Cited by 4
2. Gathering and Assessing Pieces of Evidence
White, T., Amano, T., Boersch‐Supan, P. H., Christie, A. P., Freckleton, R. P., Quinzin, M. C., Rezaie, A. M., Sutherland, W. J., & Yamashita, H. (2022) · Open Book Publishers · doi:10.11647/obp.0321.02 · Cited by 4
Pot la psicologia rescatar-se a si mateixa?. Incentius, biaix i replicabilitat
Fernando, B., César, P. L. J., & Vadillo, M. A. (2018) · Anuari de Psicologia de la Societat Valenciana de Psicologia · doi:10.7203/anuari.psicologia.18.2.231 · Cited by 5
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
7. Framing the Problem and Identifying Potential Solutions
Esmail, N., Green, R. E., Petrovan, S. O., Salafsky, N., Taylor, N. G., & Wilson, J. D. (2022) · Open Book Publishers · doi:10.11647/obp.0321.07 · Cited by 4
Post-Publication Peer Review for RealPreprint
Ikeda, K., Yamada, Y., & Takahashi, K. (2020) · doi:10.31234/osf.io/sp3j5 · Cited by 4
Probabilistic forecasting of replication studiesPreprint
Pawel, S., & Held, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/fhwb7 · Cited by 4
Coville, A., & Vivalt, E. (2017) · doi:10.31222/osf.io/5nsh3_v1 · Cited by 4
Predicting replication outcomes and generalizability of results
Dreber, A. (2025) · European Review of Agricultural Economics · doi:10.1093/erae/jbaf035 · Cited by 3
Self-citationChandrashekar, S. P., Viganola, D., Dreber, A., Johannesson, M., Pfeiffer, T., Siegel, A., & Feldman, G. (2026) · Royal Society Open Science · doi:10.1098/rsos.250377 · Cited by 3
Self-citation10. How Conservation Practice Can Generate Evidence
Ockendon, N., Cadotte, M. W., Eklund, J., Ferraro, P. J., Fisher, M., Hancock, M. H., & Worthington, T. A. (2022) · Open Book Publishers · doi:10.11647/obp.0321.10 · Cited by 4
Metadata
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
