The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4×
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 2
- Z Maniadis, F Tufano, JA List, One swallow doesn’t make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104, 277–290 (2014).
- R Hanson, Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds EM Patashnik, AS Gerber (Brookings Institution Press, Washington, DC), pp. 151–173 (2006).
Cited by
Sorted by citations
Items 201–300 of 326
Conceptual replication study of fifteen JDM effects: Insights from the Polish sample
Sobków, A., Sobczak, A., Latacz, K., Gambuś, P., Tyc, A., Goryczka, K., Barcik, K., Janosz, A., Hetmańczuk, W., Gonczar, T., et al. (2022) · Polish Psychological Bulletin · doi:10.24425/ppb.2022.141862 · Cited by 5
Greenslade, P., Florentine, S. K., Hansen, B. D., & Gell, P. (2016) · Environmental Monitoring and Assessment · doi:10.1007/s10661-016-5478-x · Cited by 3
The Role of Mindsets and Attitudes in Determining Future Awareness
Valerio, M. (2019) · Anticipation Science · doi:10.1007/978-3-030-03623-2_2 · Cited by 3
Extending the price constraints of betting markets
Axén, G., & Cortis, D. (2019) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2019.07.001 · Cited by 3
Avoiding Bias in the Search for Implicit Bias
Cyrus-Lai, W., Tierney, W., du Plessis, C., Nguyen, M., Schaerer, M., Clemente, E. G., & Uhlmann, E. L. (2022) · Psychological Inquiry · doi:10.1080/1047840x.2022.2106762 · Cited by 3
What should comparative cognition expect from direct replication studies?Preprint
Farrar, B. G., & Clayton, N. S. (2019) · doi:10.31234/osf.io/4urm8 · Cited by 3
Locus of Control and Prosocial BehaviorPreprint
Andor, M., Cox, J. C., Gerster, A., Price, M., Sommer, S., & Tomberg, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4192186 · Cited by 3
The Impact of Joint versus Separate Prediction Mode on Forecasting Accuracy
Imas, A., Jung, M. H., Saccardo, S., & Vosgerau, J. (2022) · National Bureau of Economic Research · doi:10.3386/w30611 · Cited by 3
Whose Preferences Matter for Redistribution: Cross-country Evidence
Maréchal, M. A., Cohn, A., Yusof, J., & Fisman, R. J. (2023) · National Bureau of Economic Research · doi:10.3386/w31974 · Cited by 3
Whose Preferences Matter for Redistribution: Cross-Country EvidencePreprint
Maréchal, M. A., Cohn, A., Yusof, J., & Fisman, R. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4676764 · Cited by 3
The Alternative Factors Leading to Replication Crisis: Prediction and Evaluation
Chernov, G. (2024) · Evaluation Review · doi:10.1177/0193841x241229106 · Cited by 3
Weidener, L., & Shilina, S. (2025) · Frontiers in Blockchain · doi:10.3389/fbloc.2025.1650188 · Cited by 3
Making prepublication independent replication mainstream
Tierney, W., Schweinsberg, M., & Uhlmann, E. L. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000894 · Cited by 2
What’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2
8 Making Decisions for Policy and Practice
Cook, C. N., Burgman, M. A., Rumpff, L., & Thornton, A. (2022) · Open Book Publishers · doi:10.11647/obp.0321.08 · Cited by 3
The Language of (Non)Replicable Social Science
Herzenstein, M., Rosario, S., Oblander, S., & Netzer, O. (2024) · Psychological Science · doi:10.1177/09567976241254037 · Cited by 2
Huber, C., Holzmeister, F., Johannesson, M., König-Kersting, C., Dreber, A., Huber, J., & Kirchler, M. (2026) · The Journal of Finance · doi:10.2139/ssrn.5048949 · Cited by 3
Self-citationTransparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 2
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 3
Amano, T., Bako, L. A., Best, M., Boenisch, N., Boersch‐Supan, P. H., Browne, D., Buckley, Y. M., Burgman, M. A., Cadotte, M. W., Canessa, S., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.12 · Cited by 2
4. Presenting Conclusions from Assessed Evidence
Sutherland, W. J., Amano, T., Boenisch, N., Cheng, S. H., Christie, A. P., Godfray, H. C. J., Grainger, M., McLean, A. R., Morgan, W. H., Ockendon, N., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.04 · Cited by 2
A Prototype Hybrid Prediction Market for Estimating Replicability of Published Work
Chakravorti, T., Fraleigh, R. D., Fritton, T., McLaughlin, M., Singh, V. P., Griffin, C. H., Kwasnica, A. M., Pennock, D. M., Giles, C. L., & Rajtmajer, S. M. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230093 · Cited by 2
Gruen, A., Mattingly, K., Morwitch, E., Bossaerts, F., Clifford, M., Nash, C., Ioannidis, J. P. A., & Ponsonby, A. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104783 · Cited by 2
ChatGPT struggles to recognize reproducible science
Chang, J. R., & Nordling, T. E. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02428-z · Cited by 3
A regression-based algorithm for frequent itemsets mining
Jia, Z., & Wang, Z. (2019) · Data Technologies and Applications · doi:10.1108/dta-03-2019-0037 · Cited by 2
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 2
Can Expert Prediction Markets Forecast Climate-Related Risks?
Roulston, M. S., & Kaivanto, K. (2024) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-24-0135.1 · Cited by 2
Betting and belief: Prediction markets and attribution of climate change
Nay, J. J., der Linden, M. V., & Gilligan, J. M. (2016) · Winter Simulation Conference · doi:10.1109/wsc.2016.7822215 · Cited by 2
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2
Laypeople Can Predict Which Social Science Studies ReplicatePreprint
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2019) · doi:10.31234/osf.io/egw9d · Cited by 2
Locus of Control and Prosocial BehaviorPreprint
Andor, M., Cox, J. C., Gerster, A., Price, M., Sommer, S., & Tomberg, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4190181 · Cited by 2
Policy Choice and the Wisdom of CrowdsPreprint
Otis, N. G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4200841 · Cited by 4
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v5 · Cited by 2
Crowdsourced versus large language models forecasting: evidence for the accuracy–correlation effect
Jeddi, Y., Segovia‐Martín, J., & Servan-Schreiber, E. (2026) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2024.0456 · Cited by 1
9. Creating Evidence-Based Policy and Practice
Downey, H., Boenisch, N., Browne, D., Buckley, Y. M., Cranston, G. R., Dicks, L. V., Hughes, J., Latrémouille, C., Morgan, W. H., Muir, M. J., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.09 · Cited by 1
Psychology and Law, Meet Open Science
McAuliff, B. D., Fessinger, M. B., Perillo, A. D., & Perillo, J. T. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197649138.013.5 · Cited by 1
Joint-outcome prediction markets for climate risks
Roulston, M. S., & Kaivanto, K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309164 · Cited by 1
Do “Evidence‐Based Policy” Clearinghouses Provide Good Advice for Local Policymakers?
Orr, L. L. (2025) · Journal of Policy Analysis and Management · doi:10.1002/pam.70077 · Cited by 1
Reproducibility of Survey Results: A New Method to Quantify Similarity of Human Subject Pools
Khamesi, A. R., Musmeci, R., Silvestri, S., & Baker, D. A. (2020) · Globecom · doi:10.1109/globecom42002.2020.9348076 · Cited by 1
Key methodological issues in researching gifted education and advanced academics
McBee, M. T. (2021) · Routledge eBooks · doi:10.4324/9781003025443-1 · Cited by 1
6. Identifying Stakeholders and Collaborating with Communities
Best, M., Irvine, R. L., Bako, L. A., Citegetse, G., Field, A., Roe, D., Rose, D. C., & Spencer, J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.06 · Cited by 1
How Market Prices React to Information: Evidence from Binary Options Markets
Gauriot, R., & Page, L. (2025) · The Economic Journal · doi:10.1093/ej/ueaf040 · Cited by 1
Self-Resolving Prediction Markets for Unverifiable Outcomes
Srinivasan, S., Karger, E., & Chen, Y. (2025) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3736252.3742593 · Cited by 1
Self-citationChapter 2. Experimental research
Cremers, H. R. (2017) · John Benjamins Publishing Company eBooks · doi:10.1075/z.210.02cre · Cited by 1
Sibling correlation in risk attitudes: evidence from Burkina Faso
Sepahvand, M. H., & Shahbazian, R. (2020) · The Journal of Economic Inequality · doi:10.1007/s10888-020-09466-3 · Cited by 1
Securities Based Decision Markets
Wang, W., & Pfeiffer, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-94662-3_6 · Cited by 1
Self-citationGerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 1
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 1
RES: An Interpretable Replicability Estimation System for Research Publications
Wang, Z., Feng, Q., Chatterjee, M., Zhao, X., Liu, Y., Li, Y., Singh, A. K., Shipman, F. M., III, Hu, X., & Caverlee, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21737 · Cited by 1
3. Assessing Collated and Synthesised Evidence
Sutherland, W. J., Cheng, S. H., Christie, A. P., Cooke, S. J., Koricheva, J., Randall, N., Smith, R. K., & Stewart, R. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.03 · Cited by 1
Ting, C., & Montgomery, M. (2023) · Social Epistemology · doi:10.1080/02691728.2023.2177128 · Cited by 2
Betting and Belief: Prediction Markets and Attribution of Climate ChangePreprint
Nay, J. J., der Linden, M. V., & Gilligan, J. M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2808284 · Cited by 1
The Dozen Things Experimental Economists Should Do (More Of)Preprint
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3313734 · Cited by 2
Butera, L., Grossman, P. J., Houser, D. E., & Villeval, M. C. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3557796 · Cited by 1
Thompson, J., Dreber, A., Gaunt, T. R., Gordon, M., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Lyon, M., Penton‐Voak, I. S., et al. (2022) · doi:10.31222/osf.io/gsc8f · Cited by 1
Self-citationNeves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., Abreu, M., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/njs6k · Cited by 1
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v1 · Cited by 1
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v2 · Cited by 1
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v4 · Cited by 1
Psychophysiology in 2024: Publishing on the science of brain, body, and behavior
Keil, A. (2023) · Psychophysiology · doi:10.1111/psyp.14482 · Cited by 1
Whose Preferences Matter for Redistribution: Cross-country EvidencePreprint
Maréchal, M. A., Cohn, A., Yusof, J., & Fisman, R. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4675162 · Cited by 1
Billionaire Superstar: Public Image and Demand for TaxationPreprint
Truglia, R. N. P., & Yusof, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4901045 · Cited by 1
AI Behavioral SciencePreprint
Jackson, M. O., Mei, Q., Wang, S., Xie, Y., Yuan, W., Benzell, S. G., Brynjolfsson, E., Evans, J. A., Jabarian, B., Kleinberg, J. M., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5395006 · Cited by 2
Self-citationEssays on Information and Beliefs in Dynamic Choices
Descamps, A. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.121077
Smalheiser, N. R. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-811306-6.00006-3
The Brazilian Reproducibility Initiative: a systematic assessment of Brazilian biomedical sciencePreprint
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2018) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ahf7t
The Price of Science: MOR's Organization Behavior Editorial Area
Huang, X. (2019) · Management and Organization Review · doi:10.1017/mor.2019.6
Does revolution change risk attitudes? Evidence from Burkina Faso
Sepahvand, M. H., Shahbazian, R., & Swain, R. B. (2024) · Journal of International Development · doi:10.1002/jid.3934
Psychologists' betting market hints at most reliable research findings
Hayden, E. C. (2015) · Nature · doi:10.1038/nature.2015.18754
Foretelling What Makes People Pay: Predicting the Results of Field Experiments on TV Fee EnforcementPreprint
Chadimová, K. (2021) · RePEc: Research Papers in Economics · doi:10.17632/9kgrdn2yhx.1
How Can Corporate Prediction Markets Improve Manager Decisions? An Experimental StudyPreprint
Dianat, A., & Siemroth, C. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3345569
Multiple-Study Replication Initiatives
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0008
Meta Research: Replication of Significant Results --- Modeling the Effects of p-HackingPreprint
Ulrich, R., & Miller, J. O. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/tu7zr
Protzko, J., Holcombe, A. O., & Simons, D. J. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/pq8m4
A Journal-Based Replication of 'Being Chosen to Lead'Preprint
Allan Drazen, Y., Dreber, A., Ozbay, E. Y., & Snowberg, E. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3480708
Self-citationThe limitations of scientific consensus
Prelec, D. (2021) · doi:10.5644/pi2021.200.11
Ograničenja naučnog konsenzusa
Prelec, D. (2021) · doi:10.5644/pi2021.200.01
新しくて古い心理学のかたちPreprint
Takahashi, K. (2019) · doi:10.31234/osf.io/5pzy7
Liao, T. F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.004
Chapter 1: Key Methodological Issues in Researching Gifted Education and Advanced AcademicsPreprint
McBee, M. T. (2022) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/mfdr2
Buttrick, P. (2019) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2019.04.005 · Cited by 35
Elman, C., Gerring, J., & Mahoney, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.019
Faculty Opinions recommendation of Nonreplicable publications are cited more than replicable ones.
Brembs, B. (2021) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.740150836.793586309
ResponseofOutputtoVaryingIncentiveStructuresonAmazonTurk
DellaVigna, S., Wheaton, B., Charpentier, D., & Welch, K. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.714
Dunning, T., Grossman, G., Humphreys, M., Hyde, S., McIntosh, C., & Nellis, G. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108381390.012
ExpertForecastsofRealEffortExperiment
DellaVigna, S., Ali, A., Welch, K., & Charpentier, D. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.731
Almgren, H., Parsons, N., Bossche, S. V. D., & Marinazzo, D. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/9hg3j
Expert Forecasts of Real-Effort Experiment
DellaVigna, S., Charpentier, D., Welch, K., & Ali, A. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.731-3.0
Response of Output to Varying Incentive Structures on Amazon Turk
DellaVigna, S., Welch, K., Charpentier, D., & Wheaton, B. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.714-4.0
Response of Output to Varying Incentive Structures on Amazon Turk
DellaVigna, S., Welch, K., Charpentier, D., & Wheaton, B. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.714-3.0
Expert Forecasts of Real-Effort Experiment
DellaVigna, S., Charpentier, D., Welch, K., & Ali, A. (2015) · AEA Randomized Controlled Trials · doi:10.1257/rct.731-2.0
A literature review of methods for assessment of reproducibility in sciencePreprint
Nordling, T. E. M., & Peralta, T. M. (2022) · Research Square · doi:10.21203/rs.3.rs-2267847/v3
13. Supplementary Material from Online Resources
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.13
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.14
The Impact of Joint versus Separate Prediction Mode on Forecasting AccuracyPreprint
Imas, A., Jung, M. H., Saccardo, S., & Vosgerau, J. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4258470
Name-Induced Stereotypes Do Not Affect Investors’ ChoicesPreprint
Adams, J. C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4389948
Billionaire Superstar: Public Image and Demand for TaxationPreprint
Truglia, R. N. P., & Yusof, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4892458
Metadata
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
