The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4×
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 2
- Z Maniadis, F Tufano, JA List, One swallow doesn’t make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104, 277–290 (2014).
- R Hanson, Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds EM Patashnik, AS Gerber (Brookings Institution Press, Washington, DC), pp. 151–173 (2006).
Cited by
Sorted by citations
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Self-citationVariability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Metadata
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
