The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4×
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Research works printed but not linked: 2
- Z Maniadis, F Tufano, JA List, One swallow doesn’t make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104, 277–290 (2014).
- R Hanson, Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds EM Patashnik, AS Gerber (Brookings Institution Press, Washington, DC), pp. 151–173 (2006).
Cited by
Sorted by citations
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Self-citationVariability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationBayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 810
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 484
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 471
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. G. (2017) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 402
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
How open is innovation? A retrospective and ideas forward
Dahlander, L., Gann, D., & Wallin, M. W. (2021) · Research Policy · doi:10.1016/j.respol.2021.104218 · Cited by 284
Three Recommendations for Improving the Use of p -Values
Benjamin, D. J., & Berger, J. O. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543135 · Cited by 273
Redefine statistical significancePreprint
Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E., Berk, R. A., Bollen, K. A., Brembs, B., Brown, L., Camerer, C. F., et al. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mky9j · Cited by 251
Self-citationNon est Disputandum de Generalizability? A Glimpse into The External Validity Trial
List, J. A. (2020) · National Bureau of Economic Research · doi:10.3386/w27535 · Cited by 203
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189
Self-citationPredicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2018) · Journal of Political Economy · doi:10.1086/699976 · Cited by 183
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 191
Self-citationNonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177
A review on blockchain technology and blockchain projects fostering open science
Leible, S., Schlager, S., Schubotz, M., & Gipp, B. (2019) · Frontiers in Blockchain · doi:10.3389/fbloc.2019.00016 · Cited by 134
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
Self-citationPredict science to improve science
DellaVigna, S., Pope, D. G., & Vivalt, E. (2019) · Science · doi:10.1126/science.aaz1704 · Cited by 140
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
#EEGManyLabs: Investigating the replicability of influential EEG experiments
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2021) · Cortex · doi:10.1016/j.cortex.2021.03.013 · Cited by 148
Self-citation(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 154
Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses
Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.12.007 · Cited by 126
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22566 · Cited by 117
Achievement Gaps in the Wake of COVID-19
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021) · Educational Researcher · doi:10.3102/0013189x211011237 · Cited by 134
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
Self-citationReproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104
Estimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 99
Mulder, J., & Wagenmakers, E. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.01.002 · Cited by 107
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 101
Self-citationThe Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · Southern Economic Journal · doi:10.1002/soej.12392 · Cited by 102
Predicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225826 · Cited by 94
Self-citationA discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 99
Schaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87
Self-citationReplications in Comparative Cognition: What Should We Expect and How Can We Improve?
Farrar, B. G., Boeckle, M., & Clayton, N. S. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.01.02.2020 · Cited by 90
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 94
Many Labs 2: Investigating Variation in Replicability Across Sample and SettingPreprint
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M. E., Axt, J., Babalola, M. T., Bahník, Š., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9654g · Cited by 87
Low replicability can support robust and efficient science
Lewandowsky, S., & Oberauer, K. (2020) · Nature Communications · doi:10.1038/s41467-019-14203-0 · Cited by 83
Systematizing Confidence in Open Research and Evidence (SCORE)Preprint
Alipourfard, N., Arendt, B., Benjamin, D. M., Benkler, N., Bishop, M. M., Burstein, M. H., Bush, M., Caverlee, J., Chen, Y., Clark, C., et al. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/46mnb · Cited by 75
Self-citationOptimally generate policy-based evidence before scaling
List, J. A. (2024) · Nature · doi:10.1038/s41586-023-06972-y · Cited by 74
Perceived crisis and reforms: Issues, explanations, and remedies.
de Boeck, P. A. L., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 71
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 73
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 67
Self-citationLaypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 62
How Effective Is (More) Money? Randomizing Unconditional Cash Transfer Amounts in the USPreprint
Jaroszewicz, A., Jachimowicz, J., Hauser, O., & Jamison, J. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4154000 · Cited by 72
Beyond Statistical Ritual: Theory in Psychological Science
Proulx, T., & Morey, R. D. (2021) · Perspectives on Psychological Science · doi:10.1177/17456916211017098 · Cited by 64
Transparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/9a3rw · Cited by 58
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 54
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 66
Self-citationAmano, T., Bako, L. A., Best, M., Boenisch, N., Boersch‐Supan, P. H., Browne, D., Buckley, Y. M., Burgman, M., Cadotte, M. W., Canessa, S., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321 · Cited by 61
Eitan, O., Viganola, D., Inbar, Y., Dreber, A., Johannesson, M., Pfeiffer, T., Thau, S., & Uhlmann, E. L. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.06.004 · Cited by 55
Self-citationA creative destruction approach to replication: Implicit work and sex morality across cultures
Tierney, W., Hardy, J. H., Ebersole, C. R., Viganola, D., Clemente, E. G., Gordon, M., Hoogeveen, S., Haaf, J. M., Dreber, A., Johannesson, M., et al. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104060 · Cited by 54
Self-citationChopra, F., Haaland, I., Roth, C., & Stegmann, A. (2023) · The Economic Journal · doi:10.1093/ej/uead060 · Cited by 57
How Much Information Is Incorporated into Financial Asset Prices? Experimental Evidence
Page, L., & Siemroth, C. (2018) · Review of Financial Studies · doi:10.1093/rfs/hhaa143 · Cited by 47
Credibility crisis in agricultural economics
Ferraro, P. J., & Shukla, P. (2022) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13323 · Cited by 48
Consequences of relying on statistical significance: Some illustrations
CALSTER, B. V., Steyerberg, E. W., Collins, P. G. S., & Smits, T. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.12912 · Cited by 47
Gordon, M., Viganola, D., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Twardy, C. R., et al. (2020) · Royal Society Open Science · doi:10.1098/rsos.200566 · Cited by 47
Self-citationGordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44
Self-citationQuestionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 50
fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
Yücel, M. A., Luke, R., Mesquita, R. C., von Lühmann, A., Mehler, D. M. A., Lührs, M., Gemignani, J., Abdalmalak, A., Albrecht, F., Ivo, I. D. A., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08412-1 · Cited by 46
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 49
Self-citationA Replication Study of Operations Management Experiments in Management Science
Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S. M., Leider, S., Long, X., & Tong, J. D. (2022) · Management Science · doi:10.1287/mnsc.2023.4866 · Cited by 38
How to increase public support for carbon pricing with revenue recycling
Woerner, A., Imai, T., Pace, D. D., & Schmidt, K. M. (2024) · Nature Sustainability · doi:10.1038/s41893-024-01466-9 · Cited by 38
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 48
The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 35
The Practicalities of Running Randomized Evaluations
Glennerster, R. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.002 · Cited by 35
Multiple testing with covariate adjustment in experimental economics
List, J. A., Shaikh, A. M., & Vayalinkal, A. (2023) · Journal of Applied Econometrics · doi:10.1002/jae.2985 · Cited by 33
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32
Blockchain for Science and Knowledge Creation
Bartling, S. H. (2018) · Springer eBooks · doi:10.1007/978-3-662-57611-3_10 · Cited by 32
The Heuristic Value of p in Inductive Statistical Inference
Krueger, J. I., & Heck, P. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00908 · Cited by 33
The Replication Crisis in Epidemiology: Snowball, Snow Job, or Winter Solstice?
Lash, T. L., Collin, L. J., & Dyke, M. E. V. (2018) · Current Epidemiology Reports · doi:10.1007/s40471-018-0148-x · Cited by 35
Probabilistic forecasting of replication studies
Pawel, S., & Held, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231416 · Cited by 35
Predicting and reasoning about replicability using structured groups
Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H. S., Hemming, V., Thorn, F. S., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221553 · Cited by 31
Lohmann, P. M., Pizzo, A., Bauer, J. M., Khanna, T. M., Flecke, S. L., Callaghan, M., Minx, J. C., & Reisch, L. A. (2026) · Nature Food · doi:10.1038/s43016-025-01279-9 · Cited by 35
P in the right place: Revisiting the evidential value of P‐values
Lytsy, P. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12319 · Cited by 33
Fraser, H. S., Bush, M., Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Gould, E., Hemming, V., Hamilton, D. G., Rumpff, L., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0274429 · Cited by 36
Bayesian markets to elicit private information
Baillon, A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1703486114 · Cited by 30
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 32
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
Wilkening, T. S., Martinie, M. F. R. C., & Howe, P. D. L. (2021) · Management Science · doi:10.1287/mnsc.2020.3919 · Cited by 27
Managing and defending the commons: Experimental evidence from TURFs in Chile
Chávez, C. A., Murphy, J. J., & Stranlund, J. K. (2018) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2018.07.004 · Cited by 27
Who Should Do Replication Labor?
Romero, F. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918803619 · Cited by 28
Dai, H., Young, H. P., Durant, T. J. S., Gong, G., Kang, M., Krumholz, H. M., Schulz, W. L., & Jiang, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03662 · Cited by 34
Can large language models help predict results from a complex behavioural science study?
Lippert, S., Dreber, A., Johannesson, M., Tierney, W., Cyrus-Lai, W., Uhlmann, E. L., Emotion Expression Collaboration, & Pfeiffer, T. (2024) · Royal Society Open Science · doi:10.1098/rsos.240682 · Cited by 32
Self-citationReproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 33
Large language models can predict the results of social science experiments
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026) · Nature · doi:10.1038/s41586-026-10742-x · Cited by 41
von Hippel, P. T. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211072525 · Cited by 26
Olofsson, J., Larsson, M. U., Roa, C., Wilson, D. A., & Laukka, E. J. (2019) · Behavior Genetics · doi:10.1007/s10519-019-09980-9 · Cited by 27
#EEGManyLabs: Investigating the Replicability of Influential EEG ExperimentsPreprint
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/528nr · Cited by 26
Self-citationPreregistration and reproducibility
Strømland, E. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2019.01.006 · Cited by 26
Predicting the Replicability of Social Science Lab ExperimentsPreprint
Altmejd, A., Dreber, A., Forsell, E., Ho, T., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/zamry · Cited by 23
Self-citationA Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 26
Self-citationUsing the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 22
Metadata
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
