The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3×
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4×
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055
Wolfers, J., & Zitzewitz, E. (2004) · Journal of Economic Perspectives · doi:10.1257/0895330041371321 · Cited by 889
On the relative independence of thinking biases and cognitive ability
Stanovich, K. E., & West, R. F. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.672 · Cited by 840
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818
Purdie-Vaughns, V., Steele, C. M., Davies, P. G., Ditlmann, R., & Crosby, J. R. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.615 · Cited by 786
The Value of Believing in Free Will
Vohs, K. D., & Schooler, J. W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02045.x · Cited by 659
Shnabel, N., & Nadler, A. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.116 · Cited by 618
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Nosek, B. A., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 592
Promoting Transparency in Social Science Research
Miguel, E. A., Camerer, C. F., Casey, K., Cohen, J., Esterling, K., Gerber, A. S., Glennerster, R., GREEN, D. P., Humphreys, M., Imbens, G. W., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 559
Self-citationPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
Self-citationHedonic and instrumental motives in anger regulation
Tamir, M., Mitchell, C., & Gross, J. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02088.x · Cited by 412
Detecting the Snake in the Grass
LoBue, V., & DeLoache, J. S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02081.x · Cited by 409
(2008) · doi:10.1126/science.1157679 · Cited by 406
LOGARITHMIC MARKETS CORING RULES FOR MODULAR COMBINATORIAL INFORMATION AGGREGATION
Hanson, R. D. (2012) · The Journal of Prediction Markets · doi:10.5750/jpm.v1i1.417 · Cited by 381
Rule, N. O., & Ambady, N. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02054.x · Cited by 368
Halevy, N., Bornstein, G., & Sagiv, L. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02100.x · Cited by 355
Masicampo, E. J., & Baumeister, R. F. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02077.x · Cited by 331
Interpreting the predictions of prediction markets
Manski, C. F. (2006) · Economics Letters · doi:10.1016/j.econlet.2006.01.004 · Cited by 330
Nairne, J. S., Pandeirada, J. N. S., & Thompson, S. R. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02064.x · Cited by 319
Dynamics of self-regulation: How (un)accomplished goal actions affect motivation.
Koo, M., & Fishbach, A. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.2.183 · Cited by 303
Schmidt, J. R., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.514 · Cited by 297
One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects
Maniadis, Z., Tufano, F., & List, J. A. (2014) · American Economic Review · doi:10.1257/aer.104.1.277 · Cited by 280
Alter, A. L., & Oppenheimer, D. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02062.x · Cited by 256
Förster, J., Liberman, N., & Kuschel, S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.579 · Cited by 207
Interpreting Prediction Market Prices as ProbabilitiesPreprint
Wolfers, J. J., & Zitzewitz, E. W. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.898597 · Cited by 202
Hewitt, J. K. (2012) · Behavior Genetics · doi:10.1007/s10519-011-9504-z · Cited by 190
The cost of lower self-esteem: Testing a self- and social-bonds model of health.
Stinson, D. A., Logel, C., Zanna, M. P., Holmes, J. G., Cameron, J. J., Wood, J. V., & Spencer, S. J. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.412 · Cited by 177
Fischer, P. A., Greitemeyer, T., & Frey, D. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.382 · Cited by 173
Estes, Z., Verges, M., & Barsalou, L. W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02051.x · Cited by 168
Multidimensional visual statistical learning.
Turk‐Browne, N. B., Isola, P., Scholl, B. J., & Treat, T. A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.399 · Cited by 163
Chapter 80 Results from a Dozen Years of Election Futures Markets Research
Berg, J., Forsythe, R., Nelson, F., & Rietz, T. (2008) · Handbook of Experimental Economics Results · doi:10.1016/s1574-0722(07)00080-7 · Cited by 161
Working memory costs of task switching.
Liefooghe, B., Barrouillet, P., Vandierendonck, A., & Camos, V. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.478 · Cited by 154
PREDICTION MARKETS: AN EXTENDED LITERATURE REVIEW
Tziralis, G., & Tatsiópoulos, I. P. (2012) · The Journal of Prediction Markets · doi:10.5750/jpm.v1i1.421 · Cited by 152
Interpreting Prediction Market Prices as Probabilities
Wolfers, J. J., & Zitzewitz, E. W. (2006) · National Bureau of Economic Research · doi:10.3386/w12200 · Cited by 149
The Threat of Appearing Prejudiced and Race-Based Attentional Biases
Richeson, J. A., & Trawalter, S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02052.x · Cited by 141
Mirman, D., & Magnuson, J. S. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.65 · Cited by 141
Fischer, P., Schulz‐Hardt, S., & Frey, D. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.2.94.2.231 · Cited by 129
Decision making and learning while taking sequential risks.
Pleskac, T. J. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.1.167 · Cited by 121
WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
Ioannidis, J. P. A., & Doucouliagos, H. (2013) · Journal of Economic Surveys · doi:10.1111/joes.12032 · Cited by 120
An attention-based associative account of adjacent and nonadjacent dependency learning.
Pacton, S., & Perruchet, P. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.1.80 · Cited by 119
Semantic interference in a delayed naming task: Evidence for the response exclusion hypothesis.
Janssen, N., Schirm, W., Mahon, B. Z., & Caramazza, A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.249 · Cited by 115
"Walking on eggshells": How expressing relationship insecurities perpetuates them.
Lemay, E. P., & Clark, M. S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.420 · Cited by 112
Modelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103
McCrea, S. M. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.274 · Cited by 103
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 101
1/f noise and effort on implicit measures of bias.
Correll, J. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.48 · Cited by 93
Perceptual Mechanisms That Characterize Gender Differences in Decoding Women's Sexual Intent
Farris, C. A., Treat, T. A., Viken, R. J., & McFall, R. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02092.x · Cited by 93
Temporal Selection is Suppressed, Delayed, and Diffused During the Attentional Blink
Vul, E., Nieuwenstein, M., & Kanwisher, N. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02046.x · Cited by 89
Yap, M. J.-M., Balota, D. A., Tse, C., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.495 · Cited by 89
Could gambling save science? Encouraging an honest consensus
Hanson, R. (1995) · Social Epistemology · doi:10.1080/02691729508578768 · Cited by 82
The ultimate sampling dilemma in experience-based decision making.
Fiedler, K. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.186 · Cited by 80
Preschoolers' Perspective Taking in Word Learning
Nurmsoo, E., & Bloom, P. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02069.x · Cited by 80
Replication effort provokes praise—and ‘bullying’ charges
Bohannon, J. (2014) · Science · doi:10.1126/science.344.6186.788 · Cited by 72
Loving Those Who Justify Inequality
Lau, G. P., Kay, A. C., & Spencer, S. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02040.x · Cited by 68
A single-system account of the relationship between priming, recognition, and fluency.
Berry, C. J., Shanks, D. R., & Henson, R. N. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.97 · Cited by 67
Priming addition facts with semantic relations.
Bassok, M., Pedigo, S. F., & Oskarsson, A. T. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.343 · Cited by 65
The effects of an implemental mind-set on attitude strength.
Henderson, M. D., de Liver, Y., & Gollwitzer, P. M. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.396 · Cited by 59
Bressan, P., & Stranieri, D. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02060.x · Cited by 38
Intentional forgetting is easier after two "shots" than one.
Sahakyan, L., Delaney, P. F., & Waldum, E. R. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.408 · Cited by 36
Stereotypes and retrieval-provoked illusory source recollections.
Dodson, C. S., Darragh, J., & Williams, A. L. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.460 · Cited by 33
Contextual effects on reading aloud: Evidence for pathway control.
Reynolds, M., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.50 · Cited by 25
An Experiment on Prediction Markets in Science
Almenberg, J., Kittlitz, K., & Pfeiffer, T. (2009) · PLoS ONE · doi:10.1371/journal.pone.0008500 · Cited by 24
Self-citationWolf, S. T., Insko, C. A., Kirchner, J. L., & Wildschut, T. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.479 · Cited by 19
Now you see it, now you don't: Repetition blindness for nonwords.
Morris, A. L., & Still, M. L. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.146 · Cited by 15
Eliciting Predictions and Recommendations for Decision Making
Chen, Y., Kash, I. A., Ruberry, M. E., & Shnayder, V. (2014) · ACM Transactions on Economics and Computation · doi:10.1145/2556271 · Cited by 15
Self-citationAccounting for occurrences: A new view of the use of contingency information in causal judgment.
White, P. A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.204 · Cited by 14
Intentional Forgetting Is Easier After Two "Shots" Than One
Sahakyan, L., Delaney, P. F., & Waldum, E. R. (2007) · PsycEXTRA Dataset · doi:10.1037/e527342012-084 · Cited by 2
Janssen, N., Schirm, W., Mahon, B., & Caramazza, A. (2007) · PsycEXTRA Dataset · doi:10.1037/e512682013-476
Research works printed but not linked: 2
- Z Maniadis, F Tufano, JA List, One swallow doesn’t make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104, 277–290 (2014).
- R Hanson, Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds EM Patashnik, AS Gerber (Brookings Institution Press, Washington, DC), pp. 151–173 (2006).
Cited by
Sorted by citations
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Self-citationVariability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
Metadata
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
