Using prediction markets to estimate the reproducibility of scientific research
Open accessAbstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the comparison of reproducibility indicators, the fraction of studies that replicated with P<0.05 in the original direction was significantly higher in this experimental economics project (61.1%, 11/18) than in the RPP psychological sciences project (36.1%, 35/97) (chi-square χ2=3.96, P=0.047).
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
Experimental — accuracy not yet measured'Prediction market beliefs' differed significantly between this experimental economics project (75.1%, n=18) and the RPP psychological sciences project (55.1%, n=44) (Mann-Whitney U test, z=3.21, P=0.001).
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The Results report a mean final prediction-market price of 55% for 44 psychology studies, consistent with the cited 55.1% after rounding. Our copy does not report the experimental-economics figure of 75.1% for 18 studies or the between-project Mann–Whitney test (z=3.21, P=0.001). (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe prediction market methodology used in this study is similar to the one used in Dreber et al. (26).
“The prediction market methodology used in this study is similar to the one used in Dreber et al. ( 26 ).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
The RPP prediction market study found a significant Pearson correlation between prediction market beliefs and replication outcomes of 0.42.
“That study found a significant Pearson correlation between the prediction market beliefs and the replication outcomes of 0.42,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. (26) study and a sample of economists participated in this study).
“The trading platforms and the participant pool differed between the two prediction market studies (a sample of psychologists participated in the Dreber et al. ( 26 ) study and a sample of economists, see above, participated in this study).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
A recent prediction market study on replications in psychology found that predictions revealed by market prices are correlated with actual replication outcomes and are more strongly correlated than surveyed beliefs.
“More recently a prediction market study on replications in psychology has yielded promising results ( 26 ), in the sense that predictions revealed by market prices are correlated with actual replication outcomes (and are more strongly correlated than surveyed beliefs).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
In the RPP prediction markets, the Pearson correlation between survey beliefs and replication outcomes was 0.27 and non-significant.
“For the survey the Pearson correlation to the replication outcomes was 0.27 and non-significant in the RPP prediction markets,”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Contrary to Dreber et al. (26), prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
Dreber et al. (26) presented prediction market results for 44 studies in RPP.
“Dreber et al. ( 26 ) presented prediction market results for 44 studies in RPP.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Dreber et al. [26]
Contrary to a recent prediction market study on a subset of the RPP studies (n=44), the prediction market in this study does not predict replication outcomes better than the survey.
“Contrary to a recent prediction market study on a subset of the studies (n=44) included in the RPP project ( 26 ), the prediction market thus does not predict replication outcomes better than the survey.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Prediction markets can aggregate private information on reproducibility and can generate and disseminate a consensus among market participants.
“Prediction markets can aggregate private information on reproducibility, and can generate and disseminate a consensus among market participants.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The fraction of studies with 'Original effect size within replication 95% CI' did not differ significantly between this experimental economics project (66.7%, 12/18) and the RPP psychological sciences project (47.4%, 45/95) (chi-square χ2=2.25, P=0.133).
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe fraction of studies with 'Meta-analytic estimate significant in the original direction' did not differ significantly between this experimental economics project (77.8%, 14/18) and the RPP psychological sciences project (68.0%, 51/75) (chi-square χ2=0.66, P=0.417).
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
Experimental — accuracy not yet measured'Survey beliefs' differed significantly between this experimental economics project (71.1%, n=18) and the RPP psychological sciences project (53.7%, n=43) (Mann-Whitney U test, z=4.89, P<0.001).
“Also for 'Survey beliefs' the same test was used, and this difference (71.1% (n=18) versus 53.7% (n=43)) was also significant (z=4.89, P<0.001).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The Results section reports survey predictions for the psychology replication project, but does not compare them with an experimental economics project. Searches for 71.1%, 53.7%, and a Mann–Whitney test found no passage reporting the stated difference, sample sizes, or significance test. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.
“The RPP project did not directly report the relative effect size of the replication, but instead used the 'effect size difference' as a reproducibility indicator.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
The paper’s reproducibility | replications | prediction markets section defines its own successful-replication criterion as a result with P < 0.05 in the same direction as the original. Neither that passage nor the passages retrieved for “effect size difference” and “relative effect size” say whether the RPP reported relative effect sizes or used effect size difference as an indicator. Page numbers are unavailable in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measuredThe 'Replication effect-size (% of original effect size)' did not differ significantly between this experimental economics project (65.7%, n=18) and the RPP psychological sciences project (44.5%, n=94) (Mann-Whitney U test, z=1.39, P=0.166).
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [26]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 3× · see citing works
“Lower statistical power increases the probability that a statistically significant effect is a false positive (4, 14).”
BackgroundSelf-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 4× · see citing works
“J Exp Psychol Learn Mem Cogn 34(2):408–414.”
BackgroundInfluentialHealth insurance and the demand for medical care: evidence from a randomized experiment.
Manning, W. G., Newhouse, J. P., Duan, N., Keeler, E. B., Leibowitz, A. A., & Marquis, M. S. (1987) · PubMed · Cited by 2,045 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536 · see citing works
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · see citing works
Wolfers, J., & Zitzewitz, E. (2004) · Journal of Economic Perspectives · doi:10.1257/0895330041371321 · Cited by 889 · see citing works
On the relative independence of thinking biases and cognitive ability
Stanovich, K. E., & West, R. F. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.672 · Cited by 840 · see citing works
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · see citing works
Purdie-Vaughns, V., Steele, C. M., Davies, P. G., Ditlmann, R., & Crosby, J. R. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.615 · Cited by 786 · see citing works
The Value of Believing in Free Will
Vohs, K. D., & Schooler, J. W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02045.x · Cited by 659 · see citing works
Shnabel, N., & Nadler, A. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.116 · Cited by 618 · see citing works
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 605 · see citing works
Promoting Transparency in Social Science Research
Miguel, E. A., Camerer, C. F., Casey, K., Cohen, J., Esterling, K., Gerber, A. S., Glennerster, R., GREEN, D. P., Humphreys, M., Imbens, G. W., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 544 · see citing works
Self-citationPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Self-citationHedonic and instrumental motives in anger regulation
Tamir, M., Mitchell, C., & Gross, J. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02088.x · Cited by 412 · see citing works
(2008) · doi:10.1126/science.1157679 · Cited by 406 · see citing works
Detecting the Snake in the Grass
LoBue, V., & DeLoache, J. S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02081.x · Cited by 382 · see citing works
LOGARITHMIC MARKETS CORING RULES FOR MODULAR COMBINATORIAL INFORMATION AGGREGATION
Hanson, R. D. (2012) · The Journal of Prediction Markets · doi:10.5750/jpm.v1i1.417 · Cited by 381 · see citing works
Halevy, N., Bornstein, G., & Sagiv, L. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02100.x · Cited by 355 · see citing works
Rule, N. O., & Ambady, N. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02054.x · Cited by 342 · see citing works
Masicampo, E. J., & Baumeister, R. F. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02077.x · Cited by 331 · see citing works
Interpreting the predictions of prediction markets
Manski, C. F. (2006) · Economics Letters · doi:10.1016/j.econlet.2006.01.004 · Cited by 330 · see citing works
Nairne, J. S., Pandeirada, J. N. S., & Thompson, S. R. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02064.x · Cited by 319 · see citing works
Schmidt, J. R., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.514 · Cited by 297 · see citing works
Dynamics of self-regulation: How (un)accomplished goal actions affect motivation.
Koo, M., & Ayelet Fishbach, B. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.2.183 · Cited by 291 · see citing works
One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects
Maniadis, Z., Tufano, F., & List, J. A. (2014) · American Economic Review · doi:10.1257/aer.104.1.277 · Cited by 280 · see citing works
Alter, A. L., & Oppenheimer, D. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02062.x · Cited by 243 · see citing works
Förster, J., Liberman, N., & Kuschel, S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.579 · Cited by 207 · see citing works
Interpreting Prediction Market Prices as ProbabilitiesPreprint
Wolfers, J. J., & Zitzewitz, E. W. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.898597 · Cited by 202 · see citing works
Hewitt, J. K. (2012) · Behavior Genetics · doi:10.1007/s10519-011-9504-z · Cited by 190 · see citing works
The cost of lower self-esteem: Testing a self- and social-bonds model of health.
Stinson, D. A., Logel, C., Zanna, M. P., Holmes, J. G., Cameron, J. J., Wood, J. V., & Spencer, S. J. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.412 · Cited by 177 · see citing works
Estes, Z., Verges, M., & Barsalou, L. W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02051.x · Cited by 168 · see citing works
Multidimensional visual statistical learning.
Turk‐Browne, N. B., Isola, P., Scholl, B. J., & Treat, T. A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.399 · Cited by 163 · see citing works
Fischer, P. A., Greitemeyer, T., & Frey, D. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.382 · Cited by 162 · see citing works
Chapter 80 Results from a Dozen Years of Election Futures Markets Research
Berg, J., Forsythe, R., Nelson, F., & Rietz, T. (2008) · Handbook of Experimental Economics Results · doi:10.1016/s1574-0722(07)00080-7 · Cited by 161 · see citing works
Working memory costs of task switching.
Liefooghe, B., Barrouillet, P., Vandierendonck, A., & Camos, V. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.478 · Cited by 154 · see citing works
PREDICTION MARKETS: AN EXTENDED LITERATURE REVIEW
Tziralis, G., & Tatsiópoulos, I. P. (2012) · The Journal of Prediction Markets · doi:10.5750/jpm.v1i1.421 · Cited by 152 · see citing works
Interpreting Prediction Market Prices as Probabilities
Wolfers, J. J., & Zitzewitz, E. W. (2006) · National Bureau of Economic Research · doi:10.3386/w12200 · Cited by 149 · see citing works
The Threat of Appearing Prejudiced and Race-Based Attentional Biases
Richeson, J. A., & Trawalter, S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02052.x · Cited by 138 · see citing works
Mirman, D., & Magnuson, J. S. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.65 · Cited by 138 · see citing works
Fischer, P., Schulz‐Hardt, S., & Frey, D. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.2.94.2.231 · Cited by 129 · see citing works
Decision making and learning while taking sequential risks.
Pleskac, T. J. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.1.167 · Cited by 121 · see citing works
An attention-based associative account of adjacent and nonadjacent dependency learning.
Pacton, S., & Perruchet, P. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.1.80 · Cited by 119 · see citing works
WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
Ioannidis, J. P. A., & Doucouliagos, H. (2013) · Journal of Economic Surveys · doi:10.1111/joes.12032 · Cited by 118 · see citing works
Semantic interference in a delayed naming task: Evidence for the response exclusion hypothesis.
Janssen, N., Schirm, W., Mahon, B. Z., & Caramazza, A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.249 · Cited by 115 · see citing works
"Walking on eggshells": How expressing relationship insecurities perpetuates them.
Lemay, E. P., & Clark, M. S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.420 · Cited by 112 · see citing works
Modelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103 · see citing works
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 101 · see citing works
McCrea, S. M. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.274 · Cited by 100 · see citing works
1/f noise and effort on implicit measures of bias.
Correll, J. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.48 · Cited by 93 · see citing works
Perceptual Mechanisms That Characterize Gender Differences in Decoding Women's Sexual Intent
Farris, C. A., Treat, T. A., Viken, R. J., & McFall, R. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02092.x · Cited by 93 · see citing works
Temporal Selection is Suppressed, Delayed, and Diffused During the Attentional Blink
Vul, E., Nieuwenstein, M., & Kanwisher, N. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02046.x · Cited by 89 · see citing works
Yap, M. J.-M., Balota, D. A., Tse, C., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.495 · Cited by 89 · see citing works
Could gambling save science? Encouraging an honest consensus
Hanson, R. (1995) · Social Epistemology · doi:10.1080/02691729508578768 · Cited by 82 · see citing works
The ultimate sampling dilemma in experience-based decision making.
Fiedler, K. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.186 · Cited by 80 · see citing works
Preschoolers' Perspective Taking in Word Learning
Nurmsoo, E., & Bloom, P. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02069.x · Cited by 80 · see citing works
Replication effort provokes praise—and ‘bullying’ charges
Bohannon, J. (2014) · Science · doi:10.1126/science.344.6186.788 · Cited by 72 · see citing works
Loving Those Who Justify Inequality
Lau, G. P., Kay, A. C., & Spencer, S. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02040.x · Cited by 68 · see citing works
A single-system account of the relationship between priming, recognition, and fluency.
Berry, C. J., Shanks, D. R., & Henson, R. N. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.97 · Cited by 67 · see citing works
Priming addition facts with semantic relations.
Bassok, M., Pedigo, S. F., & Oskarsson, A. T. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.343 · Cited by 65 · see citing works
The effects of an implemental mind-set on attitude strength.
Henderson, M. D., de Liver, Y., & Gollwitzer, P. M. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.396 · Cited by 59 · see citing works
Bressan, P., & Stranieri, D. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02060.x · Cited by 38 · see citing works
Intentional forgetting is easier after two "shots" than one.
Sahakyan, L., Delaney, P. F., & Waldum, E. R. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.2.408 · Cited by 36 · see citing works
Stereotypes and retrieval-provoked illusory source recollections.
Dodson, C. S., Darragh, J., & Williams, A. L. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.3.460 · Cited by 33 · see citing works
Contextual effects on reading aloud: Evidence for pathway control.
Reynolds, M., & Besner, D. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.50 · Cited by 25 · see citing works
An Experiment on Prediction Markets in Science
Almenberg, J., Kittlitz, K., & Pfeiffer, T. (2009) · PLoS ONE · doi:10.1371/journal.pone.0008500 · Cited by 24 · see citing works
Self-citationWolf, S. T., Insko, C. A., Kirchner, J. L., & Wildschut, T. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.479 · Cited by 19 · see citing works
Now you see it, now you don't: Repetition blindness for nonwords.
Morris, A. L., & Still, M. L. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.146 · Cited by 15 · see citing works
Eliciting Predictions and Recommendations for Decision Making
Chen, Y., Kash, I. A., Ruberry, M. E., & Shnayder, V. (2014) · ACM Transactions on Economics and Computation · doi:10.1145/2556271 · Cited by 15 · see citing works
Self-citationAccounting for occurrences: A new view of the use of contingency information in causal judgment.
White, P. A. (2008) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.34.1.204 · Cited by 13 · see citing works
Intentional Forgetting Is Easier After Two "Shots" Than One
Sahakyan, L., Delaney, P. F., & Waldum, E. R. (2007) · PsycEXTRA Dataset · doi:10.1037/e527342012-084 · Cited by 2 · see citing works
Janssen, N., Schirm, W., Mahon, B., & Caramazza, A. (2007) · PsycEXTRA Dataset · doi:10.1037/e512682013-476
Flagged first, then by citations
Research works printed but not linked: 3
- Maniadis Z, Tufano F, List JA (2014) One swallow doesn't make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104(1): 277-290.
- Hanson R (2006) Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds Patashnik EM, Gerber AS (Brookings Institution Press, Washington, DC), pp 151-173.
- Hanson R (2007) Logarithmic market scoring rules for modular combinatorial in- formation aggregation. J Pred Markets 1(1):3-15.
Cited by
Sorted by citations
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772 · see citing works
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Self-citationVariability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097 · see citing works
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Self-citationBayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 732 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398 · see citing works
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. G. (2017) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 378 · see citing works
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307 · see citing works
How open is innovation? A retrospective and ideas forward
Dahlander, L., Gann, D., & Wallin, M. W. (2021) · Research Policy · doi:10.1016/j.respol.2021.104218 · Cited by 265 · see citing works
Three Recommendations for Improving the Use of p -Values
Benjamin, D. J., & Berger, J. O. (2019) · The American Statistician · doi:10.1080/00031305.2018.1543135 · Cited by 258 · see citing works
Redefine statistical significancePreprint
Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E., Berk, R. A., Bollen, K. A., Brembs, B., Brown, L., Camerer, C. F., et al. (2017) · doi:10.31234/osf.io/mky9j · Cited by 247 · see citing works
Self-citationNon est Disputandum de Generalizability? A Glimpse into The External Validity Trial
List, J. A. (2020) · National Bureau of Economic Research · doi:10.3386/w27535 · Cited by 197 · see citing works
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189 · see citing works
Self-citationPredicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2018) · Journal of Political Economy · doi:10.1086/699976 · Cited by 183 · see citing works
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 182 · see citing works
Self-citationNonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177 · see citing works
A Review on Blockchain Technology and Blockchain Projects Fostering Open Science
Leible, S., Steffen Schlager, C., Schubotz, M., & Gipp, B. (2019) · Frontiers in Blockchain · doi:10.3389/fbloc.2019.00016 · Cited by 176 · see citing works
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · see citing works
Self-citationPredict science to improve science
DellaVigna, S., Pope, D. G., & Vivalt, E. (2019) · Science · doi:10.1126/science.aaz1704 · Cited by 140 · see citing works
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137 · see citing works
#EEGManyLabs: Investigating the replicability of influential EEG experiments
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2021) · Cortex · doi:10.1016/j.cortex.2021.03.013 · Cited by 136 · see citing works
Self-citation(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses
Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.12.007 · Cited by 126 · see citing works
Predicting Experimental Results: Who Knows What?
DellaVigna, S., & Pope, D. G. (2016) · National Bureau of Economic Research · doi:10.3386/w22566 · Cited by 117 · see citing works
Achievement Gaps in the Wake of COVID-19
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021) · Educational Researcher · doi:10.3102/0013189x211011237 · Cited by 134 · see citing works
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105 · see citing works
Self-citationReproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104 · see citing works
Estimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 99 · see citing works
Mulder, J., & Wagenmakers, E. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.01.002 · Cited by 99 · see citing works
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98 · see citing works
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y.-L., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 97 · see citing works
Self-citationThe Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · Southern Economic Journal · doi:10.1002/soej.12392 · Cited by 97 · see citing works
Predicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225826 · Cited by 89 · see citing works
Self-citationA discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 87 · see citing works
Replications in Comparative Cognition: What Should We Expect and How Can We Improve?
Farrar, B. G., Boeckle, M., & Clayton, N. S. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.01.02.2020 · Cited by 86 · see citing works
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 86 · see citing works
Many Labs 2: Investigating Variation in Replicability Across Sample and SettingPreprint
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M. E., Axt, J., Babalola, M. T., Bahník, Š., et al. (2018) · doi:10.31234/osf.io/9654g · Cited by 81 · see citing works
Low replicability can support robust and efficient science
Lewandowsky, S., & Oberauer, K. (2020) · Nature Communications · doi:10.1038/s41467-019-14203-0 · Cited by 80 · see citing works
Systematizing Confidence in Open Research and Evidence (SCORE)Preprint
Alipourfard, N., Arendt, B., Benjamin, D. M., Benkler, N., Bishop, M. M., Burstein, M. H., Bush, M., Caverlee, J., Chen, Y., Clark, C., et al. (2021) · doi:10.31235/osf.io/46mnb · Cited by 69 · see citing works
Self-citationOptimally generate policy-based evidence before scaling
List, J. A. (2024) · Nature · doi:10.1038/s41586-023-06972-y · Cited by 69 · see citing works
Perceived crisis and reforms: Issues, explanations, and remedies.
de Boeck, P. A. L., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 67 · see citing works
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 64 · see citing works
Creative destruction in science
Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., et al. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.07.002 · Cited by 64 · see citing works
Self-citationSchaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87 · see citing works
Self-citationLaypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920919667 · Cited by 65 · see citing works
How Effective Is (More) Money? Randomizing Unconditional Cash Transfer Amounts in the USPreprint
Jaroszewicz, A., Jachimowicz, J., Hauser, O., & Jamison, J. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4154000 · Cited by 63 · see citing works
Beyond Statistical Ritual: Theory in Psychological Science
Proulx, T., & Morey, R. D. (2021) · Perspectives on Psychological Science · doi:10.1177/17456916211017098 · Cited by 58 · see citing works
Transparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 57 · see citing works
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 56 · see citing works
Examining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 56 · see citing works
Self-citationAmano, T., Bako, L. A., Best, M., Boenisch, N., Boersch‐Supan, P. H., Browne, D., Buckley, Y. M., Burgman, M., Cadotte, M. W., Canessa, S., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321 · Cited by 53 · see citing works
Eitan, O., Viganola, D., Inbar, Y., Dreber, A., Johannesson, M., Pfeiffer, T., Thau, S., & Uhlmann, E. L. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.06.004 · Cited by 51 · see citing works
Self-citationA creative destruction approach to replication: Implicit work and sex morality across cultures
Tierney, W., Hardy, J. H., Ebersole, C. R., Viganola, D., Clemente, E. G., Gordon, M., Hoogeveen, S., Haaf, J. M., Dreber, A., Johannesson, M., et al. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104060 · Cited by 50 · see citing works
Self-citationChopra, F., Haaland, I., Roth, C., & Stegmann, A. (2023) · The Economic Journal · doi:10.1093/ej/uead060 · Cited by 48 · see citing works
How Much Information Is Incorporated into Financial Asset Prices? Experimental Evidence
Page, L., & Siemroth, C. (2018) · Review of Financial Studies · doi:10.1093/rfs/hhaa143 · Cited by 46 · see citing works
Credibility crisis in agricultural economics
Ferraro, P. J., & Shukla, P. (2022) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13323 · Cited by 44 · see citing works
Consequences of relying on statistical significance: Some illustrations
CALSTER, B. V., Steyerberg, E. W., Collins, P. G. S., & Smits, T. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.12912 · Cited by 44 · see citing works
Gordon, M., Viganola, D., Bishop, M., Chen, Y., Dreber, A., Goldfedder, B., Holzmeister, F., Johannesson, M., Liu, Y., Twardy, C. R., et al. (2020) · Royal Society Open Science · doi:10.1098/rsos.200566 · Cited by 44 · see citing works
Self-citationGordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44 · see citing works
Self-citationQuestionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 42 · see citing works
fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
Yücel, M. A., Luke, R., Mesquita, R. C., von Lühmann, A., Mehler, D. M. A., Lührs, M., Gemignani, J., Abdalmalak, A., Albrecht, F., Ivo, I. D. A., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08412-1 · Cited by 41 · see citing works
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 38 · see citing works
Self-citationA Replication Study of Operations Management Experiments in Management Science
Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S. M., Leider, S., Long, X., & Tong, J. D. (2022) · Management Science · doi:10.1287/mnsc.2023.4866 · Cited by 38 · see citing works
How to increase public support for carbon pricing with revenue recycling
Woerner, A., Imai, T., Pace, D. D., & Schmidt, K. M. (2024) · Nature Sustainability · doi:10.1038/s41893-024-01466-9 · Cited by 38 · see citing works
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 37 · see citing works
The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 35 · see citing works
The Practicalities of Running Randomized Evaluations
Glennerster, R. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.002 · Cited by 35 · see citing works
Multiple testing with covariate adjustment in experimental economics
List, J. A., Shaikh, A. M., & Vayalinkal, A. (2023) · Journal of Applied Econometrics · doi:10.1002/jae.2985 · Cited by 33 · see citing works
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32 · see citing works
Blockchain for Science and Knowledge Creation
Bartling, S. H. (2018) · Springer eBooks · doi:10.1007/978-3-662-57611-3_10 · Cited by 32 · see citing works
The Heuristic Value of p in Inductive Statistical Inference
Krueger, J. I., & Heck, P. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00908 · Cited by 32 · see citing works
The replication crisis in epidemiology: snowball, snow job, or winter solstice?
Lash, T., Collin, L., & Dyke, M. E. V. (2018) · Current Epidemiology Reports · doi:10.1007/s40471-018-0148-x · Cited by 24 · see citing works
Probabilistic forecasting of replication studies
Pawel, S., & Held, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231416 · Cited by 32 · see citing works
Predicting and reasoning about replicability using structured groups
Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H. S., Hemming, V., Thorn, F. S., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221553 · Cited by 31 · see citing works
Lohmann, P. M., Pizzo, A., Bauer, J. M., Khanna, T. M., Flecke, S. L., Callaghan, M., Minx, J. C., & Reisch, L. A. (2026) · Nature Food · doi:10.1038/s43016-025-01279-9 · Cited by 30 · see citing works
P in the right place: Revisiting the evidential value of P‐values
Lytsy, P. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12319 · Cited by 30 · see citing works
Fraser, H. S., Bush, M., Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Gould, E., Hemming, V., Hamilton, D. G., Rumpff, L., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0274429 · Cited by 30 · see citing works
Bayesian markets to elicit private information
Baillon, A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1703486114 · Cited by 30 · see citing works
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 30 · see citing works
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28 · see citing works
Wilkening, T. S., Martinie, M. F. R. C., & Howe, P. D. L. (2021) · Management Science · doi:10.1287/mnsc.2020.3919 · Cited by 27 · see citing works
Managing and defending the commons: Experimental evidence from TURFs in Chile
Chávez, C. A., Murphy, J. J., & Stranlund, J. K. (2018) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2018.07.004 · Cited by 27 · see citing works
Who Should Do Replication Labor?
Romero, F. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918803619 · Cited by 27 · see citing works
Dai, H., Young, H. P., Durant, T. J. S., Gong, G., Kang, M., Krumholz, H. M., Schulz, W. L., & Jiang, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.03662 · Cited by 27 · see citing works
Can large language models help predict results from a complex behavioural science study?
Lippert, S., Dreber, A., Johannesson, M., Tierney, W., Cyrus-Lai, W., Uhlmann, E. L., Emotion Expression Collaboration, & Pfeiffer, T. (2024) · Royal Society Open Science · doi:10.1098/rsos.240682 · Cited by 26 · see citing works
Self-citationReproducibility in Computing Research: An Empirical Study
Raghupathi, W., Raghupathi, V., & Ren, J. (2022) · IEEE Access · doi:10.1109/access.2022.3158675 · Cited by 26 · see citing works
Large language models can predict the results of social science experiments
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026) · Nature · doi:10.1038/s41586-026-10742-x · Cited by 25 · see citing works
von Hippel, P. T. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211072525 · Cited by 25 · see citing works
Olofsson, J., Larsson, M. U., Roa, C., Wilson, D. A., & Laukka, E. J. (2019) · Behavior Genetics · doi:10.1007/s10519-019-09980-9 · Cited by 25 · see citing works
#EEGManyLabs: Investigating the Replicability of Influential EEG ExperimentsPreprint
Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C., Beste, C., Bland, A. R., Bradford, D. E., Bublatzky, F., Busch, N. A., et al. (2020) · Cortex; a journal devoted to the study of the nervous system and behavior · doi:10.31234/osf.io/528nr · Cited by 25 · see citing works
Self-citationPreregistration and reproducibility
Strømland, E. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2019.01.006 · Cited by 23 · see citing works
Predicting the Replicability of Social Science Lab ExperimentsPreprint
Altmejd, A., Dreber, A., Forsell, E., Ho, T., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · doi:10.31222/osf.io/zamry · Cited by 23 · see citing works
Self-citationA Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 23 · see citing works
Self-citationUsing the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22 · see citing works
Ingre, M., & Nilsonne, G. (2018) · Royal Society Open Science · doi:10.1098/rsos.181190 · Cited by 13 · see citing works
- Authors
- Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M.
- Affiliations
- Stockholm School of Economics (SE)
- Institute for Advanced Study (DE)
- Swedish National Bank (SE)
- Consensus Point, Nashville, TN 37203;
- Harvard University (US)
- Center for Open Science (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-11-09
- Volume
- 112
- Issue
- 50
- Pages
- 15343-15347
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4687569
- OpenAlex
- W2145409614
- Cited by
- 331
- References
- 78
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 22 · 2023: 31 · 2022: 53 · 2021: 41 · 2020: 55 · 2019: 34 · 2018: 26 · 2017: 17 · 2016: 14 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- prediction markets, reproducibility, replication, Reproducibility Project: Psychology, statistical significance, hypothesis testing, replication study
- MeSH terms
- Commerce, Forecasting, Probability, Surveys and Questionnaires, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
