Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 401–500 of 1,372
A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It
Reed, W. R. (2018) · Australian Economic Review · doi:10.1111/1467-8462.12262 · Cited by 18
Identification of and Correction for Publication Bias
Andrews, I., & Kasy, M. (2017) · National Bureau of Economic Research · doi:10.3386/w23298 · Cited by 33
Editorial: The Value and Status of Replications in Animal Behavior and Cognition Research
Beran, M. J. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.01.01.2020 · Cited by 19
“…know, concerns about replicability in psychology (and other disciplines) have increased, and some have argued are even to the point of creating a “crisis” for some fields (Baker, 2016; Camerer et al., 2016, 2018; Open Science Collaboration, 2015; Shrout & Rodgers, 2018; Stroebe & Strack, 2014).”
Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102327 · Cited by 24
Open Science: A Review of Its Effectiveness and Implications for Service Research
Vaerenbergh, Y. V., Hazée, S., & Zwienenberg, T. J. (2025) · Journal of Service Research · doi:10.1177/10946705251338461 · Cited by 24
“Similar studies were also conducted in economics (Camerer et al. 2016), operations management (Davis et al. 2023), and entrepreneurship (Crawford et al. 2022), among others.”
Efficient empiricism: Streamlining teaching, research, and learning in empirical courses
Dvořák, T., Halliday, S. D., O’Hara, M. E., & Swoboda, A. M. (2019) · The Journal of Economic Education · doi:10.1080/00220485.2019.1618765 · Cited by 17
Friedman, W., Keats, A., & Mutua, M. K. (2022) · Journal of Health Economics · doi:10.1016/j.jhealeco.2022.102694 · Cited by 19
“Camerer et al. (2016) and Camerer et al. (2018) show that studies that are less than fully powered – such as ours – tend to overstate the magnitude of treatment effects when they are there.”
BackgroundP-Hacking, Data Type and Data-Sharing Policy
Brodeur, A., Cook, N., & Neisser, C. (2022) · The Economic Journal · doi:10.1093/ej/uead104 · Cited by 41
Evidence of direct and indirect reciprocity in network-structured economic games
Redhead, D., Gervais, M. M., Kajokaite, K., Koster, J., Manyoma, A. H., Manyoma, D. H., McElreath, R. L., & Ross, C. T. (2024) · Communications Psychology · doi:10.1038/s44271-024-00098-1 · Cited by 22
“Due to increasing concerns about replicability in the social sciences 68,69 , we repeated our study in four Colombian communities: a coastal community ( n = 186 egos and 220 alters), a lowland community ( n = 154 egos and 178 alters), a highland community ( n = 45 egos and 53 alters), and an…”
BackgroundOn the significance of statistically insignificant results in consumer behavior experiments
Peterson, R. A., & Umesh, U. N. (2017) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-017-0528-7 · Cited by 22
Above a swamp: A theory of high-quality scientific production
Kiri, B., Lacetera, N., & Zirulia, L. (2018) · Research Policy · doi:10.1016/j.respol.2018.01.011 · Cited by 20
Statistical methods for replicability assessment
Hung, K. E., & Fithian, W. (2020) · The Annals of Applied Statistics · doi:10.1214/20-aoas1336 · Cited by 26
“For example, in economics, Camerer et al. (2016) used the same flawed metric of proportion of statistically significant results in the original direction.”
BackgroundFalse-positive pathology: improving reproducibility with the next generation of pathologists
Mazer, B. L., Homer, R., & Rimm, D. L. (2019) · Laboratory Investigation · doi:10.1038/s41374-019-0257-2 · Cited by 18
Olsen, J., Mosen, J., Voracek, M., & Kirchler, E. M. (2019) · Royal Society Open Science · doi:10.1098/rsos.190738 · Cited by 20
Louderback, E. R., Wohl, M. J. A., & LaPlante, D. A. (2020) · Addiction Research & Theory · doi:10.1080/16066359.2020.1767774 · Cited by 20
Re-Analyzing Phase III Bremelanotide Trials for “Hypoactive Sexual Desire Disorder” in Women
Spielmans, G. I. (2021) · The Journal of Sex Research · doi:10.1080/00224499.2021.1885601 · Cited by 17
“…in other fields, including psychiatric genetics (Border et al., 2019), psychiatric gene x environment interaction research (Duncan & Keller, 2011), structural brain-behavior associations (Masouleh et al., 2019), cognitive neuroscience (Szucs et al., 2017), and economics (Camerer et al., 2016).”
BackgroundTeaching for Large-Scale Reproducibility Verification
Vilhuber, L., Son, H. H., Welch, M., Wasser, D., & Darisse, M. (2022) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2022.2074582 · Cited by 20
“In several studies (Camerer et al., 2016; Chang & Li, 2017; Höffler, 2017a), at least half of the replication packages associated with surveyed manuscripts failed to (fully) reproduce the results in the manuscript when re-run.”
Measuring the effects of publication bias in political science
Esarey, J., & Wu, A. (2016) · Research & Politics · doi:10.1177/2053168016665856 · Cited by 21
“9% of the size of the original estimate, a reduction of about 44% (Camerer et al., 2016, p. 1434).(13) Most of our estimates, however, show smaller mean bias; this suggests that either (i) the proportion of null results in the population of studies is higher than we contemplated, or (ii) other factors (opportunistic m…”
BackgroundResultInfluentialBiased judgements of fairness in bargaining: A replication in the laboratory
Hippel, S., & Hoeppner, S. (2019) · International Review of Law and Economics · doi:10.1016/j.irle.2019.02.001 · Cited by 19
Mueller‐Langer, F., & Andreoli-Versbach, P. (2014) · Information Economics and Policy · doi:10.1016/j.infoecopol.2017.05.004 · Cited by 25
Goroff, D. L., Lewis, N. A., Scheel, A. M., Scherer, L. D., & Tucker, J. A. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/j8b9a · Cited by 16
Introduction to the special issue on reproducibility in neuroimaging
Poldrack, R. A., Whitaker, K. J., & Kennedy, D. N. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116357 · Cited by 17
“The last decade has seen increasing attention to the problem of scientific reproducibility, across a broad range of scientific fields (Camerer et al., 2016; Morrison, 2014; Open Science Collaboration, 2015).”
BackgroundThe replication crisis has led to positive structural, procedural, and community changesPreprint
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M. M., Breznau, N., Robertson, O. M., Kalandadze, T., et al. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/r6cvx · Cited by 23
Psychologists update their beliefs about effect sizes after replication studies
McDiarmid, A. D., Tullett, A. M., Whitt, C. M., Vazire, S., Smaldino, P. E., & Stephens, J. E. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01220-7 · Cited by 16
Economics, Facsimile Science and Societal Harm
Muller, S. (2021) · Debating higher education: philosophical perspectives · doi:10.1007/978-3-030-84447-9_11 · Cited by 16
Understanding Economic Behavior Using Open-Ended Survey Data
Haaland, I., Roth, C., Stantcheva, S., & Wohlfart, J. (2025) · Journal of Economic Literature · doi:10.1257/jel.20251780 · Cited by 73
Reproducibility in Subsurface Geoscience
Steventon, M., Jackson, C. A.-L., Ireland, M., Hall, M., Munafo, M. R., & Roberts, K. (2022) · Earth Science Systems and Society · doi:10.3389/esss.2022.10051 · Cited by 18
“Concern over the reproducibility of scientific results has gained significant traction in recent years as large-scale reviews in disciplines such as medicine (Nosek and Errington, 2017), psychology (Open Science Collaboration, 2015), and economics (Camerer et al., 2016), have often cast doubt on the reliability of pub…”
BackgroundStretch Goals, Factual/Counterfactual Reflection Strategies, and Firm Performance
Yang, M. M.-H. (2023) · Journal of Management Studies · doi:10.1111/joms.12931 · Cited by 17
Assessing the replication landscape in experimental linguistics
Kobrock, K., & Roettger, T. B. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011135 · Cited by 26
Grusky, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.107 · Cited by 50
Care to share? Experimental evidence on code sharing behavior in the social sciences
Krähmer, D., Schächtele, L., & Schneck, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0289380 · Cited by 20
Assessing the Evidential Value of Mental Fatigue and Exercise Research
Holgado, D., Mesquida, C., & Román-Caballero, R. (2023) · Sports Medicine · doi:10.1007/s40279-023-01926-w · Cited by 33
“Though replication efforts are still very scarce in the field [107], data from similar disciplines such as psychology show that only half of the original studies were replicated [106, 108] and this would be in agreement with the expected repli-cability rate of .54 (see Fig.”
BackgroundPartners or Strangers? Cooperation, Monetary Trade, and the Choice of Scale of Interaction
Bigoni, M., Camera, G., & Casari, M. (2019) · American Economic Journal Microeconomics · doi:10.1257/mic.20170280 · Cited by 20
Biases in Fiscal Multiplier Estimates
Asatryan, Z., Havlik, A., Heinemann, F., & Nover, J. (2019) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2020.101861 · Cited by 22
“Concerns are not necessarily reduced by the tendency towards more credible identification: Young (2017) claims that one third to one half of instrumental variable point estimates are wrongly declared statistically significant and replications of lab experiments are frequently unable to reproduce original findings (Cam…”
Is preclinical research in cancer biology reproducible enough?
Kane, P. B., & Kimmelman, J. (2021) · eLife · doi:10.7554/elife.67527 · Cited by 22
Heterogeneity of Beliefs and Trading Behavior — A Reexamination
Füllbrunn, S., Huber, C., Eckel, C. C., & Weitzel, U. (2022) · Journal of Financial and Quantitative Analysis · doi:10.1017/s002210902300011x · Cited by 19
“Finally, by replicating the findings of Carlé et al. (2019), we contribute to recent discussions on reproducibility and replicability in the social sciences in general, and in experimental economics and finance in particular (e.g., Camerer et al. (2016), Camerer, Dreber, and Johannesson (2019)).”
BackgroundResultThe Meta-Science of Adult Statistical Word Segmentation: Part 1
Hartshorne, J. K., Skorb, L., Dietz, S. L., Garcia, C., Iozzo, G., Lamirato, K., Ledoux, J. R., Mu, J., Murdock, K., Ravid, J., et al. (2019) · Collabra Psychology · doi:10.1525/collabra.181 · Cited by 25
“…interest in informing these debates through empirical studies: that is, meta-science (Aarts & LeBel, 2016; Bakker et al., 2016; Button et al., 2013; Camerer et al., 2016; Cova et al., 2018; Ebersole et al., 2016; Fraley & Vazire, 2014; Frank et al., 2017; Klein et al., 2014; Mahowald et al., 2017;…”
BackgroundInsights into Criteria for Statistical Significance from Signal Detection Analysis
Witt, J. K. (2019) · Meta-Psychology · doi:10.15626/mp.2018.871 · Cited by 18
Transparency and openness in behavioral medicine research
McVay, M. A., & Conroy, D. E. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz154 · Cited by 11
Crane, H. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3074083 · Cited by 16
“I do not analyze the results in [3], which performs a similar study for findings in experimental economics but for a much smaller sample of 18 studies.”
InfluentialCoville, A., & Vivalt, E. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5nsh3 · Cited by 17
LIONESS Lab – A Free Web-Based Platform for Conducting Interactive Experiments OnlinePreprint
Giamattei, M., Molleman, L., Yahosseini, K. S., et al. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3329384 · Cited by 16
Kurtulmuş, F. (2020) · Journal of Applied Philosophy · doi:10.1111/japp.12451 · Cited by 24
A toolbox to evaluate the trustworthiness of published findings
Adler, S. J., Röseler, L., & Schöniger, M. K. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114189 · Cited by 21
Kedron, P., Holler, J., & Bardin, S. (2023) · Annals of the American Association of Geographers · doi:10.1080/24694452.2023.2276115 · Cited by 15
“…of reproduction studies are likely to be limited in scope because existing reproductions typically focus on re-creating the results of a small number of studies selected based on topical interest or researcher familiarity (Open Science Collaboration 2015; Camerer et al. 2016; Camerer et al. 2018).”
BackgroundSelective and (mis)leading economics journals: Meta‐research evidence
Askarov, Z., Doucouliagos, A., Doucouliagos, H., & Stanley, T. D. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12598 · Cited by 24
Examining the replicability of online experiments selected by a decision market
Holzmeister, F., Johannesson, M., Camerer, C. F., Chen, Y., Ho, T., Hoogeveen, S., Huber, J., Imai, N., Imai, T., Jin, L. J., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02062-9 · Cited by 26
“The replication rate of 54% based on the statistical significance indicator is also similar to previous replication studies, with 36% in the RPP 15 , 61% in the EERP 14 and 62% in the SSRP 17 .”
InfluentialSelf-citationDiscounting and Digit Ratio: Low 2D:4D Predicts Patience for a Sample of Females
Aycinena, D., & Rentschler, L. (2018) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2017.00257 · Cited by 21
“Plan of Analysis Given the so called “replicability crisis” in scientific findings (see e.g., Ioannidis, 2005; Button et al., 2013; Aarts et al., 2015; Camerer et al., 2016), we attempted to limit the degrees of freedom available to us as researchers18.”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results
Coker, B., Rudin, C. D., & King, G. (2021) · Management Science · doi:10.1287/mnsc.2020.3818 · Cited by 42
“The Cornfield Conditions assumed that the causal effect was zero and deduced properties of the unmeasured confounding genetic variable, properties that were deemed biologically infeasible (Cornfield et al. 1959).”
BackgroundLeipold, S., Oderbolz, C., Greber, M., & Jäncke, L. (2019) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2018.12.016 · Cited by 13
Confounds in “Failed” Replications
Bressan, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01884 · Cited by 18
How Do We Choose Our Giants? Perceptions of Replicability in Psychological Science
Alister, M., Vickers-Jones, R., Sewell, D. K., & Ballard, T. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211018199 · Cited by 16
“Conversely, Camerer et al. (2016) ran a prediction market for replications of 18 experimental economics studies and did not find a significant relationship between the market and the actual replication rate.”
BackgroundBehavioural Land Use Policy Studies: Past, Present, and Future
Bao, H. X. H., & Robinson, G. M. (2021) · Land Use Policy · doi:10.1016/j.landusepol.2022.106013 · Cited by 14
Osborne, C., & Norris, E. (2022) · Cogent Psychology · doi:10.1080/23311908.2022.2066304 · Cited by 15
“Within psychology and across disciplines, researchers have produced an abundant number of nonreplicable research findings (e.g., Baker, 2016; Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundPower Calculations for Replication Studies
Micheloud, C., & Held, L. (2022) · Statistical Science · doi:10.1214/21-sts828 · Cited by 20
“M E ] 2 1 Collaboration, 2015), social sciences (Camerer et al., 2018) and economics (Camerer et al., 2016) among others.”
BackgroundHow failure to falsify in high-volume science contributes to the replication crisis
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2022) · eLife · doi:10.7554/elife.78830 · Cited by 25
“Lines of research have also emerged using creative assembly of experts (e.g., prediction markets; Dreber et al., 2015; Camerer et al., 2016; Camerer et al., 2018; Gordon et al., 2020 and AI- driven approaches Altmejd et al., 2019; Pawel and Held, 2020; Yang et al., 2020) to estimate confidence…”
MethodAmar, D., Ron Shamir, A., & Yekutieli, D. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005700 · Cited by 13
“ion, and therefore the reliability, of reported results. This is of vital importance as in recent years concerns have been raised in several domains regarding low reproducibility, including economics [3], psychology [4], medicine [5], and biological studies that rely on high throughput experiments such as gene express…”
BackgroundTheory and the Replication Problem
Willer, D., & Emanuelson, P. E. (2020) · Sociological Methodology · doi:10.1177/0081175020955216 · Cited by 17
Why Scientists Chase Big Problems: Individual Strategy and Social OptimalityPreprint
Bergstrom, C. T., Foster, J. G., & Song, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05822 · Cited by 21
“Federal research budgets are stagnant or contracting, while the very public replication crisis (Open Science Collaboration, 2015; Camerer et al., 2016) has shaken both lay and professional confidence in the basic institutions of science.”
BackgroundYang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T., Lagisz, M., & Nakagawa, S. (2022) · doi:10.32942/osf.io/97nv6 · Cited by 14
One finding is no finding: Toward a replication culture in family business research
Brinkerink, J., Massis, A. D., & Kellermanns, F. W. (2022) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2022.100521 · Cited by 18
Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines
Rahal, R.-M., Hamann, H., Brohmer, H., & Pethig, F. (2022) · Research Ideas and Outcomes · doi:10.3897/rio.8.e89980 · Cited by 19
Evaluating Research Transparency and Openness in Communication Sciences and Disorders Journals
Schroeder, S. R., Gaeta, L., Amin, M. E., Chow, J. C., & Borders, J. C. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00330 · Cited by 17
Panacea or Dangerous Practice: A Counterpoint to Hanisch's Argument for Prescriptive Theorizing
Horner, S., Cornelissen, J., & Zundel, M. (2024) · Journal of Management Studies · doi:10.1111/joms.13039 · Cited by 18
Most recommended medical interventions reach P < 0.005 for their primary outcomes in meta-analyses
Koletsi, D., Solmi, M., Pandis, N., Fleming, P. S., Correll, C. U., & Ioannidis, J. P. A. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz241 · Cited by 15
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 16
Last word not yet spoken: a reinvestigation of last place aversion with aversion to rank reversals
Martinangeli, A., & Windsteiger, L. (2020) · Experimental Economics · doi:10.1007/s10683-020-09682-8 · Cited by 20
Reconciling the Paradoxical Findings of Choice Overload through an Analytical Lens
Zhang, N., & Xu, H. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16954 · Cited by 19
The Status and Value of Replications in Animal Behavior Science
Brecht, K. F., Legg, E. W., Nawroth, C., Fraser, H. S., & Ostojić, L. (2021) · Animal Behavior and Cognition · doi:10.26451/abc.08.02.01.2021 · Cited by 15
“…sciences and related fields such as neuroscience (Button et al., 2013; Klapwijk et al., 2021; Mulugeta et al., 2018), economics (Berry et al., 2017; Camerer et al., 2016; Mueller-Langer et al., 2019; Schultze et al., 2019), and ecology (Fidler et al., 2017; Fraser et al., 2020; Kelly, 2019;…”
BackgroundHenderson, A., Thoelen, G., Nadler, A., Barraza, J. A., & Nave, G. (2018) · Scientific Reports · doi:10.1038/s41598-018-29928-z · Cited by 13
Self-citationOnline Place Branding: Is Geography ‘Destiny’ in a ‘Space of Flows’ World?
Briciu, V.-A., Rezeanu, C.-I., & Briciu, A. (2020) · Sustainability · doi:10.3390/su12104073 · Cited by 18
“In addition, as scholars have started to worry about the replicability crisis and the biases in the application of established methods and tools in science in general [36] and social sciences in particular [37], we acknowledge there is a need for more studies that accurately apply established methods to a variety of s…”
BackgroundNockur, L., & Pfattheicher, S. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550620961262 · Cited by 16
“Indeed, amid recent concerns about reproducibility in psychological and behavioral science (Camerer et al., 2016, 2018; Open Science Collaboration, 2015), independent replications should be of particular interest to the field (Brandt et al., 2014; Shrout & Rodgers, 2018).”
BackgroundJournal-based replication of experiments: An application to “Being Chosen to Lead”
Drazen, A., Dreber, A., Ozbay, E. Y., & Snowberg, E. (2021) · Journal of Public Economics · doi:10.1016/j.jpubeco.2021.104482 · Cited by 20
Self-citationCall for Papers-Special Issue on Experiments in Organizational Theory
(2019) · Organization Science · doi:10.1287/orsc.2018.1257 · Cited by 13
“This may be one reason why experimental results are at least as robust as those of othermethods,which amassive replication effort found (Camerer et al. 2016).”
BackgroundQuestionable Research Practices and Open Science in Quantitative CriminologyPreprint
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/bwm7s · Cited by 12
Heyard, R., Pawel, S., Frese, J., Voelkl, B., Würbel, H., McCann, S., Held, L., Wever, K. E., Hartmann, H., Townsin, L., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.242076 · Cited by 20
Reproducibility and robustness of economics and political science research
Brodeur, A., Mikola, D., Cook, N., Fiala, L., Brailey, T., Briggs, R. C., de Gendre, A., Dupraz, Y., Gabani, J., Gauriot, R., et al. (2026) · Nature · doi:10.1038/s41586-026-10251-x · Cited by 34
“This result is in stark contrast to previous projects focused on replication with new data in psychology or social science experiments 24–26 , which uncovered replication rates ranging from 50% to 66%.”
Self-citationLangley, P. C. (2016) · Current Medical Research and Opinion · doi:10.1080/03007995.2016.1190699 · Cited by 13
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.433 · Cited by 12
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i3.446 · Cited by 10
Michels, M., Rieling, J. A., & Mußhoff, O. (2023) · Q Open · doi:10.1093/qopen/qoae007 · Cited by 20
Selective reporting of placebo tests in top economics journals
Dreber, A., Johannesson, M., & Yang, Y. (2024) · Economic Inquiry · doi:10.1111/ecin.13217 · Cited by 23
Self-citationLangley, P. C., & Rhee, T. G. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i2.441 · Cited by 12
Computational chemistry experiments performed directly on a blockchain virtual computer
Hanson‐Heine, M. W. D., & Ashmore, A. P. (2020) · Chemical Science · doi:10.1039/d0sc01523g · Cited by 14
Are Results in Geomorphology Reproducible?
Church, M. A., Dudill, A., Venditti, J. G., & Frey, P. (2020) · Journal of Geophysical Research Earth Surface · doi:10.1029/2020jf005553 · Cited by 12
“Concern has subsequently spread to psychology (Open Science Collaboration, 2015), economics (Camerer et al., 2016), and to ecology (Powers & Hampton, 2019), in which the inherently stochastic nature of most field results is seen to present a severe challenge to reproducibility (Melbourne &…”
BackgroundInformation (non)aggregation in markets with costly signal acquisition
Corgnet, B., DECK, C. A., DeSantis, M., & Porter, D. (2018) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2018.08.005 · Cited by 13
Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol
Pearson, R., Fraser, H. S., Bush, M., Mody, F., Widjaja, I., Head, A., Wilkinson, D. P., Sinnott, R., Wintle, B. C., Burgman, M. A., et al. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.055 · Cited by 8
“mil/program/ systematizing-confidence-in-open-research-and-evidence predict research replicability [3, 6, 18].”
BackgroundViganola, D., Buckles, G., Chen, Y., Diego‐Rosell, P., Johannesson, M., Nosek, B. A., Pfeiffer, T., Siegel, A. C., & Dreber, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.181308 · Cited by 12
Self-citationPredicting and reasoning about replicability using structured groups
Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H. S., Hemming, V., Thorn, F. S., et al. (2023) · Royal Society Open Science · doi:10.1098/rsos.221553 · Cited by 33
Market interaction and efficient cooperation
Brandts, J., & Riedl, A. M. (2019) · European Economic Review · doi:10.1016/j.euroecorev.2019.103318 · Cited by 17
A hard-to-read font reduces the framing effect in a large sample
Korn, C. W., Ries, J., Schalk, L., Oganian, Y., & Saalbach, H. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1395-4 · Cited by 14
“The need for replications and for transparent, confirmatory hypothesis testing has come into the spotlight (Camerer et al., 2016; Open Science Collaboration, 2015).”
BackgroundRational expectations in an experimental asset market with shocks to market trends
Marquardt, P., NOUSSAIR, C. N., & Weber, M. (2019) · European Economic Review · doi:10.1016/j.euroecorev.2019.01.009 · Cited by 14
The evolution of distorted beliefs vs. mistaken choices under asymmetric error costs
Efferson, C. M., McKay, R., & Fehr, E. (2020) · Evolutionary Human Sciences · doi:10.1017/ehs.2020.25 · Cited by 16
“We did not pre-register the study because we collected the data before recent replication studies (Open-Science-Collaboration, 2015; Camerer et al., 2016) and the trend towards pre-registration that followed.”
Serra-García, M., Hansen, K. T., & Gneezy, U. (2020) · Psychological Science · doi:10.1177/0956797620923587 · Cited by 16
“(2010) study in a physics class at the University of California San Diego because replication is an important part of social science (Camerer et al., 2016; Open Science Collaboration, 2012, 2015; Simmons, Nelson, & Simonsohn, 2011; Simonsohn, 2016), and we report the suggestive results of our study in the Supplemental…”
BackgroundAlós‐Ferrer, C., & Yechiam, E. (2020) · Journal of Economic Psychology · doi:10.1016/j.joep.2020.102309 · Cited by 14
The evolving ethics of analysis, publication, and transparency in applied economics
Lybbert, T. J., & Buccola, S. T. (2020) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13131 · Cited by 15
Berger, J. L., Vogt, S., & Efferson, C. M. (2021) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2021.07.001 · Cited by 16
“We did not pre-register our design or hypotheses because we collected all data before the recent replication crisis and the trend toward pre-registration it provoked (Aarts et al., 2015; Camerer et al., 2016).”
BackgroundEvidenzbasierte | evidenzinformierte Gesundheitskommunikation
Erfurt, D. G. F. P. U. K. 2., Stehr, P., Heinemeier, D., Rossmann, C., & Verlagsgesellschaft, N. (2018) · Nomos Verlagsgesellschaft mbH & Co. KG eBooks · doi:10.5771/9783845291963 · Cited by 12
Are Reference Points Merely Lagged Beliefs Over Probabilities?Preprint
Heffetz, O. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3051829 · Cited by 13
Revisiting the replication crisis without false positivesPreprint
Bak-Coleman, J. B., Mann, R. P., Bergstrom, C. T., Gross, K., & West, J. D. (2022) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/rkyf7 · Cited by 12
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.