Evaluating replicability of laboratory experiments in economics
Open accessThe DOI matches its Crossref record; 83 of the 85 references it cites were found in a registry or an index, and 2 are not checked yet; 37 of the 55 sources its claims cite were checked for support.
Abstract
Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
In a systematic replication of 18 between-subject laboratory experiments published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, 11 (61.1%) showed a significant effect in the same direction as the original. Replicated effect sizes averaged 65.9% of original effect sizes, and peer beliefs plus original p-values/sample sizes carried some predictive information.
Key findings
- 11 of 18 replications (61.1%) had a significant effect in the same direction as the original, lower than the 92% mean power expectation if all original effects were true and accurately estimated (one-sample binomial test, P<0.001). Other indicators: 12/18 (66.7%) had the original effect size within the replication 95% CI, rising to 13/18 (72.2%) if including a study whose entire CI exceeded the original; 14/18 (77.8%) were significant in the same direction in meta-analysis; 15/18 (83.3%) fell within the 95% prediction interval, rising to 16/18 (88.9%) including one above the upper bound.
- Mean standardized effect size (r) was 0.279 in replications versus 0.474 in original studies (Wilcoxon signed-ranks test, z=-2.98, P=0.003, n=18); mean relative effect size was 65.9%.
- Prediction market mean belief was 75.2% and survey mean belief was 71.1%, both higher than the observed 61.1% replication rate but not significantly different; market beliefs correlated 0.30 (P=0.232) and survey beliefs 0.52 (P=0.028) with successful replication. Original p-value was negatively correlated and original sample size positively correlated with reproducibility indicators; only one of eight studies with original p<0.01 failed to replicate at the 5% level in the original direction.
- Compared with the Reproducibility Project: Psychology, all six reproducibility indicators were higher for experimental economics, significantly so for “Replicated with P<0.05 in original direction” (61.1% vs 36.1%, χ²=3.96, P=0.047), prediction market beliefs (75.1% vs 55.1%, P=0.001), and survey beliefs (71.1% vs 53.7%, P<0.001); average difference across indicators was 19 percentage points.
Methods
- Sample: all 18 between-subject laboratory experimental papers published in the *American Economic Review* and *Quarterly Journal of Economics* in 2011–2014, selecting the most important statistically significant finding emphasized by the original authors; exclusions included papers with prior replications, replications of previous studies, interaction-effect studies, and performance-selected treatments.
- Replications followed predefined analysis plans publicly posted before the replications and sent to original authors; sample sizes had at least 90% power (M=0.92, Mdn=0.91) to detect the original effect size at the 5% level; four replication teams conducted the studies, with random allocation to treatments where possible.
- Analysis used the same replication indicators as the Reproducibility Project: Psychology, including significant effect in same direction, original effect within replication 95% CI, meta-analytic estimate, 95% prediction interval, and relative effect size; prediction markets and surveys measured peer beliefs before replications, and Spearman correlations tested associations with original p-value and sample size.
Limitations
- The paper states the small sample of 18 replications limits statistical power and makes disciplinary comparisons premature; the prediction market correlation analysis had only about 15% power to detect a significant correlation due to n=18 and limited variation in beliefs.
- The paper states meta-analytic results assume original studies do not have publication or reporting biases; if publication bias exists, prospective power analyses will have overestimated replication power, and original effect sizes tend to be inflated.
- The paper notes standardized effect sizes are not fully comparable across the 18 studies because statistical tests use different aggregation levels; they therefore normalize original effect sizes to 1 and refrain from analyzing the correlation between original effect size and reproducibility indicators.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
23 Valid · 3 Not valid · 7 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the fraction of studies that 'Replicated with P<0.05 in original direction' differed between the studies (61.1% (11/18) versus 36.1% (35/97)) we used a chi-square test that was significant ( 2 =3.96; P=0.047).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The psychology-project figure attributed to this source is supported: Evaluating Reproducibility (p. 7) defines replication as p < .05 in the original direction and reports 35 positive replications (36.1%) among 97 originally significant effects. The 11/18 economics figure and chi-square comparison are presented as the citing authors’ own analysis, not as findings of this source. (LLM, conf=0.91)
- Partly supportedcites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The paper defines successful replication as P<0.05 in the original direction (introductory section, page unavailable), and its Discussion says that more than half of the RPP psychology findings failed to replicate. It does not report the stated 35/97 psychology or 11/18 experimental-economics counts, nor the χ²=3.96 comparison (searched those figures and the economics comparison). (LLM, conf=0.76)
plainstates an associationn = 18 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The recent Reproducibility Project Psychology (RPP) replicated 100 original studies published in three top journals in psychology. The vast majority (97) of the original studies reported 'positive findings', but in the replications the RPP only found a significant effect in the same direction for 36% of these studies ( 19 ).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
Evaluating Reproducibility (p. 7) defines the criterion as a significant effect in the same direction as the original and reports 35 positive results among 97 originally significant effects (36.1%). The Discussion (p. 12) summarizes this as 36% of the 100 replications having significant results, compared with 97% of the originals. The citing sentence accurately conveys the rounded result. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To test if the 'Original effect size within replication 95% CI' differed between the studies (66.7% (12/18) versus 47.4% (45/95)) we used the same test, but this was not significant ( 2 =2.25; P=0.133).”
What the sources say
- Supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). source
The cited paper reports a 47.4% success rate for the criterion of whether the original effect size falls within the replication 95% confidence interval (Evaluating Reproducibility, p. 9). Its supplement gives n=95 for that criterion (Moderator Analyses, p. 49), consistent with 45/95. The 66.7% economics figure and the chi-square comparison are presented as the citing authors’ own results, not findings attributed to this source. (LLM, conf=0.91)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results sections discuss psychology studies from the RPP and their replication outcomes, but do not report the 45/95 confidence-interval measure. A search for that measure and the quoted figures found no passage comparing it with 12/18 experimental-economics studies or reporting χ²=2.25 and P=0.133; page numbers are unavailable in our copy. (LLM, conf=0.76)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The same test was also used to test if the 'Meta-analytic estimate significant in the original direction' differed between the studies (77.8% (14/18) versus 68.0% (51/75)), but this difference was not significant ( 2 =0.66; P=0.417).”
What the sources say
- Partly supportedcites [19] 19. Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349 (2015). sourceExperimental — accuracy not yet measured
The ‘Combining original and replication effect sizes’ section (p. 44) supports the general point that about 70% of the psychology project's meta-analytic estimates were significant, but reports 52/74 (70.3%), not 51/75 (68.0%). The cited paper does not report the experimental-economics figure or the between-project chi-square test; those appear to be the citing authors' comparison. (LLM, conf=0.84)
- Could not verifycites [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). sourceExperimental — accuracy not yet measured
The Materials and Methods and Results describe prediction markets for a subset of RPP psychology studies, not a comparison with an experimental economics project. Our copy does not report the 14/18 versus 51/75 meta-analytic estimates or the stated chi-square test (searched the outcome phrase and the figures). (LLM, conf=0.75)
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Contrary to Dreber et al. ( 26 ) prediction market beliefs are not a more accurate indicator of replicability than survey beliefs.”
What the sources say
- Supportedcites Dreber et al. [26] 26. A. Dreber, et al., Using prediction markets to estimate the reproducibility of scientific research, Proc. Natl. Acad. Sci. U.S.A. 112 , 15343 (2015). source
The Results section reports significantly lower absolute prediction error for the prediction market than for either survey measure and concludes that the market outperformed the survey. The claim that this is not so is the citing authors’ own contrary conclusion, not a finding attributed to Dreber et al. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
45 more claims · 5 with their cited sources checked
Create a free account to see all 105 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may contribute to publication bias, selective reporting, and questionable research practices that reduce the credibility of the published literature 18-25 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“an analysis in experimental economics suggested that, of 18 studies, at least 11 had been successfully replicated (22).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“many established scientific results in the social (49, 50) and biomedical (51-53) sciences cannot readily be replicated.”
From Misinformation in and about science (2021) · cites this paper as [50]
In the citing paper
“This 35 times corresponds to the estimated publication bias implied by a large-scale replication of experimental economics papers by Camerer et al. (2016) (Table 1 of Andrews and Kasy (2018) reports that insignificant results are 0.029 times as likely to be published).”
From A unified welfare analysis of government policies. (2020) · cites this paper as Camerer et al. (2016)
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = -2.98reported p = .003 · recomputed p = .003
“z=-2.98, p = .003”
- Consistentz = 0.85reported p = .396 · recomputed p = .395
“z=0.85, p = .396”
- Consistentz = 0.33reported p = .744 · recomputed p = .741
“z=0.33, p = .744”
- Consistentz = -1.55reported p = .122 · recomputed p = .121
“z=-1.55, p = .122”
- Consistentz = -0.37reported p = .711 · recomputed p = .711
“z=-0.37, p = .711”
3 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ References and Notes:
“The mean prediction market belief is 75.2% [range 59% to 94%, 95% CI=(69.7%, 80.6%)]”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The same test was used to compare 'Prediction market beliefs' between the studies, and this difference (75.1% (n=18) versus 55.1% (n=44)) was significant (z=3.21, P=0.001).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 7. Results and data for the individual studies/markets
“The mean relative effect size of the replications is 65.9% [95% CI=(37.2%, 94.7%)].”
§ 5. Comparison of prediction market beliefs and survey beliefs
“The difference in 'Replication effect-size (% of original effect size)' was compared using a Mann-Whitney U test, but this difference (65.7% (n=18) versus 44.5% (n=94)) was not significant (z=1.39, P=0.166).”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
z-Tree: Zurich toolbox for ready-made economic experiments
Fischbacher, U. (2007) · Experimental Economics · doi:10.1007/s10683-006-9159-4 · Cited by 9,510
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Research works printed but not linked: 1
- HansonR., Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).
Cited by
Items 501–600 of 1,372
Next Generation Sequencing Claims in Precision Medicine: Questions a Formulary Committee Should Ask
Langley, P. C. (2016) · INNOVATIONS in pharmacy · doi:10.24926/iip.v7i4.473 · Cited by 9
Micheloud, C., Balabdaoui, F., & Held, L. (2023) · Statistica Neerlandica · doi:10.1111/stan.12312 · Cited by 21
“In the EERP, the replication sample sizes were calculated to reach “at least 90% power [ … ] to detect the original effect size at the [two-sided] 5% significance level” (Camerer et al., 2016, p. 1434).”
BackgroundMethodInfluentialWege aus der Vertrauenskrise: Individuelle Schritte hin zu verlässlicher und offener Forschung
Geukes, K., Schönbrodt, F. D., Utesch, T., Geukes, S., & Back, M. D. (2016) · Zeitschrift für Sportpsychologie · doi:10.1026/1612-5010/a000167 · Cited by 15
Reproducibility: A Primer on Semantics and Implications for Research
Pellizzari, E. D., Lohr, K. N., Blatecky, A. R., & Creel, D. V. (2017) · doi:10.3768/rtipress.2017.bk.0020.1708 · Cited by 13
Risky choice framing with various problem descriptions: A replication and extension study
Zhou, L., Liu, N., Liao, Y.-Q., & Aimei, L. I. (2021) · Judgment and Decision Making · doi:10.1017/s1930297500008615 · Cited by 16
Information markets and nonmarkets
Bergemann, D., & Ottaviani, M. (2021) · Handbook of Industrial Organization · doi:10.1016/bs.hesind.2021.11.008 · Cited by 24
A note on synthetic data for replication purposes in agricultural economics
Wimmer, S., & Finger, R. (2022) · Journal of Agricultural Economics · doi:10.1111/1477-9552.12505 · Cited by 15
Reporting in Experimental Philosophy: Current Standards and Recommendations for Future Practice
Polonioli, A., Vega‐Mendoza, M., Blankinship, B., & Carmel, D. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0414-3 · Cited by 12
Web-based experimental economics software: How do they compare to desirable features?
Chan, S. W., Schilizzi, S., Iftekhar, M. S., & Rosa, R. D. S. (2019) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2019.04.007 · Cited by 15
Toward a replication culture: Speech production research in the classroom
Roettger, T. B., & Baer‐Henney, D. (2019) · Phonological Data and Analysis · doi:10.3765/pda.v1art4.13 · Cited by 16
Realizing the promises of replication studies in education
Morrison, K. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2020.1838300 · Cited by 15
The Design of Replication Studies
Hedges, L. V., & Schauer, J. M. (2021) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12688 · Cited by 16
“A related project in behavioural economics tried to replicate 18 experiments each a single time and concluded that nearly a third of those attempts failed (Camerer et al., 2016).”
BackgroundMethodMcAuliffe, W. H. B., Edson, T. C., Louderback, E. R., LaRaja, A. S., & LaPlante, D. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0249926 · Cited by 14
Hensel, P. G. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1981870 · Cited by 17
Butera, L., Grossman, P. J., Houser, D. E., List, J. A., & Villeval, M. C. (2020) · National Bureau of Economic Research · doi:10.3386/w26801 · Cited by 15
“Finally, our paper contributes to the literature on the private provision of public goods, one of the most studied decision-making environments in the field of experimental eco- 8See also Camerer et al. (2016, 2018) 5 nomics (see, (Andreoni, 1995) for an early contribution).”
Open and reproducible neuroimaging: from study inception to publicationPreprint
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · doi:10.31219/osf.io/pu5vb · Cited by 12
Fraser, H. S., Bush, M., Wintle, B. C., et al. (2021) · doi:10.31222/osf.io/2pczv · Cited by 12
A Replication Study of Operations Management Experiments in Management SciencePreprint
Davis, A. M., Flicker, B., Hyndman, K., Katok, E., Keppler, S. M., Leider, S., Long, X., & Tong, J. D. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4135500 · Cited by 12
“To determine the target sample size for each site, we followed the power formula for a z-test used in other replication studies (e.g., Camerer et al. 2016).”
BackgroundMethodInfluentialUsing Survey Data to Determine a Numeric Criterion for Nutrient Pollution
Jakus, P. M., Nelson, N. M., & Ostermiller, J. D. (2017) · Water Resources Research · doi:10.1002/2017wr021527 · Cited by 12
A simple model suggesting economically rational sample-size choice drives irreproducibility
Braganza, O. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229615 · Cited by 12
“For instance [3, 4] find that true effect sizes are on average only around 50 to 60% of originally published effect sizes.”
BackgroundBuckley, J. (2024) · Journal of Technology Education · doi:10.21061/jte.v35i2.a.5 · Cited by 18
Decision Under Uncertainty: State of the Science
Gilboa, I. (2025) · Annual Review of Economics · doi:10.1146/annurev-economics-090924-041522 · Cited by 15
UThwigl — An R package for closed- and open-system uranium–thorium dating
Dosseto, A., & Marwick, B. (2021) · Quaternary Geochronology · doi:10.1016/j.quageo.2021.101235 · Cited by 11
Krypotos, A., Mertens, G., Klugkist, I. G., & Engelhard, I. M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_15 · Cited by 13
Tunç, D. U., Tunç, M. N., & Eper, Z. B. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221114835 · Cited by 21
The competitive advantage of sanctioning institutions revisited: A multilab replication
Iacono, S. L., Przepiorka, W., Buskens, V., Corten, R., van Assen, M. A. L. M., & van de Rijt, A. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad091 · Cited by 10
Campbell, H., & Gustafson, P. (2019) · The American Statistician · doi:10.1080/00031305.2018.1555101 · Cited by 14
“ee with the extent of this conclusion (e.g. Goodman & Greenland (2007), Leek & Jager (2017)), recent large-scale eorts to reproduce published results in a number of dierent elds (economics, Camerer et al. (2016); psychology, OpenScienceCollaboration (2015); oncology, Begley & Ellis (2012)), have also raise…”
BackgroundResultCompetition for novelty reduces information sampling in a research game - a registered report
Tiokhin, L., & Derex, M. (2019) · Royal Society Open Science · doi:10.1098/rsos.180934 · Cited by 11
Resource Sharing to Improve Research Quality
Hamra, G. B., Goldstein, N. D., & Harper, S. B. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.012292 · Cited by 12
“Similarly, weaker than reported effect sizes were found in a large-scale replication project in economics.(8) The challenge of replicating results is further supported by an”
ResultOkunade, A. A., & Osmani, A. R. (2020) · Applied Economics Letters · doi:10.1080/13504851.2020.1713976 · Cited by 24
“Moreover, there are recent calls in the social sciences for more replication studies (Camerer et al. 2016) to test the robustness of empirical research and policy suggestions (Hamermesh 2017; Kiri, Lacetera, and Zirulia 2018).”
BackgroundQuestionable research practices following pre-registrationPreprint
Ikeda, A., Xu, H.-Q., Fuji, N., Zhu, S., & Yamada, Y. (2019) · PsyArXiv (OSF Preprints) · doi:10.24602/sjpr.62.3_281 · Cited by 26
Replicability in Empirical Legal Research
Chin, J. M., & Zeiler, K. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-121620-085055 · Cited by 21
Merl, R., Stöckl, T., & Palan, S. (2021) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2022.106490 · Cited by 12
Bertoldo, G., Callegher, C. Z., & Altoè, G. (2022) · Meta-Psychology · doi:10.15626/mp.2020.2573 · Cited by 14
Schimmack, U., & Bartos̆, F. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290084 · Cited by 17
“First, large scale replication attempts [4, 5, 6] are inherently expensive and focus only on a limited set of pre-selected findings [7].”
BackgroundImproving Transparency in Observational Social Science Research: A Pre-Analysis Plan ApproachPreprint
Burlig, F. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2731920 · Cited by 11
The Dozen Things Experimental Economists Should Do (More of)
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019) · National Bureau of Economic Research · doi:10.3386/w25451 · Cited by 21
Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicabilityPreprint
Ebersole, C. R., Nosek, B. A., Kidwell, M. C., Buttrick, N., Baranski, E. N., Chartier, C. R., Mathur, M. B., IJzerman, H. R., Lazarević, L. B., Corker, K. S., et al. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/jmnsq · Cited by 17
Brodeur, A., Cook, N., & Heyes, A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4188289 · Cited by 17
AITurk: Using ChatGPT for Social Science ResearchPreprint
Qin, X., Huang, M., & Ding, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4922861 · Cited by 25
The duality of poverty: a replication of Mani et al. (2013) in Colombia
González-Arango, F., Corredor, J., López-Ardila, M. A., Contreras-González, M. C., Herrera-Santofimio, J., & González, J. J. (2021) · Theory and Decision · doi:10.1007/s11238-021-09836-x · Cited by 13
Huff, M., & Bongartz, E. C. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231156419 · Cited by 11
Registered reports and associated benefits for agricultural economics
Arpinon, T., & Lefebvre, M. (2024) · Q Open · doi:10.1093/qopen/qoae011 · Cited by 13
The assessment of replicability using the sum of p -values
Held, L., Pawel, S., & Micheloud, C. (2024) · Royal Society Open Science · doi:10.1098/rsos.240149 · Cited by 18
“The RPP [5], the EERP [6], the Social Sciences Replication Project ( SSRP ) [7] and the Experimental Philosophy Replicability Project ( EPRP ) [8] are large-scale replication projects which aimed to replicate important discoveries published in journals from their respective fields.”
BackgroundInfluentialEffect size benchmarks: Time for a causal renaissance
Amari, P. M., Banks, G. C., Bourque, L., Holladay-Sandidge, H. D., & O’Boyle, E. H. (2024) · The Leadership Quarterly · doi:10.1016/j.leaqua.2024.101855 · Cited by 14
How can we make sound replication decisions?
Davis‐Stober, C. P., Sarafoglou, A., Aczél, B., Chandramouli, S., Errington, T. M., Field, S. M., Fishbach, A., Freire, J., Ioannidis, J. P. A., Oberauer, K., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401236121 · Cited by 15
Smith, G. (2020) · SN Applied Sciences · doi:10.1007/s42452-020-2862-5 · Cited by 14
Prestige does not equal quality: Lack of research quality in high-prestige journals
Köhler, T., DeSimone, J. A., & Schoen, J. L. B. (2020) · Industrial and Organizational Psychology · doi:10.1017/iop.2020.53 · Cited by 12
2D:4D and Self-Employment: A Preregistered Replication Study in a Large General Population Sample
Fossen, F. M., Neyse, L., Johannesson, M., & Dreber, A. (2020) · Entrepreneurship Theory and Practice · doi:10.1177/1042258720985478 · Cited by 11
“Several recent large- scale replication projects have been conducted in the social sciences, which found evidence of strongly inflated effect sizes on average in the original studies subject to replication (Camerer et al., 2016, 2018; Open Science Collaboration, 2015). On average, the original effect sizes were about…”
Self-citationDoes the Market of Citations Reward Reproducible Work?
Raff, E. (2023) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3589806.3600041 · Cited by 23
“First, we will consider the results with respect to the previous fields of study (Medicine, Economics, Psychology, and Social) and show that we obtain consistent results and reasonably believe them to be a more reliable model.”
MethodNonspeculative bubbles revisited
Tucker, S. J., & Xu, Y. (2024) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2024.100925 · Cited by 10
Is economics self‐correcting? Replications in the American Economic Review
Peters, J., Fiala, N., & Neubauer, F. (2024) · Economic Inquiry · doi:10.1111/ecin.13222 · Cited by 29
“…forms of replicability problems and thereby emphasizes the need for more replications (Askarov et al., 2022; Brodeur et al., 2016, 2020, 2023; Camerer et al., 2016; Chang and Li, 2022; Dahal and Fiala, 2020; Ferraro and Shukla, 2020; Ioannidis et al., 2017; Peters et al., 2018; Vivalt, 2020).”
To Grasp Cognition in Action, Combine Behavioral Experiments with Protocol Analysis
Reypens, C., & Levine, S. S. (2017) · Emerald Group Publishing Limited eBooks · doi:10.1108/s2397-52102017006 · Cited by 13
Klingert, F. M. A., & Meyer, M. (2018) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.3577 · Cited by 10
“1.2 Prediction markets have been applied successfully both in research (Camerer et al. 2016; Dreber et al. 2015; Forsythe et al. 1992) and in practice (Atanasov et al. 2016; Chen & Plott 2002; Cowgill & Zitzewitz 2015).”
MethodInfluentialThe Impact of P -hacking on “Redefine Statistical Significance”
Crane, H. (2018) · Basic and Applied Social Psychology · doi:10.1080/01973533.2018.1474111 · Cited by 15
“In the real world, systematic departures from the NHST protocol cause FPR to be much higher than predicted by Equation (1) (e.g., Camerer et al., 2016; Open Science Collaboration, 2015; as noted in the section A Note About P-hacking, I refer to such departures from protocol generically as…”
BackgroundMethodInfluentialBayesian Data Analysis in Empirical Software Engineering: The Case of Missing Data
Torkar, R., FELDT, R. H., & Furia, C. A. (2020) · Springer eBooks · doi:10.1007/978-3-030-32489-6_11 · Cited by 16
Truth, Proof, and Reproducibility: There’s No Counter-Attack for the Codeless
Gray, C. T., & Marwick, B. (2019) · Communications in computer and information science · doi:10.1007/978-981-15-1960-4_8 · Cited by 11
“How we re-produce scientific results is being examined across a range of disciplines [6,35].”
BackgroundIntergenerational transmission of risk attitudes in Burkina Faso
Sepahvand, M. H., & Shahbazian, R. (2020) · Empirical Economics · doi:10.1007/s00181-020-01857-9 · Cited by 13
Kavanagh, C. M., & Kapitány, R. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.38713 · Cited by 11
Neves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., de Abreu, M. B., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · Memórias do Instituto Oswaldo Cruz · doi:10.1590/0074-02760200328 · Cited by 11
Epilogue: Randomization and Social Policy Evaluation Revisited
Heckman, J. J. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0014 · Cited by 13
The effect of gender and gender pairing on bargaining: Evidence from an artefactual field experiment
D’Exelle, B., Gutekunst, C., & Riedl, A. M. (2022) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2022.10.024 · Cited by 15
Scientific self-correction: the Bayesian way
Romero, F., & Sprenger, J. (2020) · Synthese · doi:10.1007/s11229-020-02697-x · Cited by 18
Information (Non)Aggregation in Markets with Costly Signal AcquisitionPreprint
Corgnet, B., DECK, C. A., DeSantis, M., & Porter, D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3103821 · Cited by 10
Wu, J. F., Nivargi, R., Lanka, S. S. T., Menon, A. M., Modukuri, S. A., Nakshatri, N. S., Wei, X., Wang, Z., Caverlee, J., Rajtmajer, S. M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04580 · Cited by 22
The Impact of Loss Aversion on Optimal PreventionPreprint
Macé, S., & Peter, R. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3882200 · Cited by 11
Scaling Nudges: Who Moves and HowPreprint
Saccardo, S., Dai, H., Han, M. A., Raja, N., Vangala, S., & Croymans, D. M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.3971192 · Cited by 17
Deressa, T. K., Stern, D. I., Vangronsveld, J., Minx, J. C., Lizin, S., Malina, R. M., & Bruns, S. B. (2023) · EcoEvoRxiv · doi:10.32942/x24g6z · Cited by 13
Sprawność intelektualna osób o odmiennych chronotypach
Jankowski, K. S. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial10 · Cited by 12
Inteligencja a funkcjonowanie w bliskich związkach
Leniarska, M., & Zajenkowski, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial11 · Cited by 8
Inteligencja a agresja. Czy wysokie IQ może uchronić przed wkroczeniem na ścieżkę przestępczą?
Bodecka-Zych, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial03 · Cited by 8
Inteligentny sport: o związkach inteligencji z aktywnością fizyczną i poziomem wykonania sportowego
Waleriańczyk, W., & Stolarski, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial09 · Cited by 8
Evaluating replicability of ten influential research on sensory marketing
Motoki, K., & Iseki, S. (2022) · Frontiers in Communication · doi:10.3389/fcomm.2022.1048896 · Cited by 12
“, 2018), economics (Camerer et al., 2016), and behavioral sciences (Camerer et al.”
BackgroundGamiDOC: The Importance of Designing Gamification in a Proper Way
Bassanelli, S., Bucchiarone, A., & Gini, F. (2024) · IEEE Transactions on Games · doi:10.1109/tg.2024.3364061 · Cited by 13
“Moreover, these techniques can also provide the following: 1)thepossibility of successful replication [115]; 2) the generalization; 3) the data transparency [116]; 4) allow fast publications; 5) provide control over the study design and the methodological rigor [117].”
Conventional wisdom, meta‐analysis, and research revision in economics
Gechert, S., Mey, B., Opatrný, M., Havránek, T., Stanley, T. D., Bom, P. R. D., Doucouliagos, H., Heimberger, P., Iršová, Z., & Rachinger, H. J. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12630 · Cited by 15
Pia, F. D., Shi, B. X., Al-Hamdani, Y. S., Alfè, D., Anderson, T. A., Barborini, M., Benali, A., Casula, M., Drummond, N. D., Dubecký, M., et al. (2025) · The Journal of Chemical Physics · doi:10.1063/5.0272974 · Cited by 16
Viewpoint: The costs and benefits of deception in economic experiments
Lusk, J. L. (2019) · Food Policy · doi:10.1016/j.foodpol.2018.12.009 · Cited by 9
Variation and Covariation in Large-Scale Replication Projects: An Evaluation of Replicability
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2054816 · Cited by 13
“…one-tenth for cancer research studies (Begley and Ellis 2012), one-third for psychological research studies (Open Science Collaboration 2015), six-tenths for economics research studies (Camerer et al. 2016), and sixth-tenths for research studies across the social sciences (Camerer et al. 2018).”
BackgroundIs forensic science in crisis?
Sikorski, M. (2022) · Synthese · doi:10.1007/s11229-022-03685-z · Cited by 14
“…over replicability have been raised in many other scientific disciplines such as biology (see e.g., Errington et al. 2021), physics (see e.g., Camerer et al. 2016), geography (see e.g., Kedron et al. 2021 or Sui and Kedron 2020), and a significant similarity can be found in both the causes…”
BackgroundBeyond Open Access: Conceptualizing Open Science for Knowledge Co-creation
Mačiulienė, M. (2022) · Frontiers in Communication · doi:10.3389/fcomm.2022.907745 · Cited by 14
“However, replication studies that can combat such academic misconduct are scarcely practiced (Camerer et al., 2016; Mueller-Langer et al., 2019).”
BackgroundA toolkit for open and pluralistic conservation science
Burgman, M. A., Chiaravalloti, R. M., Fidler, F. M., Huan, Y., McBride, M. F., Marcoci, A., Norman, J., Vercammen, A., Wintle, B. C., & Yu, Y. (2022) · Conservation Letters · doi:10.1111/conl.12919 · Cited by 12
“The latter has contributed to what is known as the “reproducibility crisis,” which refers to the failure to successfully replicatemany scientific findings across disciplines from preclinical medicine to economics (e.g., Begley & Ellis, 2012; Open Science Collaboration, 2015; Camerer et al, 2016).”
BackgroundPriestley, D. R., Staph, J., Koneru, S., Rajtmajer, S. M., Cwiek, A. P., Vervoordt, S. M., & Hillary, F. G. (2022) · Brain Communications · doi:10.1093/braincomms/fcac322 · Cited by 7
Success-Breeds-Success in Collective Political Behavior: Evidence from a Field Experiment
van de Rijt, A., Akın, A. İ., Willer, R., & Feinberg, M. (2016) · Sociological Science · doi:10.15195/v3.a41 · Cited by 12
“many studies to replicate (The Open Science Collaboration 2015; Camerer et al. 2016), we hope our study will inspire others to engage in efforts to solidify the empirical knowledge base on which the advancement of social science research crucially depends.”
Creating falseness—How to establish statistical evidence of the untrue
Lytsy, P. (2017) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12823 · Cited by 9
“This seems to be true in test simulations,(25) in psychological research(26) as well as in laboratory economic research.(27) No large‐scale replication study has yet been performed on medical studies, but reports from the industry suggest that many initial new drug target claims are rarely reproducible.”
BackgroundJoy and rigor in behavioral science
Collins, H. K., Whillans, A. V., & John, L. K. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.03.002 · Cited by 10
The effects of base rate neglect on sequential belief updating and real-world beliefs
Ashinoff, B. K., Buck, J., Woodford, M., & Horga, G. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010796 · Cited by 13
“Second, we used a validated belief-elicitation procedure that financially incentivized participants to report their true beliefs [35,36,38].”
MethodShen, S., Huang, Y., Jiang, C., & Li, S. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500003442 · Cited by 11
Brauer, J. R., Day, J. C., & Hammond, B. M. (2019) · Sociological Methods & Research · doi:10.1177/0049124119826158 · Cited by 9
“…about a “reproducibility crisis” have recently peaked, triggered by numerous failed attempts to replicate high-profile studies across several scientific disciplines (cf. Camerer et al. 2016; Errington et al. 2014; Freese and Peterson 2017; Jasny et al. 2011; Open Science Collaboration 2015).”
BackgroundAnderson, B., Rushby, T., Bahaj, A. S., & James, P. A. B. (2019) · Energy Research & Social Science · doi:10.1016/j.erss.2019.101260 · Cited by 13
“Both historical [8] and more recent discussions of the role of p values in inference [15,17,31] and recent commentary on the replicability crisis in both energy studies and elsewhere [4,16,18,21], remind us that decisions should never be based solely on statistical significance thresholds set purely by convention.”
BackgroundFrom Documents to Data: A Framework for Total Corpus QualityPreprint
Bodell, M. H., Magnusson, M., & Mützel, S. (2022) · doi:10.31235/osf.io/ft84u · Cited by 9
Heckelei, T., Hüttel, S., Rommel, J., & Odening, M. (2022) · Preprints.org · doi:10.20944/preprints202201.0311.v1 · Cited by 9
“Researchers have highlighted problems with the direct replicability of research results especially in experimental economics and business economics’ studies, and the sensitivity of research results to context (Camerer et al., 2016; Rahwan, Yoeli and Fasolo, 2019).”
BackgroundInfluentialReplication of null results: Absence of evidence or evidence of absence?
Pawel, S., Heyard, R., Micheloud, C., & Held, L. (2023) · eLife · doi:10.7554/elife.92311 · Cited by 25
“…in diverse fields, from the biomedical to the social sciences ( Prinz et al., 2011 ; Begley and Ellis, 2012 ; Klein et al., 2014 ; Open Science Collaboration, 2015 ; Camerer et al., 2016 ; Camerer et al., 2018 ; Klein et al., 2018 ; Cova et al., 2021 ; Errington et al., 2021 , among others).”
BackgroundEarn More Tomorrow: Overconfidence, Income Expectations, and Consumer Indebtedness
Grohmann, A., Menkhoff, L., Merkle, C., & Schmacker, R. (2019) · Journal of money credit and banking · doi:10.1111/jmcb.13157 · Cited by 19
Baaken, M. C., Kühfuss, L., Bougherara, D., Czajkowski, M., Rodríguez‐Entrena, M., Höhler, J., Lagerkvist, C. J., Paparella, A., Quendler, E., Rommel, J., et al. (2024) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13488 · Cited by 12
Marwick, B. (2025) · Journal of Archaeological Science · doi:10.1016/j.jas.2025.106281 · Cited by 14
“Similar empirical replications of large numbers of social science studies and experimental economics studies successfully replicated 61 % and 62 % of their target studies respectively (Camerer et al., 2016, 2018).”
Behavioural Economics for Understanding Recreational Fishers and Fisheries
Mackay, M., van Putten, I. E., Richter, A. P., Abatayo, A. L., Chuah, S., Dankel, D. J., Schadeberg, A., & Snyder, H. T. (2025) · Fish & fisheries series/Fish and fisheries series (Print) · doi:10.1007/978-3-031-99739-6_9 · Cited by 8
Research Replication Prediction Using Weakly Supervised Learning
Luo, T., Li, X., Wang, H., & Liu, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.132 · Cited by 12
“…replication projects which are the Registered Replication Report (RRR) (Simons et al., 2014 (Camerer et al., 2018), PsychFileDrawer (Pashler et al., 2019), Experimental Economics Replication Project (Camerer et al., 2016 Among 399 annotated samples, 201 samples are labeled as ‘1’ (replicable).”
BackgroundInfluentialAn evaluation of statistical methods for aggregate patterns of replication failure
Schauer, J. M., Fitzgerald, K. G., Peko-Spicer, S., Whalen, M. C. R., Zejnullahi, R., & Hedges, L. V. (2021) · The Annals of Applied Statistics · doi:10.1214/20-aoas1387 · Cited by 8
Chen, J., Kwan, C., Yeung, L. Y., Choi, H. Y., Lo, Y. C., Au, S. Y., Tsang, C. H., Cheng, B., & Feldman, G. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104154 · Cited by 22
Objectivity of the peer‐review process: Enduring myth, reality, and possible remedies
Baveye, P. C. (2021) · Learned Publishing · doi:10.1002/leap.1414 · Cited by 27
“It has since become increasingly clear that his verdict is on the mark for large numbers of publications in fields like psychology, sociology, medicine, and economics (e.g., Camerer et al., 2016; Muller, 2020; Nuijten, 2016; Ritchie, 2020; Świątkowski & Dompnier, 2017).”
Background
Metadata
- Authors
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.
- Affiliations
- California Institute of Technology (US)
- Stockholm School of Economics (SE)
- National University of Singapore (SG)
- Universität Innsbruck (AT)
- Swedish National Bank (SE)
- Institute for Advanced Study (DE)
- Corresponding author
- Camerer, Colin F.
- Dreber, Anna
- Forsell, Eskil
- Ho, Teck-Hua
- Huber, Jürgen
- Johannesson, Magnus
- Kirchler, Michael
- ORCID
- Venue
- Science
- Published
- 2016-03-25
- Volume
- 351
- Issue
- 6280
- Pages
- 1433-1436
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2292391400
- Cited by
- 1,314
- References
- 77
- Citations by year
- 2026: 62 · 2025: 102 · 2024: 115 · 2023: 140 · 2022: 141 · 2021: 163 · 2020: 188 · 2019: 166 · 2018: 103 · 2017: 82 · 2016: 47
- FWCI
- 326.88
- Type
- journal article
- Language
- en
- Open access
- Licence
- public-domain
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- National University of Singapore (submittedVersion, public-domain)
- Munich Personal RePEc Archive (Ludwig Maximilian University of Munich) (submittedVersion)
- Funders
- Austrian Science Fund (Y617-G11)
- Austrian National Bank (OeNB 14953)
- Behavioral and Neuroeconomics Discovery Fund
- Jan Wallander and Tom Hedelius Foundation (P2015-0001:1, P2013-0156:1)
- Knut and Alice Wallenberg Foundation
- Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1)
- Sloan Foundation (G-2015-13929)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replicability, replication study, laboratory experiments, economic research, effect size, statistical power, prediction markets, American Economic Review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.