Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decision-making tasks. Further, the subjective measures on evaluations with actual decision-making tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing human+AI teams that can reliably perform better than humans or AIs alone.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some researchers measure subjective trust using questionnaires, while others measure objective trust related behavior such as reliance.”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Buçinca et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, across many domains, empirical studies of explanations have produced mixed results [4,11,49,53,58,68,82,87], with some suggesting that explanations increase decisionmakers' tendency to rely on AI predictions even when the AI system is wrong [4,68,82,87]-a phenomenon often referred to as overreliance [10,78].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wang and Yin [82] found that example-based explanations underperform feature-based explanations for improving appropriate reliance, though other studies suggest example-based explanations are better at helping people detect model errors [11,13,49], usually by recognizing the dissimilarity between the instance and selected examples.”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“having the ML produce the maximally convincing explanation - a common objective shared in the design of many AI explanation algorithms - might be a poor choice for complementary performance [12].”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Of the possible error types in human-AI decision-making (Figure 1), overreliance is the most frequent outcome in empirical AI studies [4,8,9,53,97].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [8]
Experimental — accuracy not yet measured
22 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858 · cited 1×
“Transcripts were coded and analyzed for paerns using an inductive approach [39].”
MethodTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“explainable interfaces should be instilling in users the right amount of trust [10, 32, 33].”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Notably, Doshi-Velez & Kim [9] proposed a taxonomy for evaluation of explainable AI systems, composed of the following categories: application grounded evaluation (i.”
BackgroundExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“Motivated by the growing number of studies in interpretable and explainable AI-powered decision support systems, researchers have called for more rigorous evaluation of explainable systems [9, 14, 19].”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Because people and AI-powered systems have complementary strengths, many expected that human+AI teams would perform better on decision-making tasks than either people or AIs alone [1, 21, 22].”
BackgroundDeslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821936116 · Cited by 1,371 · cited 1×
“that decision, and incorporating that information into the decisionmaking process requires cognitive effort and the existing evidence suggests that preference does not predict performance on cognitive tasks [8, 12, 37].”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“explainable interfaces should be instilling in users the right amount of trust [10, 32, 33].”
BackgroundDecision making and the avoidance of cognitive demand
Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010) · Journal of Experimental Psychology: General · doi:10.1037/a0020198 · Cited by 1,141 · cited 1×
“However, people are reluctant to exert cognitive effort [24, 25] unless they are motivated or forced to do so.”
BackgroundHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 1×
“Even though conceptual frameworks that consider cognitive processes in decision-making with DSS have been proposed recently [41], further research is needed to thoroughly investigate how to incorporate DSS into human decision-making and the eect of cognitive processes while making system-assisted decisions.”
BackgroundTrust between humans and machines, and the design of decision aids
Muir, B. M. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80013-5 · Cited by 762
Johnston, M. E., Langton, K. B., Haynes, R. B., & Mathieu, A. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-2-199401150-00007 · Cited by 714 · cited 1×
“, management [15], medicine [20], justice [47]).”
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711 · cited 5×
“However, there is mounting evidence that human+AI teams oen perform worse than AIs alone [16, 17, 28, 34].”
BackgroundMethodInfluentialInterpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · cited 2×
“, human+AI teams working together to make complex decisions, we oen make one of two possible critical mistakes: (1) Rather than evaluating how well the human+AI team performs together on a decision-making task, we evaluate by using proxy tasks, how accurately a human can predict the decision or decision boundaries of…”
MethodPrinciples of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 1×
“These may take the form of questionnaires on attitude and confidence in the system [18] and helpfulness of the system [5, 26].”
BackgroundBlumenthal-Barby, J. S., & Krieger, H. (2015) · Medical Decision Making · doi:10.1177/0272989x14547740 · Cited by 545 · cited 1×
“ecisions [40]. „ese heuristics save time and e‡ort, and frequently lead to good outcomes, but in some situations they result in cognitive biases that systematically lead to poor decisions (see, e.g., [4]). To help people make good decision reliably, computer-based Decision Support Systems (DSS) have been used across n…”
BackgroundUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task [3, 17, 23, 46].”
MethodMetrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · cited 2×
“[19] proposed metrics for explainable systems that are grounded in the subjective evaluation of a system (e.”
BackgroundBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 1×
“Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task [3, 17, 23, 46].”
MethodCognitive forcing strategies in clinical decisionmaking
Croskerry, P. (2003) · Annals of Emergency Medicine · doi:10.1067/mem.2003.22 · Cited by 440 · cited 2×
“One approach for mitigating cognitive biases in decision-making is to use cognitive forcing strategies, which introduce self-awareness and self-monitoring of decision-making [7].”
MethodCognitive Effort, Affect, and Choice
Garbarino, E. C., & Edell, J. A. (1997) · Journal of Consumer Research · doi:10.1086/209500 · Cited by 422 · cited 2×
“They also make substantially different decisions depending on whether they choose to exert cognitive effort or not [12, 37].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
The Theoretical Framework of Cognitive Informatics
Wang, Y. (2007) · International Journal of Cognitive Informatics and Natural Intelligence · doi:10.4018/jcini.2007010101 · Cited by 400
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 6×
“Although not universally effective [38], these strategies have shown promising results as they improve decision-making performance, both if the human is assisted [17, 34] or is not assisted by a DSS [31].”
BackgroundMethodResultInfluentialLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 2×
“n [16] discovered that self-reported measures could be misleading, since participant’s con•dence in their performance was negatively associated with their actual performance. Similarly, Lai & Tan [28] found that humans cannot accurately estimate their own performance. More closely related to our •ndings, Poursabzi…”
BackgroundA layered reference model of the brain (LRMB)
Wang, Y., Wang, Y., Patel, S., & Patel, D. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2006.871126 · Cited by 355 · cited 1×
“of alternatives [42, 43, 45].”
Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments
Green, B., & Chen, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287563 · Cited by 275 · cited 4×
“For example, Green & Chen [16] discovered that self-reported measures could be misleading, since participant’s confidence in their performance was negatively associated with their actual performance.”
BackgroundInfluentialInterpretable Basis Decomposition for Visual Explanation
Zhou, B., Sun, Y., Bau, D., & Torralba, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_8 · Cited by 260 · cited 1×
“In explainable AI literature, subjective measures, such as user trust and experience, have been largely embraced as the focal point for the evaluation of explainable systems [35, 36, 44, 48].”
Background"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241
Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review
Lambe, K. A., O'Reilly, G., Kelly, B. D., & Curristan, S. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004417 · Cited by 215 · cited 8×
“Although not universally effective [38], these strategies have shown promising results as they improve decision-making performance, both if the human is assisted [17, 34] or is not assisted by a DSS [31].”
BackgroundInfluentialSupportingTrust in Autonomous Driving
Häuslschmid, R., von Bülow, M., Pfleging, B., & Butz, A. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025198 · Cited by 170 · cited 1×
“These may take the form of questionnaires on attitude and confidence in the system [18] and helpfulness of the system [5, 26].”
BackgroundWeitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329441 · Cited by 136 · cited 3×
“intelligent systems [2]. In explainable AI literature, subjective measures, such as user trust and experience, have been largely embraced as the focal point for the evaluation of explainable systems [35, 36, 44, 48]. Ho‡man et al. [19] proposed metrics for explainable systems that are grounded in the subjective evalua…”
BackgroundMethodHuman Evaluation of Models Built for Interpretability
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S. J., & Doshi-Velez, F. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5280 · Cited by 110 · cited 2×
“is up to them to decide whether and how to use the AI’s assistance to accomplish the task. In contrast, studies that use proxy tasks evaluate how well users are able to simulate the model’s decisions [6, 13, 27, 34] or decision boundaries [29]. In such studies, participants are speci•cally instructed to pay a−ention t…”
BackgroundSherbino, J., Kulasegaram, K., Howey, E., & Norman, G. (2014) · CJEM · doi:10.2310/8000.2013.130860 · Cited by 108 · cited 1×
“Although not universally eective [38], these strategies have shown promising results as they improve decision-making performance, both if the human is assisted [17, 34] or is not assisted by a DSS [31].”
BackgroundPredictive text encourages predictable writing
Arnold, K. C., Chauncey, K., & Gajos, K. Z. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377523 · Cited by 87 · cited 1×
“ipants to simulate the interpretable model’s predictions [27]. In addition to the evaluation task, the choice of evaluation metrics is a critical one for the correct evaluation of intelligent systems [2]. In explainable AI literature, subjective measures, such as user trust and experience, have been largely embraced a…”
BackgroundSelf-citationExplainable Autonomy: A Study of Explanation Styles for Building Clear Mental Models
Garcia, F. J. C., Robb, D. A., Liu, X., Laskov, A., Patrón, P., & Hastie, H. W. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6511 · Cited by 45 · cited 2×
“In contrast, studies that use proxy tasks evaluate how well users are able to simulate the model’s decisions [6, 13, 27, 34] or decision boundaries [29].”
Background"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06473 · Cited by 22
Research works printed but not linked: 7
- [6] Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems . 288-296. -
- [11] K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data. the MIT Press. -
- [21] Ece Kamar. 2016. Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence.. In IJCAI . 4070-4073. -
- [22] Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume - 1 . International Foundation for Autonomous Agents and Multiagent Systems, 467-474. -
- [24] Wouter Kool and Matthew Botvinick. 2018. Mental labour. Nature human behaviour 2, 12 (2018), 899-908. -
- [37] Anuj K Shah and Daniel M Oppenheimer. 2008. Heuristics made easy: An effort-reduction framework. Psychological bulletin 134, 2 (2008), 207. -
- [47] John Zeleznikow. 2004. Building intelligent legal decision support systems: Past practice and future challenges. In Applied Intelligent Systems . Springer, 201-254.
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“However, evidence suggests that explainable systems, also, have not had substantial success in reducing human overreliance on the AIs: when the AI suggests incorrect or suboptimal solutions, people still on average make poorer final decisions than they would have without AI’s assistance [2, 7, 30, 40, 67].”
BackgroundResultInfluentialSelf-citationMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606
“Since human-centered evaluations employ experiments with end-users or lay humans to assess the quality of explanations, subjective measures, such as trust and confidence, have been embraced as the focal point for the evaluation of explainable systems [33,64,65,69,92,93].”
BackgroundMethod
Metadata
- Authors
- Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L.
- Affiliations
- Harvard University (US)
- Venue
- Proceedings of the 25th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2020-03-17
- Pages
- 454-464
- OpenAlex
- W3001062618
- Cited by
- 307
- References
- 33
- Citations by year
- 2026: 52 · 2025: 53 · 2024: 57 · 2023: 55 · 2022: 46 · 2021: 32 · 2020: 12
- FWCI
- 13.83
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- explainable AI, XAI evaluation, proxy tasks, human-AI teaming, decision performance, trust in AI, think-aloud study, human-AI collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.