Explainable AI Evaluation: A Top-Down Approach for Selecting Optimal Explanations for Black Box Models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Explainable Artificial Intelligence (XAI) evaluation has grown significantly due to its extensive adoption, and the catastrophic consequence of misinterpreting sensitive data, especially in the medical field. However, the multidisciplinary nature of XAI research resulted in diverse scholars possessing significant challenges in designing proper evaluation methods. This paper proposes a novel framework of a three-layered top-down approach on how to arrive at an optimal explainer, accenting the persistent need for consensus in XAI evaluation. This paper also investigates a critical comparative evaluation of explanations in both model agnostic and specific explainers including LIME, SHAP, Anchors, and TabNet, aiming to enhance the adaptability of XAI in a tabular domain. The results demonstrate that TabNet achieved the highest classification recall followed by TabPFN, and XGBoost. Additionally, this paper develops an optimal approach by introducing a novel measure of relative performance loss with emphasis on faithfulness and fidelity of global explanations by quantifying the extent to which a model’s capabilities diminish when eliminating topmost features. This addresses a conspicuous gap in the lack of consensus among researchers regarding how global feature importance impacts classification loss, thereby undermining the trust and correctness of such applications. Finally, a practical use case on medical tabular data is provided to concretely illustrate the findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Another challenge is the lack of standardized metrics to evaluate the quality and usefulness of explanations. What is considered a good explanation depends on the clinical context, the user’s expertise, and the decision at hand [29,30].”
From Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges (2025) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Narayan, V., Mall, P. K., Alkhayyat, A. H., Abhishek, K., Kumar, S., & Pandey, P. (2023) · Journal of Sensors · doi:10.1155/2023/8276738 · Cited by 58
RetractedSMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“The toolbox encompasses several explainers such as Saliency Map [44], Grad-CAM [29], Integrated Gradients [45], etc. Dalex [46] is a model-agnostic explanation toolbox that supports popular models such as Caret [47], random forests, and Gradient Boosting Machines.”
MethodAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 6×
“To address these challenges, Anchors introduced high-precision, rule-based, and model-agnostic explanations that utilize reinforcement learning by applying simple if-then rules [14,59].”
BackgroundMethodResultInfluentialMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 10×
“A study conducted by Carvalho et al. [6] showed that when two models contribute similar features to arrive at a prediction with consistent explanations, they exhibit a high degree of consistency, signifying a robust explanatory characteristic.”
BackgroundInfluentialTabNet: Attentive Interpretable Tabular Learning
Arik, S. Ö., & Pfister, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16826 · Cited by 1,771
Visualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535
Challenges of Big Data analysis
Fan, J., Han, F., & Liu, H. (2014) · National Science Review · doi:10.1093/nsr/nwt032 · Cited by 1,482
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268
Techniques for Interpretable Machine Learning
Du, M., Liu, N., & Hu, X. (2018) · arXiv · doi:10.48550/arxiv.1808.00033 · Cited by 1,261
pdp: An R Package for Constructing Partial Dependence Plots
Greenwell, B. (2017) · The R Journal · doi:10.32614/rj-2017-016 · Cited by 1,189
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a SecondPreprint
Hollmann, N., Müller, S., Eggensperger, K., & Hutter, F. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2207.01848 · Cited by 1,155
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Layer-Wise Relevance Propagation: An Overview
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_10 · Cited by 750
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 8×
“Robust explanations are desired because the user expects the model to behave consistently by generating similar explanations for similar instances [3,6,36,61].”
BackgroundInfluentialEffective Heart Disease Prediction Using Machine Learning Techniques
Bhatt, C. M., Patel, P., Ghetia, T., & Mazzeo, P. L. (2023) · Algorithms · doi:10.3390/a16020088 · Cited by 586
Diabetes prediction using machine learning and explainable AI techniques
Tasin, I., Nabil, T. U., Islam, S., & Khan, R. (2023) · Healthcare Technology Letters · doi:10.1049/htl2.12039 · Cited by 344
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268 · cited 1×
“AIX360 [51] is an alternative tool supporting tabular, text, image, and time-series data.”
MethodTransTab: Learning Transferable Tabular Transformers Across Tables
Wang, Z., & Sun, J. (2022) · neural information processing systems · doi:10.52202/068431-0210 · Cited by 262
Contributions to the Theory of Games (AM-28), Volume II
(1953) · Princeton University Press eBooks · doi:10.1515/9781400881970 · Cited by 260
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314229 · Cited by 256
Why model why? Assessing the strengths and limitations of LIMEPreprint
Dieber, J., & Kirrane, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00093 · Cited by 254
DALEX: Explainers for Complex Predictive Models in R
Biecek, P. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3670939 · Cited by 247
OpenXAI: Towards a Transparent Evaluation of Model ExplanationsPreprint
Agarwal, C., Ley, D., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Zitnik, M., & Lakkaraju, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11104 · Cited by 223
Explainable AI for Bioinformatics: Methods, Tools and Applications
Karim, M. R., Islam, T., Shajalal, M., Beyan, O., Lange, C., Cochez, M., Rebholz-Schuhmann, D., & Decker, S. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad236 · Cited by 196
Kibria, H. B., Nahiduzzaman, M., Goni, M. O. F., Ahsan, M., & Haider, J. (2022) · Sensors · doi:10.3390/s22197268 · Cited by 186
Explainable AI for Bioinformatics: Methods, Tools, and Applications
Karim, M. R., Islam, T., Shajalal, M., Beyan, O., Lange, C., Cochez, M., Rebholz-Schuhmann, D., & Decker, S. (2022) · Briefings Bioinform. · doi:10.48550/arxiv.2212.13261 · Cited by 176
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170 · cited 1×
“Another study by Mohammad Naiseh et al. [71] explored four different explainability methods including local, example-based, counter-factual, and global explanations during human-AI collaborative decision-making tasks to investigate how different explanations impact trust calibration in clinical…”
MethodGrezmak, J. E., Zhang, J., Wang, P. E., Loparo, K. A., & Gao, R. X. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2958787 · Cited by 166
Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., & Guyon, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.10201 · Cited by 158
Model Agnostic Supervised Local Explanations
Plumb, G., Molitor, D., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1807.02910 · Cited by 149
An automatic heart disease prediction using cluster-based bi-directional LSTM (C-BiLSTM) algorithm
Dileep, P., Rao, K. N., Bodapati, P., Gokuruboyina, S., Peddi, R., Grover, A., & Sheetal, A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07064-0 · Cited by 144
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · cited 1×
“Despite the numerous tabular data across various domains, it is surprising that most XAI techniques are not inherently suitable for such data format, and engaging their applications constantly presents challenges [27].”
BackgroundXAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131 · cited 2×
“Similarly, a metareview conducted by Clement et al. [36] demonstrates a high correlation between the evaluation method and the complexity of the development process.”
BackgroundMethodHedström, A., Weber, L., Bareeva, D., Krakowczyk, D., Motzkus, F., Samek, W., Lapuschkin, S., & Höhne, M. M. C. (2022) · arXiv · doi:10.48550/arxiv.2202.06861 · Cited by 116
SuperTML: Two-Dimensional Word Embedding for the Precognition on Structured Tabular Data
Sun, B., Yang, L., Zhang, W., Lin, M., Dong, P., Young, C., & Dong, J. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00360 · Cited by 114
OpenXAI: Towards a Transparent Evaluation of Model Explanations
Agarwal, C., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Žitnik, M., & Lakkaraju, H. (2022) · neural information processing systems · doi:10.52202/068431-1148 · Cited by 98
An Empirical Evaluation of AI Deep Explainable Tools
Hailemariam, Y., Yazdinejad, A., Parizi, R. M., Srivastava, G., & Dehghantanha, A. (2020) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps50303.2020.9367541 · Cited by 90
To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
Amparore, E., Perotti, A., & Bajardi, P. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.479 · Cited by 90
Alibi Explain: Algorithms for Explaining Machine Learning Models
Klaise, J., Looveren, A. V., Vacanti, G., & Coca, A. (2021) · Journal of Machine Learning Research · Cited by 78
Xplique: A Deep Learning Explainability ToolboxPreprint
Fel, T., Hervier, L., Vigouroux, D., Poché, A., Plakoo, J., Cadène, R., Chalvidal, M., Colin, J., Boissin, T., Béthune, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.04394 · Cited by 77
XAI Evaluation: Evaluating Black-Box Model Explanations for Prediction
Zhang, Y., Xu, F., Zou, J., Petrosian, O. L., & Krinkin, K. V. (2021) · 2021 II International Conference on Neural Networks and Neurotechnologies (NeuroNT) · doi:10.1109/neuront53022.2021.9472817 · Cited by 76
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
A novel model usability evaluation framework (MUsE) for explainable artificial intelligence
Dieber, J., & Kirrane, S. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.11.017 · Cited by 67
Duell, J., Fan, X., Burnett, B., Aarts, G., & Zhou, S.-M. (2021) · 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) · doi:10.1109/bhi50953.2021.9508618 · Cited by 62
How model accuracy and explanation fidelity influence user trust
Papenmeier, A., Englebienne, G., & Seifert, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12652 · Cited by 58
A Review: Heart Disease Prediction in Machine Learning & Deep Learning
Bakar, W. A. W. A., Josdi, N. L. N. B., Man, M. B., & Zuhairi, M. A. B. (2023) · IEEE International Colloquium on Signal Processing & Its Applications (CSPA) · doi:10.1109/cspa57446.2023.10087837 · Cited by 57
Mandloi, S., Zuber, M., & Gupta, R. K. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16708-9 · Cited by 43
TransTab: Learning Transferable Tabular Transformers Across TablesPreprint
Wang, Z., & Sun, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09328 · Cited by 37
Burger, C., Chen, L., & Le, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.792 · Cited by 37
PTab: Using the Pre-trained Language Model for Modeling Tabular DataPreprint
Liu, G., Yang, J., & Wu, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.08060 · Cited by 37
Sanderson, J. E., Tengtrairat, N., Woo, W. L., Mao, H., & Al-Nima, R. R. O. (2023) · International Journal of Remote Sensing · doi:10.1080/01431161.2023.2288945 · Cited by 26
Self-citationFlood Prediction and Analysis on the Relevance of Features using Explainable Artificial Intelligence
Kadiyala, S. P., & Woo, W. L. (2021) · Artificial Intelligence and Complex Systems Conference (AICSconf) · doi:10.1145/3516529.3516530 · Cited by 26
Self-citationBelaid, M. K., Hüllermeier, E., Rabus, M., & Krestel, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.14160 · Cited by 24
Shapley Value as an Aid to Biomedical Machine Learning: a Heart Disease Dataset Analysis
Scapin, D., Cisotto, G., Gindullina, E., & Badia, L. (2022) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid54584.2022.00113 · Cited by 21
Integrated Gradients for Feature Assessment in Point Cloud-Based Data Sets
Schwegler, M., Müller, C. E., & Reiterer, A. (2023) · Algorithms · doi:10.3390/a16070316 · Cited by 15
Explaining COVID-19 diagnosis with Taylor decompositions
Hassan, M. M., AlQahtani, S. A., Alelaiwi, A. A., & Papa, J. P. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08021-7 · Cited by 13
Flood Prediction and Analysis on the Relevance of Features using Explainable Artificial IntelligencePreprint
Kadiyala, S. P., & Woo, W. L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05046 · Cited by 9
Self-citationScafarto, G., Posocco, N., & Bonnefoy, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26387-3_21 · Cited by 9
Investigating the impact of calibration on the quality of explanations
Löfström, H., Löfström, T., Johansson, U., & Sönströd, C. (2023) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-023-09837-2 · Cited by 9
Spurious Correlations and Where to Find ThemPreprint
Sreekumar, G., & Boddeti, V. N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.11043 · Cited by 9
The Merging Path Plot: adaptive fusing of k-groups with likelihood-based model selectionPreprint
Sitko, A., & Biecek, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04412 · Cited by 8
An Experimental Investigation into the Evaluation of Explainability MethodsPreprint
Stassin, S., Englebert, A., Nanfack, G., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16361 · Cited by 3
Burger, C., Chen, L., & Lê, T. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.12351
Research works printed but not linked: 4
- Gurgul, 2022, A Parameters-Based Heart Disease Prediction Model, Age, 53, 28
- Thien, T.Q. Concept and Pattern Discovery for Reliable and Explainable AI. [PhD Thesis, University of Tsukuba].
- Salih, A., Raisi-Estabragh, Z., Galazzo, I.B., Radeva, P., Petersen, S.E., Menegaz, G., and Lekadir, K. (2023). Commentary on Explainable Artificial Intelligence Methods: SHAP and LIME. arXiv.
- Famiglini, L., Campagner, A., and Cabitza, F. (2023). ECAI 2023, IOS Press.
Cited by
Abbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236
“What is considered a good explanation depends on the clinical context, the user’s expertise, and the decision at hand [29,30].”
Machucho-Cadena, R., & González, O. (2025) · Systems · doi:10.3390/systems13040264 · Cited by 142
Interpretable and explainable predictive machine learning models for data-driven protein engineering
Medina-Ortiz, D., Khalifeh, A., Anvari-Kazemabad, H., & Davari, M. D. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108495 · Cited by 63
Muñoz-Ordóñez, J. F., Cobos, C., Vidal-Rojas, J.-C., & Herrera, F. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/4934696 · Cited by 15
Compressive Strength of Geopolymer Concrete Prediction Using Machine Learning Methods
Stel’makh, S. A., Beskopylny, A. N., Shcherban’, E. M., Razveeva, I., Oganesyan, S., Shakhalieva, D. M., Chernil'nik, A. A., & Onore, G. (2025) · Algorithms · doi:10.3390/a18120744 · Cited by 12
Metadata
- Authors
- Mirzaei, S., Mao, H., Al-Nima, R. R. O., & Woo, W. L.
- Affiliations
- Northumbria University (GB)
- Northern Technical University (IQ)
- Corresponding author
- Woo, Wai Lok
- Venue
- Information
- Publisher
- MDPI AG
- Published
- 2023-12-20
- Volume
- 15
- Issue
- 1
- Pages
- 4
- ISSN
- 2078-2489 (ISSN-L)
- OpenAlex
- W4389994734
- Semantic Scholar
- 9132bec4065348584e0fc3ea944ca4c217868030
- Cited by
- 25
- References
- 54
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 7
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Information (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Northumbria University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- black-box models, tabular data, lime, SHAP, anchors, TabNet, TabPFN, faithfulness, fidelity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.