What Does Explainable AI Really Mean? A New Conceptualization of Perspectives
Open accessAbstract
We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The definition of XAI given by D. Gunning misses to consider other purposes motivating the need for interpretable AI models, such as causality, transferability, informativeness, fairness and confidence.
“However, it misses to consider other purposes motivating the need for interpretable AI models, such as causality, transferability, informativeness, fairness and confidence [5, 24, 25, 26].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [24]
Scarce contributions have attempted to define XAI goals from a conceptual perspective.
“Unfortunately, scarce contributions have attempted to define such goals from a conceptual perspective [5, 13, 24, 30].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [24]
A truly explainable model should not leave explanation generation to the users as different explanations may be deduced depending on their background knowledge.
“As stated in [24], a truly explainable model should not leave explanation generation to the users as different explanations may be deduced depending on their background knowledge.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [24]
Sandboxes have already been used in other contexts to test and validate Fintech or Blockchain technologies in the European Union.
“They have already been used in other contexts to test and validate Fintech [130] or Blockchain 17 technologies in the European Union.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [17]
Cites
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · see citing works
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · see citing works
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Explainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147 · see citing works
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800 · see citing works
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636 · see citing works
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652 · see citing works
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · see citing works
Artificial intelligence and knowledge management: A partnership between human and AI
Jarrahi, M. H., Askay, D., Eshraghi, A., & Smith, P. (2023) · Business Horizons · doi:10.1016/j.bushor.2022.03.002 · Cited by 604 · see citing works
Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them
Santoni de Sio, F., & Mecacci, G. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00450-x · Cited by 490 · see citing works
Solving the Black Box Problem: A Normative Framework for Explainable Artificial Intelligence
Zednik, C. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00382-7 · Cited by 419 · see citing works
Artificial Intelligence and Patient-Centered Decision-Making
Bjerring, J. C., & Busch, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00391-6 · Cited by 274 · see citing works
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302 · see citing works
Weber, P., Carl, K. V., & Hinz, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-023-00320-0 · Cited by 292 · see citing works
“…2019) or Finance (e.g., Cao 2020, 2022; Goodell et al. 2021; Zheng et al. 2019) or XAI in general (e.g., Arrieta et al. 2019; Ciatto et al. 2020; Doran et al. 2017; Rosenfeld and Richardson 2019; Sanneman and Shah 2020; Verhagen et al. 2021), so far, there is no paper reviewing the current…”
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283 · see citing works
“…as: (1) Opaque, where the input to output mappings are invisible to the user, (2) Interpretable system, where a user can both see and understand how the inputs are mapped to output, and (3) Comprehensible systems, that provides some additional output, such as visualizations, text etc. [44].”
BackgroundMueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215 · see citing works
Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187 · see citing works
Walsh, C. G., Chaudhry, B., Dua, P., Goodman, K. W., Kaplan, B., Kavuluru, R., Solomonides, A., & Subbian, V. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooz054 · Cited by 118 · see citing works
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Towards platform observability
Rieder, B., & Hofmann, J. (2020) · Internet Policy Review · doi:10.14763/2020.4.1535 · Cited by 132 · see citing works
AI explainability and governance in smart energy systems: A review
Alsaigh, R., Mehmood, R., & Katib, I. (2023) · Frontiers in Energy Research · doi:10.3389/fenrg.2023.1071291 · Cited by 121 · see citing works
Futia, G., & Vetrò, A. (2020) · Information · doi:10.3390/info11020122 · Cited by 94 · see citing works
Le, T.-N., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00996 · Cited by 97 · see citing works
Neural-symbolic integration and the Semantic Web
Hitzler, P., Bianchi, F., Ebrahimi, M., & Sarker, M. K. (2020) · Semantic Web · doi:10.3233/sw-190368 · Cited by 70 · see citing works
Varanasi, R. A., & Goyal, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580903 · Cited by 52 · see citing works
Ackerhans, S., Wehkamp, K., Petzina, R., Dumitrescu, D., & Schultz, C. (2025) · JMIR Formative Research · doi:10.2196/64266 · Cited by 38 · see citing works
Jaigirdar, F. T., Rudolph, C., Oliver, G., Watts, D., & Bain, C. (2020) · 2020 IEEE 6th International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic50333.2020.00030 · Cited by 18 · see citing works
Porębski, A. (2023) · Research Handbook on Law and Technology · doi:10.4337/9781803921327.00037 · Cited by 4 · see citing works
- Authors
- Doran, D., Schulz, S., & Besold, T. R.
- ORCID
- Besold, Tarek R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1710.00794
- OpenAlex
- W2762968537
- Cited by
- 333
- References
- 2
- Citations by year
- 2026: 5 · 2025: 17 · 2024: 36 · 2023: 57 · 2022: 50 · 2021: 72 · 2020: 47 · 2019: 30 · 2018: 14 · 2017: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- explainable AI, opacity, interpretability, comprehensibility, automated reasoning, explanation generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
