DARPA's Explainable Artificial Intelligence Program
Open accessAbstract
Dramatic success in machine learning has led to a new wave of AI applications (for example, transportation, security, medicine, finance, defense) that offer tremendous benefits but cannot explain their decisions and actions to human users. DARPA's explainable artificial intelligence (XAI) program endeavors to create AI systems whose learned models and decisions can be understood and appropriately trusted by end users. Realizing this goal requires methods for learning more explainable models, designing effective explanation interfaces, and understanding the psychologic requirements for effective explanations. The XAI developer teams are addressing the first two challenges by creating ML techniques and developing principles, strategies, and human‐computer interaction techniques for generating effective explanations. Another XAI team is addressing the third challenge by summarizing, extending, and applying psychologic theories of explanation to help the XAI evaluator define a suitable evaluation framework, which the developer teams will use to test their systems. The XAI teams completed the first of this 4‐year program in May 2018. In a series of ongoing evaluations, the developer teams are assessing how well their XAM systems' explanations improve user understanding, user trust, and user task performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous works [29, 53, 50] have studied the problem of explainability and transparency from philosophy, cognitive psychology, and social psychology.
“In terms of specific dimensions of trustworthy AI, previous works such as [29, 53, 50] have studied the problem of explainability and transparency from philosophy, cognitive psychology, and social psychology.”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [29]
The Abstract identifies cognitive psychology as a perspective on explainability research. The “Naturalistic Decision-Making Foundations of XAI” section describes drawing on philosophy of science and specializations within psychology to assess explanations, but our copy does not specifically identify social psychology or establish all three perspectives for this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredApplying XAI concepts may require choosing between model interpretability and model performance.
“While applying XAI concepts to models, we may have to choose between model interpretability and model performance [26].”
From Explainable AI (XAI): Core Ideas, Techniques, and Solutions (2023) · cites this paper as [26]
The DARPA 4-year project requested technical teams to provide an explainable model and explanation interface to support users making decisions.
“Similar to the 4-year project conducted by DARPA (Gunning & Aha, 2019), the technical teams were requested to provide an explainable model and explanation interface to support users making decisions.”
From Artificial intelligence in intelligent tutoring systems toward sustainable education: a systematic review (2023) · cites this paper as Gunning & Aha, 2019
People often ignore AI recommendations because they do not trust them.
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [9]
Understandability is important to calibrate trust and reliance, and to find mistakes.
“This is a common criteria for AI-based DAs [9], and is important to calibrate trust & reliance, and to find mistakes.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [9]
Issues such as warranted and unwarranted distrust mean that AI systems are often deployed and then largely ignored.
“However, issues such as warranted and unwarranted distrust [13] mean that AI systems are often deployed and then largely ignored [9,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [9]
Humans can understand and describe how an AI system reached a decision using Explainable Artificial Intelligence (XAI).
“Humans can understand and describe how an AI system reached a decision using Explainable Artificial Intelligence (XAI) [212].”
From Federated learning-based AI approaches in smart healthcare: Concepts, taxonomies, challenges and open issues (2023) · cites this paper as [212]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · see citing works
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
Heafield, K. (2016) · The Association for Computational Linguistics · Cited by 718 · see citing works
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 523 · see citing works
Interpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M. B., Preece, A. D., Julier, S., Rao, R., et al. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 469 · see citing works
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418 · see citing works
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255 · see citing works
MovieGraphs: Towards Understanding Human-Centric Situations from Videos
Vicol, P. A., Tapaswi, M., Castrejón, L., & Fidler, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00895 · Cited by 130 · see citing works
Towards Explanation of DNN-based Prediction with Guided Feature Inversion
Du, M., Liu, N., Song, Q., & Hu, X. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220099 · Cited by 123 · see citing works
Practical Probabilistic Programming
Pfeffer, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21295-6_2 · Cited by 122 · see citing works
Establishing Appropriate Trust via Critical States
Huang, S. H., Bhatia, K., Abbeel, P., & Dragan, A. D. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593649 · Cited by 94 · see citing works
Explaining Explanation, Part 1: Theoretical Foundations
Hoffman, R. R., & Klein, G. A. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.54 · Cited by 83 · see citing works
Explaining Explanation, Part 4: A Deep Dive on Deep Nets
Hoffman, R. R., Miller, T., Mueller, S. T., Klein, G. A., & Clancey, W. J. (2018) · IEEE Intelligent Systems · doi:10.1109/mis.2018.033001421 · Cited by 54 · see citing works
Explaining Explanation, Part 2: Empirical Foundations
Hoffman, R. R., Mueller, S. T., & Klein, G. A. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.3121544 · Cited by 53 · see citing works
Causal Learning and Explanation of Deep Neural Networks via Autoencoded ActivationsPreprint
Harradon, M., Druce, J., & Ruttenberg, B. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00541 · Cited by 47 · see citing works
An Interpretable Classification Framework for Information Extraction from Online Healthcare Forums
Gao, J. T., Liu, N., Lawley, M., & Hu, X. (2017) · Journal of Healthcare Engineering · doi:10.1155/2017/2460174 · Cited by 36 · see citing works
Dodge, J. E., Penney, S., Hilderbrand, C., Anderson, A. W., & Burnett, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174136 · Cited by 31 · see citing works
Explaining Explanation, Part 3: The Causal Landscape
Klein, G. A. (2018) · IEEE Intelligent Systems · doi:10.1109/mis.2018.022441353 · Cited by 30 · see citing works
Gogate, V., & Domingos, P. (2016) · Communications of the ACM · doi:10.1145/2936726 · Cited by 28 · see citing works
Recurrent Predictive State Policy NetworksPreprint
Hefny, A., Marinho, Z., Sun, W. Y., Srinivasa, S. S., & Gordon, G. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01489 · Cited by 12 · see citing works
Bayesian Teaching of Image Categories.
Vong, W. K., Sojitra, R. B., Reyes, A., et al. (2018) · eScholarship (California Digital Library) · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Belbute-Peres, F., and Kolter, J. Z. 2017. A Modular Differ- entiable Rigid Body Physics Engine. Paper presented at the Neural Information Processing Systems Deep Reinforcement Learning Symposium. Long Beach, CA, December 7.
- Hendricks, L. A.; Hu, R.; Darrell, T.; and Akata, Z. 2018. Grounding Visual Explanations. Presented at the European Conference of Computer Vision (ECCV). Munich, Germany; September 8-14. doi.org/10.1007/978-3-030-01216-8_17 · DOI · Paper page
- Kim, J., and Canny, J. 2017. Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention. In Proceedings of the International Conference on Computer Vi- sion, 2942-50. New York: IEEE. doi.org/10.1109/ICCV. 2017.320 · DOI
- Qi, Z., and Li, F. 2017. Learning Explainable Embeddings for Deep Networks. Paper presented at the NIPS Workshop on Interpreting, Explaining and Visualizing Deep Learning.
- Ramanishka, V.; Das, A.; Zhang, J.; and Saenko, K. 2017. Top- Down Visual Saliency Guided by Captions. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, 7206-15. New York, IEEE.
- Yang, S. C.-H., and Shafto, P. 2017. Explainable Artificial In- telligence via Bayesian Teaching. Paper presented at the 31st Conference on Neural Information Processing Systems Workshop on Teaching Machines, Robots and Humans. Long Beach, CA, December 9.
- David W. Aha is acting director of NRL's Navy Center for Applied Research in AI in Washington, D.C. His interests include goal reasoning, XAI, case-based reasoning, and machine learning, among other topics. He has coorganized many events on these topics (for example, the IJCAI-18 XAI Workshop), launched the UCI Repository for ML Databases, served as an AAAI Councilor, and leads DARPA XAI's eval- uation team.
Cited by
Sorted by citations
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · see citing works
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386 · see citing works
Chowdhury, S., Dey, P., Joel-Edgar, S., Bhattacharya, S., Rodriguez-Espindola, O., Abadie, A., & Truong, L. (2023) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100899 · Cited by 927 · see citing works
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716 · see citing works
Lin, C.-C., Huang, A. Y. Q., & Lu, O. H. T. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00260-y · Cited by 669 · see citing works
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“To promote the research on AI trustworthiness, stakeholders are actively setting up various forms of collaboration such as research workshops on trustworthy AI and cooperative projects like DARPA XAI [162].”
BackgroundInfluentialA., S., & R., S. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100230 · Cited by 432 · see citing works
Rahman, A., Hossain, M. S., Muhammad, G., Kundu, D., Debnath, T., Rahman, M., Khan, M. S. I., Tiwari, P., & Band, S. S. (2023) · Cluster Computing · doi:10.1007/s10586-022-03658-4 · Cited by 399 · see citing works
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312 · see citing works
Trust and medical AI: the challenges we face and the expertise needed to overcome them
Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa268 · Cited by 287 · see citing works
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283 · see citing works
“2, where non-linear models with better performance, such as DNNs are less explainable, and linear models which are least accurate, such as decision trees, are more explainable [2,21,45].”
BackgroundMethodInfluentialRadiology artificial intelligence: a systematic review and evaluation of methods (RAISE)
Kelly, B. S., Judge, C., Bollard, S. M., Clifford, S. M., Healy, G. M., Aziz, A., Mathur, P., Islam, S., Yeom, K. W., Lawlor, A., et al. (2022) · European Radiology · doi:10.1007/s00330-022-08784-6 · Cited by 252 · see citing works
Yigitcanlar, T., Corchado, J. M., Mehmood, R., Li, R. Y. M., Mossberger, K., & Desouza, K. (2021) · Journal of Open Innovation: Technology, Market, and Complexity · doi:10.3390/joitmc7010071 · Cited by 245 · see citing works
A survey of visual analytics for Explainable Artificial Intelligence methods
Alicioglu, G., & Sun, B. (2022) · Computers & Graphics · doi:10.1016/j.cag.2021.09.002 · Cited by 221 · see citing works
Hybrid physics-informed neural networks for lithium-ion battery modeling and prognosis
Nascimento, R. G., Corbetta, M., Kulkarni, C. S., & Viana, F. A. C. (2021) · Journal of Power Sources · doi:10.1016/j.jpowsour.2021.230526 · Cited by 224 · see citing works
Kenny, E. M., Ford, C., Quinn, M., & Keane, M. T. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103459 · Cited by 211 · see citing works
Kute, D. V., Pradhan, B., Shukla, N., & Alamri, A. (2021) · IEEE Access · doi:10.1109/access.2021.3086230 · Cited by 209 · see citing works
Sokol, K., & Flach, P. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00637-y · Cited by 186 · see citing works
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172 · see citing works
Explainable AI for 6G Use Cases: Technical Aspects and Research Challenges
Wang, S., Qureshi, M. A., Miralles-Pechuán, L., Huynh-The, T., Gadekallu, T. R., & Liyanage, M. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3386872 · Cited by 170 · see citing works
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167 · see citing works
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · see citing works
“To stimulate the development of XAI techniques, the Defense Advanced Research Projects Agency (DARPA) launched an XAI program in 2017 [223], [224], the goal of which is to develop new techniques capable of making intelligent systems explainable.”
MethodXAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Talaat, F. M., Aljadani, A., Badawy, M., & Elhosseini, M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-023-09232-2 · Cited by 105 · see citing works
Enhancing Explainability of Neural Networks Through Architecture Constraints
Yang, Z., Zhang, A., & Sudjianto, A. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3007259 · Cited by 106 · see citing works
Explainability and causability in digital pathology
Plass, M., Kargl, M., Kiehl, T., Regitnig, P., Geißler, C., Evans, T., Zerbe, N., Carvalho, R., Holzinger, A., & Müller, H. (2023) · The Journal of Pathology Clinical Research · doi:10.1002/cjp2.322 · Cited by 99 · see citing works
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · see citing works
Explainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87 · see citing works
Jung, J., Lee, H., Jung, H., & Kim, H. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16110 · Cited by 87 · see citing works
Kleizen, B., Van Dooren, W., Verhoest, K., & Tan, E. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101834 · Cited by 80 · see citing works
Hsu, C.-Y., & Li, W. (2023) · International Journal of Geographical Information Science · doi:10.1080/13658816.2023.2191256 · Cited by 75 · see citing works
HIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72 · see citing works
Towards relatable explainable AI with the perceptual process
Zhang, W., & Lim, B. Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501826 · Cited by 67 · see citing works
Abas Mohamed, Y., Ee Khoo, B., Shahrimie Mohd Asaari, M., Ezane Aziz, M., & Rahiman Ghazali, F. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105689 · Cited by 57 · see citing works
Leveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52 · see citing works
Artificial Intelligence in Cardiovascular Imaging: “Unexplainable” Legal and Ethical Challenges?
Lang, M., Bernier, A., & Knoppers, B. M. (2022) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2021.10.009 · Cited by 37 · see citing works
Müller, H., & Holzinger, A. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103546 · Cited by 30 · see citing works
Toward Accountable and Explainable Artificial Intelligence Part Two: The Framework Implementation
Vice, J., & Khan, M. M. (2022) · IEEE Access · doi:10.1109/access.2022.3163523 · Cited by 11 · see citing works
L'acte administratif algorithmique
Stepanov, A. (2026) · doi:10.70675/adc05be3za62ez4b4fz81e7zabf744e60dbf
- Authors
- Gunning, D., & Aha, D. W.
- Affiliations
- United States Naval Research Laboratory (US)
- ORCID
- Gunning, David
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2019-06
- Volume
- 40
- Issue
- 2
- Pages
- 44-58
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W2954503794
- Cited by
- 1,360
- References
- 24
- Citations by year
- 2026: 250 · 2025: 288 · 2024: 238 · 2023: 178 · 2022: 154 · 2021: 148 · 2020: 96 · 2019: 8
- FWCI
- 46.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- AI Magazine (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- explainable AI, human-computer interaction, user trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
