Explainable goal-driven agents and robots-a comprehensive review

Sado, Fatai; Loo, Chu Kiong; Liew, Wei Shiung; Kerzel, Matthias; Wermter, Stefan

doi:10.1145/3564240

Cited by 93Explainable Artificial Intelligence (XAI)Multi-Agent Systems and NegotiationAI-based Problem Solving and Planning

Explainable goal-driven agents and robots-a comprehensive review

Open access

Sado, Fatai, Loo, Chu Kiong, Liew, Wei Shiung, Kerzel, Matthias, Wermter, Stefan·2023·ACM Computing Surveys

DOI 10.1145/3564240

Open this paper in CoChat — free account Open PDF

Abstract

Recent applications of autonomous agents and robots have brought attention to crucial trust-related challenges associated with the current generation of artificial intelligence (AI) systems. AI systems based on the connectionist deep learning neural network approach lack capabilities of explaining their decisions and actions to others, despite their great successes. Without symbolic interpretation capabilities, they are ‘black boxes’, which renders their choices or actions opaque, making it difficult to trust them in safety-critical applications. The recent stance on the explainability of AI systems has witnessed several approaches to eXplainable Artificial Intelligence (XAI) ; however, most of the studies have focused on data-driven XAI systems applied in computational sciences. Studies addressing the increasingly pervasive goal-driven agents and robots are sparse at this point in time. This paper reviews approaches on explainable goal-driven intelligent agents and robots, focusing on techniques for explaining and communicating agents’ perceptual functions (e.g., senses, vision) and cognitive reasoning (e.g., beliefs, desires, intentions, plans, and goals) with humans in the loop. The review highlights key strategies that emphasize transparency, understandability, and continual learning for explainability. Finally, the paper presents requirements for explainability and suggests a road map for the possible realization of effective goal-driven explainable agents and robots.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1145/3564240

Cites

187 of 187 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

AI Agents are modular systems driven and enabled by LLMs and LIMs for task-specific automation.
“characterizing AI Agents as modular systems driven and enabled by LLMs and LIMs for taskspecific automation”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [47]
The Introduction describes goal-driven agents and robots as autonomous systems that accomplish goals, and §3.1.8 describes an agent architecture with four submodules. These passages support the task-oriented and modular aspects, but the paper does not characterize agents as driven or enabled by LLMs and LIMs. (LLM, conf=0.83)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · cited 2× · see citing works
“A Region Proposal Network (RPN) [175] is used to generate a collection of object proposals from an image and transfers to the graph-inference module the extracted features of object regions.”
Method
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2× · see citing works
“Lundberg and Lee [141] utilize SHapley Additive exPlanation (SHAP) values, an extension of Shapley values, as a united strategy to explain any machine learning model output.”
Method
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1× · see citing works
“CAMs [230] address this issue by creating a heatmap of discriminative image regions over the input image that shows what parts of the input image contributed how much to the CNN’s classification of the image as belonging to a selected class.”
Background
The self-organizing map
Kohonen, T. (1990) · Neurocomputing · doi:10.1016/s0925-2312(98)00030-7 · Cited by 10,686 · cited 2× · see citing works
“, learning from demonstration [209]), unsupervised learning [115], and reinforcement learning.”
Background
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909 · cited 1× · see citing works
“[186] based on gradients flowing into the final convolutional layer to solve this problem.”
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes, R. E., & Nilsson, N. J. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90010-5 · Cited by 4,882 · cited 2× · see citing works
“The model employs communicable predicates for natural language communication, i.e., Boolean classiiers similar to traditional STRIPS-style [45] planning predicates with associated natural language descriptions, to translate attributes from sets of states to natural language and back (Fig.”
Background
The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Intention, Plans, and Practical Reason
Mccann, H., & Bratman, M. (1991) · Noûs · doi:10.2307/2215590 · Cited by 3,083 · cited 3× · see citing works
“According to Bratman [18], belief and desire are mental attitudes (pro-attitudes) driving action.”
Background
Case-Based Reasoning
Kolodner, J. (1988) · International Joint Conference on Artificial Intelligence · doi:10.1016/c2009-0-27670-7 · Cited by 2,964 · cited 1×
“In the CBR framework, agent learning to expand its knowledge is executed by evaluating and integrating new experiences into the case-library and/or re-indexing and reusing previous experiences [116].”
Background
An Assumption-Based TMS
de Kleer, J. (1987) · Artificial Intelligence · doi:10.1016/0004-3702(86)90080-9 · Cited by 2,034 · see citing works
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 2× · see citing works
“Previous reviews on reactive RL agents can be found in Lin [135].”
Background
The intelligent ASIMO: system overview and integration
Sakagami, Y., Watanabe, R., Aoyama, C., Matsunaga, S., Higaki, N., & Fujimura, K. (2002) · IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/irds.2002.1041641 · Cited by 1,554 · cited 2× · see citing works
“Other similar robots are Honda Asimo [118] which uses a commercial hardware electronic system for speech synthesis, and ASKA [107] receptionist robots, with a conversational speech dialogue system that can recognize users’ questions and answer the users’ questions by a text-to-speech voice processing with other additi…”
Method
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271 · cited 1× · see citing works
“[224] to generate structured scene representations from input images.”
Background
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225 · cited 1× · see citing works
“[7] classify rule extraction algorithms into three categories.”
Method
Explanation and understanding
Sora, A. (2015) · doi:10.1057/9781137372406.0008 · Cited by 1,220 · cited 2×
“The motivation here is that humans can clarify and comprehend their behaviors or actions in terms of underlying beliefs, desires, intentions, or goals [70] [95].”
Background
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101 · see citing works
RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism
Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1608.05745 · Cited by 1,029 · cited 2× · see citing works
“One solution to improve interpretability or visualization was introducing an intermediate attention layer [40, 188].”
Method
Graph R-CNN for Scene Graph GenerationPreprint
Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018) · European Conference on Computer Vision · doi:10.48550/arxiv.1808.00191 · Cited by 977 · cited 1× · see citing works
“[226] present Graph R-CNN, a model for constructing scene graphs that can also perform object detection and infer object relations in images.”
Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions
Dale, R., & Reiter, E. (1995) · Cognitive Sciences · doi:10.1207/s15516709cog1902_3 · Cited by 931 · cited 1× · see citing works
“The robot’s REG module then uses situated references to clarify to users the mistakes and suggest strategies for solutions [50].”
Method
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · cited 2× · see citing works
“Recurrent neural networks (RNNs) are a popular deep learning framework for temporal modeling, particularly in medical applications for modeling disease progression [62] and predicting diagnoses based on patient records [136].”
BackgroundMethod
Social Robots for Long-Term Interaction: A Survey
Leite, I., Martinho, C., & Paiva, A. (2013) · International Journal of Social Robotics · doi:10.1007/s12369-013-0178-y · Cited by 871 · cited 1× · see citing works
“[129] for a more comprehensive review of social robots.”
Background
What can the brain teach us about building artificial intelligence?
Botvinick, M., Barrett, D. G., Battaglia, P., de Freitas, N., Kumaran, D., Leibo, J., Lillicrap, T., Modayil, J., Mohamed, S., Rabinowitz, N., et al. (2016) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17000140 · Cited by 794 · cited 1× · see citing works
“They can also adapt the intentions or purposes as the need arises and may be required to account for their actions [31, 124] and embody such capabilities in the context of lifelong developmental learning [218].”
Background
Limits on theory of mind use in adults
Keysar, B., Lin, S., & Barr, D. J. (2003) · Cognition · doi:10.1016/s0010-0277(03)00064-7 · Cited by 791 · cited 1× · see citing works
“People may make false assumptions about others’ experiences and motives in many crucial circumstances [65], a phenomenon of attributing incorrect mental states to others [110].”
Background
Inductive Logic Programming
Zaverucha, G., Costa, V. S., & Paes, A. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-662-44923-3 · Cited by 745 · cited 1× · see citing works
“A method of inductive logic programming (ILP) is employed to learn the relational representations of the tool models [127].”
Method
The Belief-Desire-Intention Model of Agency
Georgeff, M., Pell, B., Pollack, M., Tambe, M., & Wooldridge, M. (1998) · ATAL · doi:10.1007/3-540-49057-4_1 · Cited by 731 · see citing works
Two neural correlates of consciousness
Block, N. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.12.006 · Cited by 713 · cited 1× · see citing works
“The idea behind the rationale generation is that people can communicate effectively by verbalizing reasonable motives for their actions, even though the verbalized reasoning is not aligned with the decision-making neural processes of the human brain [19].”
Background
Effects of nonverbal communication on efficiency and robustness in human-robot teamwork
Breazeal, C., Kidd, C. D., Thomaz, A., Hoffman, G., & Berlin, M. (2005) · 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2005.1545011 · Cited by 663 · cited 1× · see citing works
“, transparency) of XGDAI was shown to enhance the quality of task performance [26] and to enable”
Background
Handbook of Natural Language Processing
Dale, R., Somers, H., & Moisl, H. (2001) · Computational Linguistics · doi:10.1162/coli.2000.27.4.602 · Cited by 636 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1× · see citing works
“Thus, data-driven XAI implies explaining the decision made by a machine learning “blackbox” system, given input data [81].”
Method
Social eye gaze in human-robot interaction
Admoni, H., & Scassellati, B. (2017) · J. Hum. Robot Interact. · doi:10.5898/jhri.6.1.admoni · Cited by 587 · see citing works
How People Explain Behavior: A New Theoretical Framework
Malle, B. F. (1999) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0301_2 · Cited by 538 · cited 1× · see citing works
“The motivation here is that humans can clarify and comprehend their behaviors or actions in terms of underlying beliefs, desires, intentions, or goals [70] [95].”
Background
Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks
Giles, C. L., Miller, C. B., Chen, D., Chen, H.-H., Sun, G.-Z., & Lee, Y.-C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.393 · Cited by 513 · cited 1× · see citing works
“For example, using quantization algorithms [75], clustering outputs to infer FSAs [162, 229], and recursively testing inputs and outputs [215].”
Method
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504 · cited 1× · see citing works
“From the training data, CNNs learn to extract a deep hierarchy of task-relevant visual features [10, 52, 117, 204].”
Method
Explainable artificial intelligence model to predict acute critical illness from electronic health records
Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448 · see citing works
Explainable Reinforcement Learning Through a Causal Lens
Madumal, P., Miller, T., Sonenberg, L., & Vetere, F. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1905.10958 · Cited by 445 · cited 2×
“• Game applications used explainability techniques to describe gameplay behavior, whether of non-player characters (NPCs) [102] or player characters [94].”
Background
Theory and evaluation of human robot interactions
Scholtz, J. (2003) · 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the · doi:10.1109/hicss.2003.1174284 · Cited by 428 · cited 2× · see citing works
“Humans, on the other hand, play the role of supervisor, operator, mechanic, bystander, or teammate [184].”
Background
Preparing for the future of Artificial Intelligence
Bundy, A. (2017) · AI & SOCIETY · doi:10.1007/s00146-016-0685-0 · Cited by 425 · cited 2× · see citing works
“Although the need for explainability in AI systems has been long established during the MYCIN era, also known as the era of the expert systems [167, 168], the current drive for explainability has been motivated by recent governmental efforts from the European Union, United States (USA) [29], and China [150], which hav…”
Background
Transparent, explainable, and accountable AI for robotics
Wachter, S., Mittelstadt, B., & Floridi, L. (2017) · Science Robotics · doi:10.1126/scirobotics.aan6080 · Cited by 413 · cited 3× · see citing works
“Although the need for explainability in AI systems has been long established during the MYCIN era, also known as the era of the expert systems [167, 168], the current drive for explainability has been motivated by recent governmental efforts from the European Union, United States (USA) [29], and China [150], which hav…”
Background
Explainable Agents and Robots: Results from a Systematic Literature Review
Anjomshoae, S., Najjar, A., Calvaresi, D., & Främling, K. (2019) · Adaptive Agents and Multi-Agent Systems · doi:10.65109/kczb5817 · Cited by 402 · cited 8× · see citing works
“This upsurge can be seen as the effect of the public pressure on the explainability of AI systems and initiatives by several national government agencies like the “right to explanation” by the GDPR [8, 31].”
BackgroundInfluential
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Preprint
Hase, P., & Bansal, M. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2005.01831 · Cited by 396 · cited 1× · see citing works
“Hase and Bansal [58] study how techniques used for generating explanations afect the users’ ability to predict what the explaining agent will do in a given situation.”
Method
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · see citing works
Computational Generation of Referring Expressions: A Survey
Krahmer, E., & van Deemter, K. (2012) · International Conference on Computational Logic · doi:10.1162/coli_a_00088 · Cited by 384 · cited 1× · see citing works
“Many scholarly works on REG can be found spanning the last five decades [119].”
Background
Improving Robot Controller Transparency Through Autonomous Policy Explanation
Hayes, B., & Shah, J. A. (2017) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020233 · Cited by 359 · cited 8× · see citing works
“APE [88] is one example of a Reactive technique used in robot-human collaboration, while PeCoX [157] is considered a Hybrid technique.”
MethodInfluential
Plan Explanations as Model Reconciliation: Moving Beyond Explanation as Soliloquy
Chakraborti, T., Sreedharan, S., Zhang, Y., & Kambhampati, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/23 · Cited by 359 · see citing works
Effects of Robot Motion on Human-Robot Collaboration
Dragan, A., Bauman, S., Forlizzi, J., & Srinivasa, S. S. (2015) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696473 · Cited by 354 · see citing works
Situation awareness-based agent transparency and human-autonomy teaming effectiveness
Chen, J. Y. C., Lakhmani, S. G., Stowers, K., Selkowitz, A. R., Wright, J. L., & Barnes, M. (2018) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2017.1315750 · Cited by 341 · see citing works
Expressing thought: Improving robot readability with animation principles
Takayama, L., Dooley, D., & Ju, W. (2011) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/1957656.1957674 · Cited by 335 · see citing works
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Yu, L., Tan, H., Bansal, M., & Berg, T. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.375 · Cited by 297 · cited 2× · see citing works
“Referring expression is a language construction or expression used to identify particular objects within a scene [228].”
Background
The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine
Mirchi, N., Bissonnette, V., Yilmaz, R., Ledwos, N., Winkler-Schwartz, A., & Del Maestro, R. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229596 · Cited by 296 · see citing works
Visual Analytics for Explainable Deep LearningPreprint
Choo, J., & Liu, S.-X. (2018) · IEEE Computer Graphics and Applications · doi:10.48550/arxiv.1804.02527 · Cited by 273 · cited 3× · see citing works
“Choo and Liu [28] describe a deep learning neural network’s (DNN) interpretability as determining the input attributes that account for output predictions.”
Background
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254 · see citing works
Extraction of rules from discrete-time recurrent neural networks
Omlin, C., & Giles, C. L. (1996) · Neural Networks · doi:10.1016/0893-6080(95)00086-0 · Cited by 251 · cited 5× · see citing works
“Finite-state machines or automata (FSAs) are computational models that assume one of any number of inite states at any time and transition to other states in response to particular inputs.”
MethodInfluential
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
Ehsan, U., Harrison, B., Chan, L., & Riedl, M. O. (2017) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278736 · Cited by 225 · cited 1× · see citing works
“ARGs are expected to generate explanations or rationales in the natural language for the agent’s behavior as if a person had performed such behavior [58].”
Background
Who should i blame? effects of autonomy and transparency on attributions in human-robot interaction
Kim, T., & Hinds, P. (2006) · ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2006.314398 · Cited by 217 · see citing works
Extracting Automata from Recurrent Neural Networks Using Queries and CounterexamplesPreprint
Weiss, G., Goldberg, Y., & Yahav, E. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1711.09576 · Cited by 216 · cited 1× · see citing works
“For example, using quantization algorithms [75], clustering outputs to infer FSAs [162, 229], and recursively testing inputs and outputs [215].”
Method
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210 · cited 13× · see citing works
“ARG [59], on the other hand, is a Reactive technique for producing human-like rationalization for agents playing a game.”
BackgroundMethodInfluential
Explainable Agency for Intelligent Autonomous Systems
Langley, P., Meadows, B., Sridharan, M., & Choi, D. (2017) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.19108 · Cited by 202 · cited 1×
“As discussed in the next subsection, there is a growing application of goal-driven XAI in the current AI-dependent world [125].”
Background
Situation Awareness-Based Agent Transparency
Chen, J. Y. C., Procci, K., Boyce, M. W., Wright, J. L., Garcia, A., & Barnes, M. (2014) · doi:10.21236/ada600351 · Cited by 201 · cited 1× · see citing works
“SAT is an agent transparency model for enhancing an operator’s situational awareness [60] in the agent’s environment [23, 39].”
Automated Planning
Chowdhary, P. K. R. (2008) · Handbook of Knowledge Representation · doi:10.1007/978-81-322-3972-7_15 · Cited by 193 · cited 1×
“The model extends the Nau [104] online planning model.”
Method
Recommendation quality, transparency, and website quality for trust-building in recommendation agents
Nilashi, M., Jannach, D., bin Ibrahim, O., Esfahani, M. D., & Ahmadi, H. (2016) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2016.09.003 · Cited by 190 · see citing works
Explaining Action by Emotion
Döring, S. (2003) · The Philosophical Quarterly · doi:10.1111/1467-9213.00307 · Cited by 188 · cited 1× · see citing works
“, ’I expect (emotion) you would exercise regularly since I want (goal) you to be physically it, and I suspect (belief) you are not currently exercising adequately’ [38] [56].”
Background
A conversational robot in an elderly care center: an ethnographic study
Sabelli, A. M., Kanda, T., & Hagita, N. (2011) · Proceedings of the 6th international conference on Human-robot interaction · doi:10.1145/1957656.1957669 · Cited by 187 · cited 2× · see citing works
“Robovie [178] is another example in this category that can communicate in English using a vocabulary of about 300 sentences for speaking and 50 words for recognition.”
Background
Interpretable Representation Learning for Healthcare via Capturing Disease Progression through Time
Bai, T., Zhang, S., Egleston, B. L., & Vucetic, S. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219904 · Cited by 180 · cited 1× · see citing works
“For example, the Timeline model [12] aggregates contextual information of patient medical codes using an attention mechanism to predict future diagnoses of future medical visits.”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning
Garcez, A. D., Gori, M., Lamb, L. C., Serafini, L., Spranger, M., & Tran, S. N. (2019) · arXiv · doi:10.48550/arxiv.1905.06088 · Cited by 175 · see citing works
Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks
Esteban, C., Staeck, O., Baier, S., Yang, Y., & Tresp, V. (2016) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi.2016.16 · Cited by 172 · cited 1× · see citing works
“Recurrent neural networks (RNNs) are a popular deep learning framework for temporal modeling, particularly in medical applications for modeling disease progression [62] and predicting diagnoses based on patient records [136].”
Background
Theory of mind: mechanisms, methods, and new directions
Byom, L., & Mutlu, B. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00413 · Cited by 165 · cited 1× · see citing works
“ToM tends to reason about other people’s perceptions, beliefs, and goals and take them into account [30].”
Background
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163 · cited 1× · see citing works
“However, saliency maps have several disadvantages [3].”
Learning Finite State Machines With Self-Clustering Recurrent Networks
Zeng, Z., Goodman, R., & Smyth, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.6.976 · Cited by 162 · cited 6× · see citing works
“For example, using quantization algorithms [75], clustering outputs to infer FSAs [162, 229], and recursively testing inputs and outputs [215].”
MethodInfluential
Explanation in Second Generation Expert Systems
Swartout, W. R., & Moore, J. D. (1993) · Second Generation Expert Systems · doi:10.1007/978-3-642-77927-5_24 · Cited by 160 · cited 1× · see citing works
“Agents must explain the decisions they made during plan generation, stating considered alternatives, to report which actions they executed and why, explain how actual events diverged from the plan and how they adapted in response, and communicate decisions and reasons in a human-understandable way to gain the user’s t…”
Background
That's on my mind! robot to human intention communication through on-board projection on shared floor space
Chadalavada, R., Andreasson, H., Krug, R., & Lilienthal, A. (2015) · European Conference on Mobile Robots · doi:10.1109/ecmr.2015.7403771 · Cited by 160 · see citing works
Social Robots that Interact with People
Breazeal, C., Takanishi, A., & Kobayashi, T. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_59 · Cited by 142 · cited 1× · see citing works
“They were typically designed as simplistic input/output mapping speech combinations [27].”
Background
Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural NetworksPreprint
Siqueira, H., Magg, S., & Wermter, S. (2020) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2001.06338 · Cited by 141 · see citing works
Self-citation
Learning Generalized Reactive Policies using Deep Neural Networks
Groshev, E., Tamar, A., Goldstein, M., Srivastava, S., & Abbeel, P. (2017) · International Conference on Automated Planning and Scheduling · doi:10.48550/arxiv.1708.07280 · Cited by 138
Software Agents for Future Communication Systems
Hayzelden, A. L., & Bigham, J. (1999) · Springer Berlin Heidelberg · doi:10.1007/978-3-642-58418-3 · Cited by 135 · cited 1×
“The BDI model is the most widely used architecture to enforce such actions, where the beliefs of an agent about the world (its picture of the world), desires (goal), and intentions are internally expressed, and realistic reasoning is applied to select an action [90].”
Method
Goal-Driven Autonomy in a Navy Strategy Simulation
Molineaux, M., Klenk, M., & Aha, D. (2010) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7576 · Cited by 134 · cited 11× · see citing works
“ARTUE is designed to deal with unexpected environmental situations by irst explaining these changes and then developing new goals which integrate the explicit information on the hidden environmental aspects.”
BackgroundMethodInfluential
Introspective Multistrategy Learning: On the Construction of Learning Strategies
Cox, M. T., & Ram, A. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00047-8 · Cited by 134 · cited 1× · see citing works
“It adopts case-based knowledge representations as frames linked together by explanatory patterns to represent general causal structures [45, 46].”
Background
Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations
Sequeira, P., & Gervasio, M. (2019) · Artificial Intelligence · doi:10.48550/arxiv.1912.09007 · Cited by 130
A Deep Learning-Based Hybrid Framework for Object Detection and Recognition in Autonomous Driving
Li, Y.-F., Wang, H.-X., Dang, L., Nguyen, T. N., Han, D., Lee, A., Jang, I., & Moon, H. (2020) · IEEE Access · doi:10.1109/access.2020.3033289 · Cited by 128 · see citing works
Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs
Kulesza, T., Stumpf, S., Burnett, M., Wong, W.-K., Riche, Y., Moore, T., Oberst, I., Shinsel, A., & McIntosh, K. (2010) · IEEE Symposium on Visual Languages / Human-Centric Computing Languages and Environments · doi:10.1109/vlhcc.2010.15 · Cited by 124 · see citing works
Stakeholders in Explainable AI
Preece, A., Harborne, D., Braines, D., Tomsett, R., & Chakraborty, S. (2018) · arXiv · doi:10.48550/arxiv.1810.00184 · Cited by 117 · cited 1× · see citing works
“Although the need for explainability in AI systems has been long established during the MYCIN era, also known as the era of the expert systems [167, 168], the current drive for explainability has been motivated by recent governmental efforts from the European Union, United States (USA) [29], and China [150], which hav…”
Background
Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital PathologyPreprint
Holzinger, A., Malle, B., Kieseberg, P., Roth, P., Müller, H., Reihs, R., & Zatloukal, K. (2017) · arXiv.org · doi:10.48550/arxiv.1712.06657 · Cited by 113 · cited 1× · see citing works
“The motivation to discover how available data contributes to a decision is an important aspect of this branch of XAI and whether the machine learning process can reliably replicate the same decision, given similar data and specific circumstances [95].”
Background
GRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning
Santucci, V., Baldassarre, G., & Mirolli, M. (2016) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2538961 · Cited by 113 · see citing works
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object InteractionPreprint
Zhou, L., Louis, N., & Corso, J. (2018) · British Machine Vision Conference · doi:10.48550/arxiv.1805.02834 · Cited by 111 · cited 1× · see citing works
“[231] propose a weakly-supervised framework for reference-aware visual grounding for instructional videos.”
Background
The Impact of POMDP-Generated Explanations on Trust and Performance in Human-Robot Teams
Wang, N., Pynadath, D., & Hill, S. (2016) · Adaptive Agents and Multi-Agent Systems · doi:10.65109/mjoe4434 · Cited by 105 · see citing works
Perpetual Self-Aware Cognitive Agents
Cox, M. T. (2007) · The AI Magazine · doi:10.1609/aimag.v28i1.2027 · Cited by 103 · cited 3× · see citing works
“Meta-AQUA is an introspective learning system proposed by Cox [30] for a self-aware cognitive agent.”
BackgroundInfluential
Interpretable Predictions of Clinical Outcomes with An Attention-based Recurrent Neural Network
Sha, Y., & Wang, M. D. (2017) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3107411.3107445 · Cited by 102 · cited 2× · see citing works
“One solution to improve interpretability or visualization was introducing an intermediate attention layer [40, 188].”
Method
Preferred Explanations: Theory and Generation via Planning
Sohrabi, S., Baier, J. A., & McIlraith, S. A. (2011) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7845 · Cited by 102 · cited 1× · see citing works
“An aspect of planning is Plan Explanation, in which the primary objective is to enable individuals to understand the plans of the (agent) planner [16, 185, 193], including interpreting the agents’ plans into humanunderstandable form and the creation of interfaces that facilitate this understanding.”
Background
INGRESS: Interactive visual grounding of referring expressions
Shridhar, M., Mittal, D., & Hsu, D. (2020) · Int. J. Robotics Res. · doi:10.1177/0278364919897133 · Cited by 98 · cited 1× · see citing works
“In human-robot interaction (HRI) scenarios, visual grounding can enable a robot to accurately locate an object amongst other objects in the workspace [191].”
Background
GOAL‐DRIVEN AUTONOMY FOR RESPONDING TO UNEXPECTED EVENTS IN STRATEGY SIMULATIONS
Klenk, M., Molineaux, M., & Aha, D. (2013) · International Conference on Climate Informatics · doi:10.1111/j.1467-8640.2012.00445.x · Cited by 96 · see citing works
The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue
Foster, M. E., Bard, E., Guhe, M., Hill, R. L., Oberlander, J., & Knoll, A. (2008) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/1349822.1349861 · Cited by 94 · cited 2× · see citing works
“[67], in which the authors argue that a multimodal reference involving manipulation actions with the objects would provide a richer referring approach for conversational partners in situated dialogue.”
Background
Data mining using rule extraction from Kohonen self-organising maps
Malone, J., McGarry, K., Wermter, S., & Bowerman, C. (2006) · Neural computing & applications (Print) · doi:10.1007/s00521-005-0002-1 · Cited by 92 · cited 1× · see citing works
“[144] which extracts propositional “IF-THEN” type rules from self-organized mapping networks.”
BackgroundSelf-citation
Generation of Referring Expressions: Assessing the Incremental Algorithm
van Deemter, K., Gatt, A., van der Sluis, I., & Power, R. (2012) · Cognitive Sciences · doi:10.1111/j.1551-6709.2011.01205.x · Cited by 89 · cited 1× · see citing works
“REG algorithms have traditionally been applied as computational models of people’s ability to refer to objects [54].”
Method
Using Perceptual and Cognitive Explanations for Enhanced Human-Agent Team Performance
Neerincx, M. A., van der Waa, J., Kaptein, F., & van Diggelen, J. (2018) · Interacción · doi:10.1007/978-3-319-91122-9_18 · Cited by 88 · cited 11× · see citing works
“APE [88] is one example of a Reactive technique used in robot-human collaboration, while PeCoX [157] is considered a Hybrid technique.”
BackgroundMethodInfluential
Explaining robot actions
Lomas, M., Chevalier, R., Cross, E., Garrett, R. C., Hoare, J., & Kopack, M. (2012) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2157689.2157748 · Cited by 88 · see citing works
Interactive Visualization and Manipulation of Attention-based Neural Machine Translation
Lee, J., Shin, J.-H., & Kim, J. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-2021 · Cited by 87 · cited 2× · see citing works
“Theoretical scenarios can be simulated by manipulating attention weights and observing the subsequent prediction changes using attention-weight visualization tools [128, 138].”
Background
Do You Get It? User-Evaluated Explainable BDI Agents
Broekens, J., Harbers, M., Hindriks, K., van den Bosch, K., Jonker, C., & Meyer, J. (2010) · Multiagent System Technologies · doi:10.1007/978-3-642-16178-0_5 · Cited by 86 · see citing works
Expressive Lights for Revealing Mobile Service Robot State
Baraka, K., Paiva, A., & Veloso, M. (2015) · ROBOT · doi:10.1007/978-3-319-27146-0_9 · Cited by 83 · see citing works
Make it personal: A social explanation system applied to group recommendations
Quijano-Sánchez, L., Sauer, C., Recio-Garcia, J. A., & Díaz-Agudo, B. (2017) · Expert systems with applications · doi:10.1016/j.eswa.2017.01.045 · Cited by 81 · see citing works
Preparing for the future
Kölbel, A. (1995) · The Veterinary Record · doi:10.1136/vr.h4626 · Cited by 75 · cited 1×
“Although the need for explainability in AI systems has been long established during the MYCIN era, also known as the era of the expert systems [111] [112], the current drive for explainability has been motivated by recent governmental eforts from the European Union, United States (USA) [21], and China [101], which hav…”
Background
Designs for explaining intelligent agents
Haynes, S. R., Cohen, M. A., & Ritter, F. (2009) · Int. J. Hum. Comput. Stud. · doi:10.1016/j.ijhcs.2008.09.008 · Cited by 75 · see citing works
Design of an intelligible mobile context-aware application
Lim, B. Y., & Dey, A. (2011) · Mobile HCI · doi:10.1145/2037373.2037399 · Cited by 72 · see citing works
Design and Evaluation of Explainable BDI Agents
Harbers, M., van den Bosch, K., & Meyer, J.-J. (2010) · 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology · doi:10.1109/wi-iat.2010.115 · Cited by 70 · cited 17× · see citing works
“Communicate (1a) (1b) (1c) (1d) (2a) (3a) (3b) (3c) (3d) (4a) (4b) GDA Explainable BDI SAT Meta-AQUA Proactive KAGR XAI-Plan Explainable NPC Debrief CREATIVE ARG MXRL MSX APE PeCoX Generate Comprehend Optimize XGDAI Fig.”
BackgroundMethodInfluential
Extracting Relational Explanations From Deep Neural Networks: A Survey From a Neural-Symbolic Perspective
Townsend, J., Chaton, T., & Monteiro, J. M. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2944672 · Cited by 70 · cited 2× · see citing works
“Neural-symbolic (NS) models combine the efficient learning and inference power of traditional NN and the interpretability of symbolic knowledge extraction and reasoning [70, 204].”
BackgroundMethod
Knowledge Graph Grounded Goal Planning for Open-Domain Conversation Generation
Xu, J., Wang, H., Niu, Z.-Y., Wu, H., & Che, W. (2020) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6474 · Cited by 70 · see citing works
Human-Centric AI for Trustworthy IoT Systems With Explainable Multilayer Perceptrons
García-Magariño, I., Muttukrishnan, R., & Lloret, J. (2019) · IEEE Access · doi:10.1109/access.2019.2937521 · Cited by 70 · see citing works
Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual AttentionPreprint
Ghosh, S., Burachas, G., Ray, A., & Ziskind, A. (2019) · arXiv.org · doi:10.48550/arxiv.1902.05715 · Cited by 69 · see citing works
Explaining Autonomous Driving by Learning End-to-End Visual Attention
Cultrera, L., Seidenari, L., Becattini, F., Pala, P., & Bimbo, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00178 · Cited by 66 · see citing works
Personalised self-explanation by robots: The role of goals versus beliefs in robot-action explanation for children and adults
Kaptein, F., Broekens, J., Hindriks, K., & Neerincx, M. A. (2017) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2017.8172376 · Cited by 64 · cited 1× · see citing works
“[105] describe how context-awareness and personalization are essential for XGDAI, few works in the literature fulfill the criteria.”
Background
Towards a Theory of Explanations for Human–Robot Collaboration
Sridharan, M., & Meadows, B. (2019) · KI - Künstliche Intelligenz · doi:10.1007/s13218-019-00616-y · Cited by 63 · see citing works
A Unified Framework for Explanation-Based Generalization of Partially Ordered and Partially Instantiated Plans
Kambhampati, S., & Kedar, S. (1994) · Artificial Intelligence · doi:10.1016/0004-3702(94)90011-6 · Cited by 63 · see citing works
Translating Neuralese
Andreas, J., Dragan, A., & Klein, D. (2017) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p17-1022 · Cited by 61 · see citing works
Towards Providing Explanations for AI Planner DecisionsPreprint
Borgo, R., Cashmore, M., & Magazzeni, D. (2018) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.1810.06338 · Cited by 60 · cited 10× · see citing works
“The algorithm uses a preliminary set of plans as input; the user chooses one of the plan’s actions; the XAI-Plan node generates explanatory plans and communicates with the user through the user interface (Fig.”
BackgroundMethodInfluential
Learning from Demonstration for Goal-Driven Autonomy
Weber, B., Mateas, M., & Jhala, A. (2012) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8311 · Cited by 60 · cited 1×
“[214] propose an observational learning approach to reduce the number of domain knowledge acquisitions necessary to implement the real-time strategy game StarCraft.”
Background
Effects of Agent Transparency on Operator Trust
Boyce, M. W., Chen, J. Y. C., Selkowitz, A. R., & Lakhmani, S. G. (2015) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2701973.2702059 · Cited by 59 · cited 1× · see citing works
“SAT is an agent transparency model for enhancing an operator’s situational awareness [60] in the agent’s environment [23, 39].”
Background
It Takes Two to Tango: Towards Theory of AI's MindPreprint
Chandrasekaran, A., Yadav, D., Chattopadhyay, P., Prabhu, V., & Parikh, D. (2017) · arXiv.org · doi:10.48550/arxiv.1704.00717 · Cited by 58 · see citing works
Visual Scene Understanding for Autonomous Driving Using Semantic Segmentation
Hofmarcher, M., Unterthiner, T., Arjona-Medina, J., Klambauer, G., Hochreiter, S., & Nessler, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_15 · Cited by 51 · see citing works
Coactive design of explainable agent-based task planning and deep reinforcement learning for human-UAVs teamwork
Wang, C., Wu, L.-Z., Yan, C., Wang, Z., Long, H., & Yu, C. (2020) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2020.05.001 · Cited by 50 · see citing works
Autonomous task planning and situation awareness in robotic surgery*
Ginesi, M., Meli, D., Roberti, A., Sansonetto, N., & Fiorini, P. (2020) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros45743.2020.9341382 · Cited by 50
Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks
Gao, X.-F., Gong, R., Zhao, Y.-Z., Wang, S., Shu, T., & Zhu, S.-C. (2020) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/ro-man47096.2020.9223595 · Cited by 48 · see citing works
Improving robot transparency: Real-time visualisation of robot AI substantially improves understanding in naive observers
Wortham, R., Theodorou, A., & Bryson, J. (2017) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2017.8172491 · Cited by 48 · see citing works
Behavior Explanation as Intention Signaling in Human-Robot Teaming
Gong, Z., & Zhang, Y. (2018) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2018.8525675 · Cited by 44 · see citing works
ASKA: receptionist robot with speech dialogue system
Nisimura, R., Uchida, T., Lee, A., Saruwatari, H., Shikano, K., & Matsumoto, Y. (2002) · IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/irds.2002.1043936 · Cited by 44 · cited 3×
“Communication techniques used in robot-human collaborative tasks mainly include human-like expressive abilities such as gestures [148], motion [148], and verbal communication [114, 160, 163, 178, 228].”
BackgroundMethod
Improving Human-Robot Interaction Through Explainable Reinforcement Learning
Tabrez, A., & Hayes, B. (2019) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1109/hri.2019.8673198 · Cited by 43 · cited 11×
“A color-based collaborative version of Sudoku and an autonomous robot are used to implement RARE [133].”
MethodInfluential
Rocco: A RoboCup Soccer Commentator System
Voelz, D., André, E., Herzog, G., & Rist, T. (1998) · Robot Soccer World Cup · doi:10.1007/3-540-48422-1_4 · Cited by 43
Visual Interrogation of Attention-Based Models for Natural Language Inference and Machine Comprehension
Liu, S.-S., Li, T., Li, Z.-M., Srikumar, V., Pascucci, V., & Bremer, P. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2007 · Cited by 39 · cited 1× · see citing works
“Theoretical scenarios can be simulated by manipulating attention weights and observing the subsequent prediction changes using attention-weight visualization tools [128, 138].”
Background
Memory-Based Explainable Reinforcement Learning
Cruz, F., Dazeley, R., & Vamplew, P. (2019) · Australasian Conference on Artificial Intelligence · doi:10.1007/978-3-030-35288-2_6 · Cited by 36 · cited 14× · see citing works
“[47] to enable an RL agent to explain to users why it selected an action over other possible actions.”
BackgroundMethodInfluential
Unmanned Aerial Vehicle Control Through Domain-based Automatic Speech Recognition
Contreras, R., Ayala, A., & Cruz, F. (2020) · De Computis · doi:10.3390/computers9030075 · Cited by 35 · see citing works
A purely reactive navigation scheme for dynamic environments using Case-Based Reasoning
Urdiales, C., Pérez, E., Vázquez-Salceda, J., Sànchez-Marrè, M., & Sandoval, F. (2006) · Auton. Robots · doi:10.1007/s10514-006-7231-8 · Cited by 35 · cited 14×
“CBR is used in many XGDAI approaches to acquire domain knowledge at run-time to minimize the prerequisite domain engineering for generating explanations.”
BackgroundMethodInfluential
Explainable PCGML via Game Design PatternsPreprint
Guzdial, M., Reno, J., Chen, J., Smith, G., & Riedl, M. O. (2018) · AIIDE Workshops · doi:10.48550/arxiv.1809.09419 · Cited by 34 · see citing works
Visualizations for an Explainable Planning Agent
Chakraborti, T., Fadnis, K. P., Talamadupula, K., Dholakia, M., Srivastava, B., Kephart, J., & Bellamy, R. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/849 · Cited by 34 · see citing works
Robot Navigation by Waypoints
Wang, Y., Mulvaney, D., Sillitoe, I., & Swere, E. (2008) · J. Intell. Robotic Syst. · doi:10.1007/s10846-008-9209-6 · Cited by 31
From self-assessment to frustration, a small step toward autonomy in robotic navigation
Jauffret, A., Cuperlier, N., Tarroux, P., & Gaussier, P. (2013) · Front. Neurorobot. · doi:10.3389/fnbot.2013.00016 · Cited by 30 · see citing works
Towards Situated Dialogue: Revisiting Referring Expression Generation
Fang, R., Liu, C.-S., She, L.-B., & Chai, J. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1038 · Cited by 30 · cited 1× · see citing works
“, in assembly tasks) and often need to refer to an object in the environment that is being shared or manipulated [63].”
Background
Scenario and design process of childcare robot PaPeRo
Osada, J., Ohnaka, S., & Sato, M. (2006) · IFAC Symposium on Advances in Control Education · doi:10.1145/1178823.1178930 · Cited by 30 · cited 5× · see citing works
“PaPeRo uses a dictionary of commonly used words and phrases and could also be updated by the designers.”
BackgroundInfluential
Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
Pan, X., Zhang, T.-N., Ichter, B., Faust, A., Tan, J., & Ha, S. (2019) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra40945.2020.9196602 · Cited by 28 · see citing works
The role of emotion in self-explanations by cognitive agents
Kaptein, F., Broekens, J., Hindriks, K., & Neerincx, M. A. (2017) · 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw.2017.8272595 · Cited by 28 · see citing works
Incident Command: Tales from the Hot Seat
Flin, R., & Arbuthnot, K. (2017) · Routledge eBooks · doi:10.4324/9781315252483 · Cited by 28 · cited 3× · see citing works
“According to Flin and Arbuthnot [65], revealing explainable BDI agents’ actual mental state to trainees may make them aware of their (false) assumptions about them.”
Background
Towards explainable multi-objective probabilistic planning
Sukkerd, R., Simmons, R., & Garlan, D. (2018) · Proceedings of the 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems · doi:10.1145/3196478.3196488 · Cited by 27 · see citing works
On applying AI-driven flight data analysis for operational spacecraft model-based diagnostics
Tipaldi, M., Feruglio, L., Denis, P., & D'Angelo, G. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.04.012 · Cited by 27 · see citing works
Communication in Human-Agent Teams for Tasks with Joint Action
Li, S.-R., Sun, W., & Miller, T. (2015) · COIN@AAMAS/IJCAI · doi:10.1007/978-3-319-42691-4_13 · Cited by 25 · see citing works
Improving intelligibility and control in Ubicomp
Vermeulen, J. (2010) · UbiComp '10 Adjunct · doi:10.1145/1864431.1864493 · Cited by 25 · cited 1× · see citing works
“Explainability simplifies the system operations, making it easier and quicker for human operators to intervene and diagnose errors [210].”
Background
A Methodology for Developing Self-Explaining Agents for Virtual Training
Harbers, M., van den Bosch, K., & Meyer, J. (2009) · Multi-Agent Logics, Languages, and Organisations Federated Workshops · doi:10.1007/978-3-642-13338-1_10 · Cited by 25 · see citing works
What's in your mind?
Wandell, B. A. (2008) · Nature Neuroscience · doi:10.1038/nn0408-384 · Cited by 24 · see citing works
Auditory system in a personal robot, PaPeRo
Sato, M., Sugiyama, A., & Ohnaka, S. (2006) · 2006 Digest of Technical Papers International Conference on Consumer Electronics · doi:10.1109/icce.2006.1598289 · Cited by 24 · cited 1×
“PaPeRo uses an electronic hardware auditory system for human-robot communication [182] and can recognize multiple utterances, give a quiz to children who provide answers to the quiz using a special microphone and tell in natural language the names of the children who got the correct answer.”
Method
Deep, Consistent Behavioral Decision Making with Planning Features for Autonomous Vehicles
Qian, L., Xu, X., Zeng, Y.-J., & Huang, J. (2019) · Electronics · doi:10.3390/electronics8121492 · Cited by 23 · see citing works
How AI Wins Friends and Influences People in Repeated Games With Cheap Talk
Oudah, M., Rahwan, T., Crandall, T., & Crandall, J. (2018) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11486 · Cited by 22 · see citing works
Incorporating Transparency During Trust-Guided Behavior Adaptation
Floyd, M. W., & Aha, D. (2016) · International Conference on Case-Based Reasoning · doi:10.1007/978-3-319-47096-2_9 · Cited by 19 · cited 2× · see citing works
“In a similar efort, Floyd and Aha [47] applied two CBR systems to assist agent learning and explanation generation.”
Method
Visually Interactive Location-Aware Computing
Rehman, K., Stajano, F., & Coulouris, G. (2005) · Ubiquitous Computing · doi:10.1007/11551201_11 · Cited by 19 · see citing works
Agent Architecture for Adaptive Behaviors in Autonomous Driving
da Lio, M., Donà, R., Papini, G. P. R., & Gurney, K. (2020) · IEEE Access · doi:10.1109/access.2020.3007018 · Cited by 19 · see citing works
Clustering Behavior to Recognize Subjective Beliefs in Human-Agent Teams
Pynadath, D., Wang, N., Rovira, E., & Barnes, M. (2018) · Adaptive Agents and Multi-Agent Systems · doi:10.65109/xwaa3999 · Cited by 18 · see citing works
Effect of Expressive Lights on Human Perception and Interpretation of Functional Robot
Song, S., & Yamada, S. (2018) · CHI Extended Abstracts · doi:10.1145/3170427.3188547 · Cited by 18 · cited 3×
“Similarly, simple communication methods such as expressive lights [13, 194] are unexplored territory for XGDAI.”
BackgroundInfluential
MIRIAM: A Multimodal Interface for Explaining the Reasoning Behind Actions of Remote Autonomous Systems
Hastie, H., Chiyah-Garcia, J., Robb, D. A., Laskov, A., & Patrón, P. (2018) · International Conference on Multimodal Interaction · doi:10.1145/3242969.3266297 · Cited by 17 · see citing works
Effects of agent transparency and communication framing on human-agent teaming
Wohleber, R. W., Stowers, K., Chen, J. Y. C., & Barnes, M. (2017) · IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/smc.2017.8123160 · Cited by 17 · see citing works
Learning Spatial Models for Navigation
Epstein, S. L., Aroor, A., Evanusa, M. S., Sklar, E., & Parsons, S. (2015) · Conference On Spatial Information Theory · doi:10.1007/978-3-319-23374-1_19 · Cited by 17 · cited 1× · see citing works
“WHY-Plan is implemented in SemaFORR, a cognitive-based hybrid robot controller [61].”
Method
ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning
van der Waa, J., van Diggelen, J., Neerincx, M. A., & Raaijmakers, S. (2018) · International Conference on Agents and Artificial Intelligence · doi:10.5220/0006542603140321 · Cited by 17 · cited 3× · see citing works
“PeCoX’s perceptual XAI explains the perceptual aspect of the agent’s behavior using a proposed Intuitive Confidence Measure (ICM) [207] and a method of contrastive explanations [176] that involve the classification of facts and foils, e.”
Method
Explaining Deep Neural Networks using Unsupervised ClusteringPreprint
Liu, Y.-H., & Arik, S. O. (2020) · arXiv.org · doi:10.48550/arxiv.2007.07477 · Cited by 16 · cited 1× · see citing works
“From the training data, CNNs learn to extract a deep hierarchy of task-relevant visual features [10, 52, 117, 204].”
Method
What’s on Your Mind, NICO?
Kerzel, M., Ambsdorf, J., Becker, D. L., Lu, W., Strahl, E., Spisak, J., Gäde, C., Weber, T., & Wermter, S. (2022) · KI - Künstliche Intelligenz · doi:10.1007/s13218-022-00772-8 · Cited by 15 · see citing works
Self-citation
Explain yourself! Effects of Explanations in Human-Robot Interaction
Ambsdorf, J., Munir, A., Wei, Y., Degkwitz, K., Harms, H. M., Stannek, S., Ahrens, K., Becker, D., Strahl, E., Weber, T., et al. (2022) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/ro-man53752.2022.9900558 · Cited by 15 · cited 1× · see citing works
“[4], participants’ perceptions towards robots were evaluated as they watched two NICO robots playing tic-tac-toe against each other.”
BackgroundSelf-citation
Explanation-Aware Experience Replay in Rule-Dense Environments
Sovrano, F., Raymond, A., & Prorok, A. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3135927 · Cited by 15 · cited 1× · see citing works
“[195] propose a concept called “eXplanation Awareness” (XA) for organizing acquired experiences into clusters labeled on a case-by-case basis.”
Background
Emotion Recognition Using Explainable Genetically Optimized Fuzzy ART Ensembles
Liew, W. S., Loo, C., & Wermter, S. (2021) · IEEE Access · doi:10.1109/access.2021.3072120 · Cited by 15 · cited 1× · see citing works
“Example of a SHAP summary plot (top) and feature dependence plot (bottom) describing feature importance for affect recognition [133].”
BackgroundSelf-citation
Strategic Tasks for Explainable Reinforcement Learning
Pocius, R., Neal, L., & Fern, A. (2019) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.330110007 · Cited by 15 · cited 1×
“[166], deep RL is used to provide visual interpretations in saliency maps to describe an agent’s decisions in a partially observable game scenario.”
Method
Exploration of intention expression for robots
Shindev, I., Sun, Y., Coovert, M., Pavlova, J., & Lee, T. (2012) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2157689.2157778 · Cited by 15
Intelligent autonomous vehicles with an extendable knowledge base under meaningful human control
Beckers, G., Sijs, J., van Diggelen, J., van Dijk, R. V., Bouma, H., Lomme, M., Hommes, R., Hillerstrom, F., van der Waa, J., van Velsen, A., et al. (2019) · Security + Defence · doi:10.1117/12.2533740 · Cited by 14 · see citing works
Expression of intention by rotational head movements for teleoperated mobile robot
Mikawa, M., Yoshikawa, Y., & Fujisawa, M. (2018) · International Workshop on Advanced Motion Control · doi:10.1109/amc.2019.8371097 · Cited by 14 · cited 1×
“Communication techniques used in robot-human collaborative tasks mainly include human-like expressive abilities such as gestures [148], motion [148], and verbal communication [114, 160, 163, 178, 228].”
Background
Emotionally expressive robot behavior improves human-robot collaboration
Novikova, J., Watts, L., & Inamura, T. (2015) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2015.7333645 · Cited by 14
HeTROPY: Explainable learning diagnostics via heterogeneous maximum-entropy and multi-spatial knowledge representation
Huo, Y., Wong, D. F., Ni, L., Chao, L., & Zhang, J. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106389 · Cited by 12 · see citing works
BEN: An Agent Architecture for Explainable and Expressive Behavior in Social Simulation
Bourgais, M., Taillandier, P., & Vercouter, L. (2019) · EXTRAAMAS@AAMAS · doi:10.1007/978-3-030-30391-4_9 · Cited by 12 · see citing works
Auto-Extraction, Representation and Integration of a Diabetes Ontology Using Bayesian Networks
McGarry, K., Garfield, S., & Wermter, S. (2007) · Computer-Based Medical Systems · doi:10.1109/cbms.2007.26 · Cited by 12 · cited 1× · see citing works
“[146] develop a method for automatically generating and integrating biological ontology into Bayesian network frameworks to enable reasoning and prediction.”
MethodSelf-citation
A conceptual foundation for autonomous learning in unforeseen situations
Kennedy, C. (1998) · Proceedings of the 1998 IEEE International Symposium on Intelligent Control (ISIC) held jointly with IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) Intell · doi:10.1109/isic.1998.713709 · Cited by 12 · cited 1× · see citing works
“APE is a strategy proposed by Hayes and Shah [88] for a class of reactive agent robot controllers that rely on blackbox-trained reinforcement learning models [108] or on hard-coded conditional statement-driven policies to enable the robot to autonomously synthesize policy descriptions and respond to natural language q…”
Method
Abstracting Deep Neural Networks into Concept Graphs for Concept Level Interpretability
Kori, A., Natekar, P., Krishnamurthi, G., & Srinivasan, B. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-93080-6_15 · Cited by 11 · cited 1× · see citing works
“From the training data, CNNs learn to extract a deep hierarchy of task-relevant visual features [10, 52, 117, 204].”
Method
Knowledge Extraction from Transducer Neural Networks
Wermter, S. (2004) · Applied intelligence (Boston) · doi:10.1023/a:1008320219610 · Cited by 11 · cited 1× · see citing works
“Transducer extraction [217] was then able to obtain detailed symbolic information while preserving sequential information.”
BackgroundSelf-citation
The effect of explanations on perceived control and behaviors in intelligent systems
Holliday, D., Wilson, S. M., & Stumpf, S. (2013) · CHI Extended Abstracts · doi:10.1145/2468356.2468389 · Cited by 10 · see citing works
Towards Research on Goal Reasoning with the TAO Sandbox
Auslander, B., Molineaux, M., Aha, D., Munro, A., & Pizzini, Q. (2009) · doi:10.21236/ada553303 · Cited by 10 · cited 1×
“The ARTUE system is implemented in a sandbox simulation scenario [11].”
Method
XAI4Wind: A Multimodal Knowledge Graph Database for Explainable Decision Support in Operations & Maintenance of Wind Turbines
Chatterjee, J., & Dethlefs, N. (2020) · arXiv.org · doi:10.48550/arxiv.2012.10489 · Cited by 9
Explainable and Adaptable Augmentation in Knowledge Attention Network for Multi-Agent Deep Reinforcement Learning Systems
Ho, J., & Wang, C.-M. (2020) · International Conference on Artificial Intelligence and Knowledge Engineering · doi:10.1109/aike48582.2020.00031 · Cited by 9 · cited 1× · see citing works
“Explainer systems provide insights on how and why individual agents take certain actions and how the cumulative choices produce emergent behavior [92].”
Background
Bootstrapping Knowledge Graphs From Images and Text
Mao, J.-Y., Yao, Y., Heinrich, S., Hinz, T., Weber, C., Wermter, S., Liu, Z., & Sun, M. (2019) · Front. Neurorobot. · doi:10.3389/fnbot.2019.00093 · Cited by 9 · cited 1× · see citing works
“[145] introduce an algorithmic methodology for encoding relational representations of visual inputs in the form of sparse knowledge graphs.”
MethodSelf-citation
An Agent-Based Approach to Realize Emergent Configurations in the Internet of Things
Alkhabbas, F., Spalazzese, R., & Davidsson, P. (2020) · Electronics · doi:10.3390/electronics9091347 · Cited by 8
Self-explanation capability for cognitive agents on-board of UCAVs to improve cooperation in a manned-unmanned fighter team
Lettl, B., & Schulte, A. (2013) · AIAA Infotech@Aerospace Conference · doi:10.2514/6.2013-4898 · Cited by 8 · see citing works
Classification of MRI Migraine Medical Data Using 3D Convolutional Neural Network
Ng, H. G., Kerzel, M., Mehnert, J., May, A., & Wermter, S. (2018) · International Conference on Artificial Neural Networks · doi:10.1007/978-3-030-01424-7_30 · Cited by 7 · cited 17× · see citing works
“Grad-CAM is then used to highlight the parts of the object which are relevant for generating the movement for a particular joint motor.”
MethodInfluentialSelf-citation
Teaching Meaningful ExplanationsPreprint
Codella, N., Hind, M., Ramamurthy, K., Campbell, M., Dhurandhar, A., Varshney, K. R., Wei, D., & Mojsilovic, A. (2018) · arXiv.org · doi:10.48550/arxiv.1805.11648 · Cited by 7 · see citing works
Symbolic state transducers and recurrent neural preference machines for text mining
Arevian, G., Wermter, S., & Panchev, C. (2003) · International Journal of Approximate Reasoning · doi:10.1016/s0888-613x(02)00085-3 · Cited by 7 · cited 1×
“In a text-mining application [9], symbolic transducers were able to encode concepts from a news corpus quickly.”
BackgroundSelf-citation
Explanation language syntax for Multi-Agent Systems
Hedhili, A., Chaari, W., & Ghédira, K. (2013) · 2013 World Congress on Computer and Information Technology (WCCIT) · doi:10.1109/wccit.2013.6618689 · Cited by 6 · cited 1×
“Under the KAGR conceptual explanation framework, events produced by the agents at run-time are intercepted, and an explanatory knowledge acquisition phase is performed to represent the knowledge attributes or details related to the execution of events in a KAGR structure [91, 183].”
Method
Revisiting Visual Grounding
Conser, E., Hahn, K., Watson, C. M., & Mitchell, M. (2019) · Proceedings of the Second Workshop on Shortcomings in Vision and Language · doi:10.18653/v1/w19-1804 · Cited by 5 · cited 1× · see citing works
“Visual grounding is the general task of locating a structured description’s components in an image [43].”
Background
Tell Me Why You Feel That Way: Processing Compositional Dependency for Tree-LSTM Aspect Sentiment Triplet Extraction (TASTE)
Sutherland, A., Bensch, S., Hellström, T., Magg, S., & Wermter, S. (2021) · International Conference on Artificial Neural Networks · doi:10.1007/978-3-030-61609-0_52 · Cited by 4 · cited 1×
“[199] introduce a context-agnostic methodology for extracting target-sentiment-cause triplets from sentences.”
MethodSelf-citation
A Cognitive Architecture for Verifiable System Ethics via Explainable Autonomy
Yilmaz, L., & Sivaraj, S. (2019) · IEEE Systems Conference · doi:10.1109/syscon.2019.8836896 · Cited by 3
Untitled work
(2017) · doi:10.9728/dcs.2024.25.7.1847 · Cited by 2 · cited 2× · see citing works
“Lundberg and Lee [141] utilize SHapley Additive exPlanation (SHAP) values, an extension of Shapley values, as a united strategy to explain any machine learning model output.”
Method
Innovative model for situation awareness in dynamic defense systems
Endsley, M. (2018) · doi:10.1201/b22181-8 · Cited by 2 · cited 1×
“SAT is an agent transparency model for enhancing an operator’s situational awareness [60] in the agent’s environment [23, 39].”
Background

Flagged first, then by citations

50 of the 237 references couldn’t be matched to a known paper.

Research works printed but not linked: 46

Dan Amir and Ofra Amir. 2018. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, United States, 1168-1176.
Kevin Baum, Holger Hermanns, and Timo Speith. 2018. From machine ethics to machine explainability and back. In International Symposium on Artificial Intelligence and Mathematics ISAIM2018. ISAIM, Fort Lauderdale, Florida, United States, 8 pages.
Julien Bidot, Susanne Biundo, Tobias Heinroth, Wolfgang Minker, Florian Nothdurft, and Bernd Schattenberg. 2010. Verbal plan explanations for hybrid planning. In Proceedings of Multikonferenz Wirtschaftsinformatik. Citeseer, 2309-2320.
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8.
Susanne Biundo and Bernd Schattenberg. 2001. From abstract crisis to concrete relief -a preliminary report on combining state abstraction and HTN planning. In Proceedings of the 6th European Conference on Planning. Springer Verlag, 157-168.
Gregory Bonaert, Youri Coppens, Denis Steckelmacher, and Ann Nowe. 2021. Explainable Reinforcement Learning Through Goal-Based Interpretability. https://openreview.net/forum?id=IlJbTsygaI6.
Max Bramer. 2005. Logic Programming with Prolog. Vol. 9. Springer.
Peter Carey. 2018. Data Protection: A Practical Guide to UK and EU Law. Oxford University Press, Inc.
Dustin Dannenhauer, Michael W. Floyd, Matthew Molineaux, and David W. Aha. 2018. Learning from exploration: Towards an explainable goal reasoning agent. In Proceedings of the Workshop on Adaptive Learning Agents (held at the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence). 1-8.
Hiromichi Fujisawa, Katsuhiko Shirai, and Yukio Sakai. 1973. Ear and voice of the WABOT -the speech-input-output system. Bulletin of Science and Engineering Research Laboratory Waseda (7 1973), 39-85.
Melinda T. Gervasio, Karen L. Myers, Eric Yeh, and Boone Adkins. 2018. Explanation to avert surprise. In IUI Work- shops, Vol. 2068.
Manuel Giuliani, Mary Ellen Foster, Amy Isard, Colin Matheson, Jon Oberlander, and Alois Knoll. 2010. Situated refer- ence in a hybrid human-robot interaction system. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics, 67-75.
Antoine Grea, Laëtitia Matignon, and Samir Aknine. 2018. How explainable plans can make planning faster. In Workshop on Explainable Artificial Intelligence. hal.archives-ouvertes.fr, Stockholm, Sweden, 58-64.
Maaike Harbers, Joost Broekens, Karel Van Den Bosch, and John-Jules Meyer. 2010. Guidelines for developing ex- plainable cognitive models. In Proceedings of the 10th International Conference on Composite Materials. Citeseer, 85-90.
Ulit Jaidee, Héctor Muñoz-Avila, and David W. Aha. 2011. Case-based learning in goal-driven autonomy agents for real-time strategy combat tasks. In Proceedings of the 19th International Conference on Case-Based Reasoning (ICCBR 2011) Workshop on Computer Games. 43-52.
Ulit Jaidee, Héctor Muñoz-Avila, and David W. Aha. 2011. Integrated learning for goal-driven autonomy. In 22nd International Joint Conference on Artificial Intelligence.
W. Lewis Johnson. 1994. Agents that learn to explain themselves. In Proceedings of the 12th AAAI National Conference on Artificial Intelligence. 1257-1263.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence. 47-53.
Kevin Knight. 1993. Are many reactive agents better than a few deliberative ones?. In Proceedings of the 13th Inter- national Joint Conference on Artificial Intelligence, Vol. 93. 432-437.
T. Kobayashi. 1985. Speech conversation system of the musician robot. Proceedings of the 1985 International Confer- ence on Advanced Robotics (1985), 483-488.
Raj Korpan and Susan L. Epstein. 2018. Toward natural explanations for a robot's navigation plans. Notes from the Explainable Robotic Systems Workshop, Human-Robot Interaction (2018).
Alfred Kranstedt and Ipke Wachsmuth. 2005. Incremental generation of multimodal deixis referring to objects. In Proceedings of the 10th European Workshop on Natural Language Generation.
Jens Kröske, Kevin O'Holleran, and Hannu Rajaniemi. 2009. Trusted reasoning engine for autonomous systems with an interactive demonstrator. In 4th SEAS DTC Technical Conference. Citeseer.
Michael L. Littman. 1994. Memoryless policies: Theoretical limitations and practical results. In From Animals to Animats 3: Proceedings of the Third International Conference on Simulation of Adaptive Behavior, Vol. 3. Cambridge, MA, USA, 238.
Ken McGarry, Stefan Wermter, and J. MacIntyre. 2001. Knowledge extraction from local function networks. In Pro- ceedings of the International Joint Conference on Artificial Intelligence. Seattle, WA, USA. https://www2.informatik.uni- hamburg.de/wtm/publications/2001/MWM01a/.
Matthew Molineaux, Dustin Dannenhauer, and David W. Aha. 2018. Towards explainable NPCs: A relational explo- ration learning agent. In Workshops at the 32nd AAAI Conference on Artificial Intelligence. 565-569.
Raymond J. Mooney and Scott Bennett. 1986. A domain independent explanation-based generalizer. In Proceedings of the 5th AAAI Conference on Artificial Intelligence. 551-555.
Hector Munoz-Avila and David W. Aha. 2010. A case study of goal-driven autonomy in domination games. In Pro- ceedings of the AAAI Workshop on Goal-Directed Autonomy.
Dana Nau, Yue Cao, Amnon Lotem, and Hector Munoz-Avila. 1999. SHOP: Simple hierarchical ordered planner. In Proceedings of the 16th International Joint Conference on Artificial Intelligence. 968-973.
Frank Puppe. 2012. Systematic Introduction To Expert Systems: Knowledge Representations and Problem-Solving Meth- ods. Springer Science & Business Media.
Anand S. Rao, Michael P. Georgeff, et al. 1995. BDI agents: From theory to practice. In Proceedings of the 1st Interna- tional Conference on Multi-Agent Systems. 312-319.
Marcel Jurriaan Robeer. 2018. Contrastive Explanation for Machine Learning. Master's thesis. Utrecht University, Netherlands.
Stephanie Rosenthal, Sai P. Selvaraj, and Manuela M. Veloso. 2016. Verbalization: Narration of autonomous robot experience. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. 862-868.
Michael K. Sahota. 1994. Reactive deliberation: An architecture for real-time intelligent control in dynamic environ- ments. In Proceedings of the 12th National Conference on Artificial Intelligence, Vol. 2. 1303-1308.
Aroua Hedhili Sbai, Wided Lejouad Chaari, and Khaled Ghedira. 2013. Intra-agent explanation using temporal and extended causal maps. Procedia Computer Science 22 (2013), 241-249.
Bastian Seegebarth, Felix Müller, Bernd Schattenberg, and Susanne Biundo. 2012. Making hybrid plans more clear to human users -A formal approach for generating sound explanations. In Proceedings of the 22nd International Conference on International Conference on Automated Planning and Scheduling. 225-233.
Katsuhiko Shirai and Hiromichi Fujisawa. 1974. An algorithm for spoken sentence recognition and its application to the speech input-output system. IEEE Transactions on Systems, Man, & Cybernetics 4, 5 (1974), 475-479.
Roykrong Sukkerd, Reid Simmons, and David Garlan. 2018. Toward explainable multi-objective probabilistic plan- ning. In 2018 IEEE/ACM 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems. IEEE, 19-25.
Rik Van den Brule, Gijsbert Bijlstra, Ron Dotsch, Daniël H. J. Wigboldus, and W. F. G. Haselager. 2013. Signaling robot trustworthiness: Effects of behavioral cues as warnings. Lecture Notes in Computer Science 8239 (2013), 583-584.
Michael Van Lent, William Fisher, and Michael Mancuso. 2004. An explainable artificial intelligence system for small- unit tactical behavior. In Proceedings of the National Conference on Artificial Intelligence. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 900-907.
Michael van Lent and John Laird. 1998. Learning by observation in a complex domain. In Proceedings of the Knowledge Acquisition Workshop.
Stefan Wermter. 1999. Preference Moore machines for neural fuzzy integration. In Proceedings of the 16th International Joint Conference on Artificial Intelligence. 840-845.
Stefan Wermter, Sascha Griffiths, and Stefan Heinrich. 2017. Crossmodal lifelong learning in hybrid neural embodied architectures. Behavioral and Brain Sciences 40 (2017), 72.
Handy Wicaksono, Claude Sammut, and Raymond Sheh. 2017. Towards explainable tool creation by a robot. In International Joint Conference on Artificial Intelligence Workshop on Explainable AI. 63.
M. Wooldridge. 1995. Conceptualising and developing agents. In Proceedings of the UNICOM Seminar on Agent Soft- ware, Vol. 42. London.
Robert H. Wortham, Andreas Theodorou, and Joanna J. Bryson. 2016. What does the robot think? Transparency as a fundamental design requirement for intelligent systems. In IJCAI Ethics for Artificial Intelligence Workshop.

Sorted by citations

AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 244 · see citing works
“To develop this review, we implemented a hybrid search methodology combining traditional academic repositories and AI-enhanced literature discovery tools.”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186 · see citing works
Explainable AI Over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions
Jagatheesaperumal, S. K., Pham, Q.-V., Ruby, R., Yang, Z., Xu, C., & Zhang, Z. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3215676 · Cited by 119 · see citing works
“healthcare [32, 33, 34], augmentation [35], robots [36, 37], and the solutions associated with ML models [38, 39, 40] for exploring insights into the decisions made by the systems.”
Background
AI-enabled Enterprise Information Systems for Manufacturing
Zdravković, M., Panetto, H., & Weichhart, G. (2021) · Enterprise Information Systems · doi:10.1080/17517575.2021.1941275 · Cited by 119 · see citing works
“Explainable AI can significantly improve safety and trust in goal-driven robots (Sado et al. 2020).”
Background
A Review on Fault Detection and Diagnosis of Industrial Robots and Multi-axis Machines
Sabry, A. H., & Amirulddin, U. A. U. (2024) · Results in Engineering · doi:10.1016/j.rineng.2024.102397 · Cited by 93 · see citing works
A Systematic Literature Review on the Application of Automation in Logistics
Ferreira, B., & Reis, J. (2023) · Logistics · doi:10.3390/logistics7040080 · Cited by 69 · see citing works
Explainable reinforcement learning (XRL): a systematic literature review and taxonomy
Bekkemoen, Y. (2023) · Machine Learning · doi:10.1007/s10994-023-06479-7 · Cited by 58 · see citing works
Industrial Foundation Models (IFMs) for intelligent manufacturing: A systematic review
Zhao, S., Liu, S., Jiang, Y., Zhao, B. M., Lv, Y., Zhang, J., Wang, L., & Zhong, R. Y. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.011 · Cited by 46 · see citing works
When Do People Want an Explanation from a Robot?
Wachowiak, L., Fenn, A., Kamran, H., Coles, A., Çeliktutan, O., & Canal, G. (2024) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/3610977.3634990 · Cited by 43 · see citing works
How Do Autonomous Vehicles Decide?
Malik, S., Khan, M., Elsayed, H., Khan, J., & Ullah, O. (2022) · Italian National Conference on Sensors · doi:10.3390/s23010317 · Cited by 33 · see citing works
“[104], the procedure may involve translating the agent’s plans into simple understandable formats and creating user interfaces that make this understanding easier, depending on the stakeholder.”
Background
Learning Controllers for Continuum Soft Manipulators: Impact of Modeling and Looming Challenges
Falotico, E., Donato, E., Alessi, C., Setti, E., Nazeer, M. S., Agabiti, C., Caradonna, D., Bianchi, D., Piqué, F., Ansari, Y., et al. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202400344 · Cited by 32 · see citing works
Fusion of Computer Vision and AI in Collaborative Robotics: A Review and Future Prospects
Cohen, Y., Biton, A., & Shoval, S. (2025) · Applied Sciences · doi:10.3390/app15147905 · Cited by 30 · see citing works
Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
Sarkar, A., Idris, M. Y. I., & Yu, Z.-Y. (2025) · arXiv.org · doi:10.48550/arxiv.2508.10523 · Cited by 25
“These limitations are more severe in safety-critical applications such as autonomous vehicles, clinical diagnosis, and environmental monitoring via satellite images, where transparency and credibility have direct effects on deployment [10–15].”
Expanding AI’s Role in Healthcare Applications: A Systematic Review of Emotional and Cognitive Analysis Techniques
Nag, P. K., Bhagat, A., & Priya, R. (2025) · IEEE Access · doi:10.1109/access.2025.3562131 · Cited by 24 · see citing works
“Transparent AI models, as discussed by [44] and [138], facilitate better decision-making and ensure accountability in clinical environments.”
Influential
From neural activations to concepts: A survey on explaining concepts in neural networks
Lee, J. H., Lanza, S., & Wermter, S. (2024) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240743 · Cited by 9 · see citing works
“Some XAI surveys have been published in recent years [15,24,31,32,45,47,49].”
Self-citation
GPT-in-the-Loop: Adaptive Decision-Making for Multiagent Systems
Nascimento, N., Alencar, P. S. C., & Cowan, D. D. (2023) · arXiv.org · doi:10.48550/arxiv.2308.10435 · Cited by 19
“In addition to problem-solving skills, the agents should also offer an explanation for their decisions (Sado et al. 2023).”
Background
Building Appropriate Mental Models: What Users Know and Want to Know about an Agentic AI Chatbot
Brachman, M., Kunde, S., Miller, S., Fucs, A., Dempsey, S., Jabbour, J., & Geyer, W. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712071 · Cited by 18 · see citing works
Approach or avoidance? Relationship between perceived AI explainability and employee job crafting.
Huo, W.-W., Xie, J., Yan, J., Long, T., & Liang, B.-Q. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105097 · Cited by 17 · see citing works
Explainable Interface for Human-Autonomy Teaming: A Survey
Kong, X.-Q., Xing, Y., Tsourdos, A., Wang, Z.-Y., Guo, W.-S., Perrusquía, A., & Wikander, A. (2024) · arXiv.org · doi:10.48550/arxiv.2405.02583 · Cited by 17
“The functional module comprises perception, localization, prediction, decision, and action [215, 282].”
Background
Understandable Robots
Kumar, S. (2022) · arXiv.org · doi:10.48550/arxiv.2209.10939 · Cited by 14
“A previous review work (Sado et al. 2020) in the field of GDXAI categorized according to the behavioral aspects of interaction between the agent and the human i.”
BackgroundMethodInfluential
Pain recognition and pain empathy from a human-centered AI perspective
Cao, S., Fu, D., Yang, X., Wermter, S., Liu, X., & Wu, H. (2021) · iScience · doi:10.1016/j.isci.2024.110570 · Cited by 14 · see citing works
Self-citation
XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees
Roth, A. M., Liang, J., & Manocha, D. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636759 · Cited by 17 · see citing works
“Motivated by the desire to understand the sometimes opaque and inscrutabble nature of many advanced deep learning methods, Explainble AI (XAI) is a growing area of exploration [10], [11].”
Background
A conversational multi-agent AI system for automated plant phenotyping
Chen, F., Stogiannidis, I., Wood, A., Bueno, D., Williams, D., Macfarlane, F., Grieve, B., Wells, D. M., Atkinson, J. A., Hawkesford, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-026-71090-y · Cited by 13 · see citing works
“An AI agent is a goal-directed software designed to perform tasks within a de ﬁ ned digital environment 10,11 .”
A Closer Look at Reward Decomposition for High-Level Robotic Explanations
Lu, W., Zhao, X.-F., Magg, S., Gromniak, M., Li, M.-D., & Wermterl, S. (2023) · International Conference on Development and Learning · doi:10.1109/icdl55364.2023.10364407 · Cited by 13 · see citing works
Explainable AI is Responsible AI: How Explainability Creates Trustworthy and Socially Responsible Artificial Intelligence
Baker, S., & Xiang, W. (2023) · arXiv · doi:10.48550/arxiv.2312.01555 · Cited by 12 · see citing works
“Human-in-the-loop debugging was also considered in the safety context [120], with explanations provided by goal-driven AIs to human safety auditors.”
Background
OntoLogX: Ontology‐Guided Knowledge Graph Extraction From Cybersecurity Logs With Large Language Models
Cotti, L., Drago, I., Rula, A., Bianchini, D., & Cerutti, F. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202501381 · Cited by 12
“AI agents are autonomous software entities engineered to perform goal-directed tasks within bounded digital environments [40, 41].”
Questioning the Robot: Using Human Non-verbal Cues to Estimate the Need for Explanations
Kontogiorgos, D., & Shah, J. A. (2025) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1109/hri61500.2025.10974079 · Cited by 12
“Explaining actions is crucial, especially when errors occur [4, 60, 82, 103, 115, 116, 118, 121].”
A Taxonomy of Explanation Types and Need Indicators in Human–Agent Collaborations
Wachowiak, L., Coles, A., Canal, G., & Çeliktutan, O. (2024) · International Journal of Social Robotics · doi:10.1007/s12369-024-01148-8 · Cited by 12 · see citing works
What Would I Do If…? Promoting Understanding in HRI through Real-Time Explanations in the Wild
Love, T., Andriella, A., & Alenyà, G. (2024) · 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN) · doi:10.1109/ro-man60168.2024.10731403 · Cited by 9 · see citing works
“However, as the potential impact robot decisions can have on people’s daily lives grows, so too does the need for the decision-making of these robots to be understood and explained [4][5], with the European Union going so far as to designate a “right to explanation” [6].”
Background
Towards Explainable Proactive Robot Interactions for Groups of People in Unstructured Environments
Love, T., Andriella, A., & Alenyà, G. (2024) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/3610978.3640734 · Cited by 9 · see citing works
GPT-in-the-Loop: Supporting Adaptation in Multiagent Systems
Nascimento, N., Alencar, P. S. C., & Cowan, D. D. (2023) · BigData Congress [Services Society] · doi:10.1109/bigdata59044.2023.10386490 · Cited by 9 · see citing works
“In addition to problem-solving skills, the agents should also explain their decisions [15].”
Background
ASTRID: A Robotic Tutor for Nurse Training to Reduce Healthcare-Associated Infections
Qian, P.-Z., Bajraktari, F., Quintero-Peña, C., Meng, Q.-X., Hamlin, S., Kavraki, L. E., & Unhelkar, V. (2025) · Robotics · doi:10.15607/rss.2025.xxi.090 · Cited by 7
“One way to address this is by explaining the robot’s capabilities and limitations to students and involving instructors in the process [57, 12, 76, 13, 14, 31, 77, 55, 61, 117, 123, 131, 99, 27, 108, 109, 92, 100].”
RACCOON: Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules
Bustamante, S., Knauer, M., Thun, J., Schneyer, S., Albu-Schäffer, A., Weber, B. M., & Stulp, F. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra55743.2025.11127843 · Cited by 7 · see citing works
“In the words of Sado et. al., “Adding a rich [natural language processing] system to existing frameworks would significantly benefit [explainable Goal-driven artificial intelligence] in its usefulness and ap-plicability” [23].”
Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures
He, Y., Li, Y.-X., White, C., & Vitercik, E. (2025) · doi:10.48550/arxiv.2505.24069 · Cited by 6
“Understanding how well LLMs reason structurally is thus essential for their safe deployment in critical domains such as robotics (Sado et al., 2023) and healthcare (Sadeghi et al., 2024).”
Analysis of Explainable Goal-Driven Reinforcement Learning in a Continuous Simulated Environment
Portugal, E., Cruz, F., Ayala, A., & Fernandes, B. (2022) · Algorithms · doi:10.3390/a15030091 · Cited by 6 · see citing works
“[15], research is still required in this area.”
Background
Real-Time Scheduling with Independent Evaluators: Explainable Multi-Agent Approach
Isakov, A. O., Peregorodiev, D., Томилов, И. В., Ye, C., Gusarova, N., Vatian, A. S., & Boukhanovsky, A. V. (2024) · Technologies · doi:10.3390/technologies12120259 · Cited by 6 · see citing works
GPT-4 as a Moral Reasoner for Robot Command Rejection
Wen, R.-C., Ferraro, F., & Matuszek, C. (2024) · International Conference on Human-Agent Interaction · doi:10.1145/3687272.3688319 · Cited by 5 · see citing works
Regression Prediction of Critical Exhaust Volumetric Flow Rate in Tunnel Fire with Two-Point Extraction Ventilation based on Neural Network Method
Yao, G.-H., Yuan, Z.-Y., Liang, C., & Yu, N.-Y. (2024) · Case Studies in Thermal Engineering · doi:10.1016/j.csite.2024.105170 · Cited by 5
Enhancing Trust in Autonomous Agents: An Architecture for Accountability and Explainability through Blockchain and Large Language Models
Fernández-Becerra, L., González-Santamarta, M. '., Guerrero-Higueras, Á., Rodríguez-Lera, F., & Olivera, V. M. (2024) · arXiv.org · doi:10.48550/arxiv.2403.09567 · Cited by 5
“In Scenario 2, significant differences were observed for Ex-planations 1, 7, 8, and 15, with p -values below the threshold of 0.05, while the other questions did not show significant variations.”
Influential
The Case for Negative Data: From Crash Reports to Counterfactuals for Reasonable Driving
Patrikar, J., Sharma, A., Veer, S., Li, B.-Y., Scherer, S. A., & Pavone, M. (2025) · arXiv.org · doi:10.48550/arxiv.2509.18626 · Cited by 4
“Yet another category of approaches, that fall within the sub-field of explainable AI, furnish human-understandable “reasoning” that the model plausibly underwent for generating a particular outcome [13].”
Agent Allocation of Moral Decisions in Human-Agent Teams: Raise Human Involvement and Explain Potential Consequences
Verhagen, R., Neerincx, M. A., & Tielman, M. L. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732157 · Cited by 3 · see citing works
“Explanation communication concerns the content and form of explanations, such as textual, visual, or hybrid [77, 84].”
Multi-Robot Support System for Fighting Wildfires in Challenging Environments: System Design and Field Test Report
Frering, L., Köfler, A., Huber, M., Pfister, S., Feischl, R., Almer, A., & Steinbauer-Wagner, G. (2023) · IEEE International Symposium on Safety, Security and Rescue Robotics · doi:10.1109/ssrr59696.2023.10499951 · Cited by 4
“To reason about the different goals and events and to select the tasks for the robots to perform automatically, we chose the BDI architecture [16] because it has a long tradition and track record of applications to multi-agent systems and naturally leads to explainability [17].”
Toward Neurocognitive-Inspired Intelligence: From AI’s Structural Mimicry to Human-Like Functional Cognition
Golilarz, N. A., Khatib, H. S. A., & Rahimi, S. (2026) · IEEE Access · doi:10.1109/access.2026.3689754 · Cited by 4
Generating trustworthy and context-aware explanations for autonomous robots using an LLM agent-based RAG architecture
Fernández-Becerra, L., Guerrero-Higueras, Á., Rodríguez-Lera, F. J., & Olivera, V. M. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2026.132631 · Cited by 3 · see citing works
Stable Heteroclinic Channel-Based Movement Primitives: Tuning Trajectories Using Saddle Parameters
Rouse, N. A., & Daltorio, K. (2024) · Applied Sciences · doi:10.3390/app14062523 · Cited by 3
“Yet, they are abstract enough that the system dynamics are less prone to the “black-box”, unexplainable dynamics seen in high-dimension frameworks [20,21].”
Background
Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective
Guo, M.-Z., Yu, F., Lan, T., & Jin, F. (2023) · arXiv.org · doi:10.48550/arxiv.2309.04707 · Cited by 3
“2019), or goaldriven interpretation (Sado et al. 2023).”
Lightweight Neural Networks-Based Safety Evaluation for Smart Construction Devices
Wang, G.-M., & Zhou, J.-L. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3192552 · Cited by 3 · see citing works
“+e disadvantage of AHP is that it cannot analyze complex systems, because some layers can affect other layers, and in turn other layers can affect it [19].”
Background
Selecting Explanation Methods for Intelligent IoT Systems: A Case-Based Reasoning Approach
Parejas-Llanovarced, H., Darias, J. M., Caro-Martínez, M., & Recio-Garcia, J. A. (2023) · International Conference on Case-Based Reasoning · doi:10.1007/978-3-031-40177-0_12 · Cited by 3 · see citing works
“Sado et al. [25] review approaches on ex-plainable goal-driven intelligent agents and robots, focusing on techniques for explaining and communicating agents’ perceptual functions and cognitive reasoning.”
Background
Socially-Aware Robot Explanations: Inferring Needs from Human Facial Expressions
Kontogiorgos, D., & Shah, J. A. (2025) · Contextualizing Explanations · doi:10.64136/qdae5886 · Cited by 2
“Although AI systems can make decisions, their ability to explain them remains limited, particularly in error-prone situations [1, 4, 9, 13, 16, 17, 18, 19].”
Temporal Counterfactual Explanations of Behaviour Tree Decisions
Love, T., Andriella, A., & Alenyà, G. (2025) · arXiv.org · doi:10.48550/arxiv.2509.07674 · Cited by 2
“Explainability is an important concern in the deployment of robots in the real world, especially when these robots operate with and around humans [49].”
Influence of Different Explanation Types on Robot-Related Human Factors in Robot Navigation Tasks
Eder, M., Könczöl, C., Kienzl, J., Mosbacher, J., Kubicek, B., & Steinbauer-Wagner, G. (2024) · 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN) · doi:10.1109/ro-man60168.2024.10731192 · Cited by 2 · see citing works
“Building on previous research in Explainable Autonomous Robots (XAR) [12] and Goal-Driven XAI [13], this paper explores the impact of explanation types on common robot-related human factors used in HRI.”
Method
Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarios
Schroeter, N., Cruz, F., & Wermter, S. (2022) · arXiv.org · doi:10.48550/arxiv.2211.12930 · Cited by 2
“Over recent years, explainability in robotic systems has gained traction as a new ﬁeld of research [Kerzel et al. , 2022] [Sado et al. , 2022].”
BackgroundMethodSelf-citation
A Neurosymbolic Multi-Agent System for Explainable and Traceable Decision-Making in Organizational Contexts
Mertzani, A., Giannopoulos, P. G., Malamas, V., Chountalas, P. T., & Dasaklis, T. K. (2026) · IEEE Access · doi:10.1109/access.2026.3695444 · Cited by 2
“These design choices are consistent with findings from explainable goal-driven agent research, which shows that transparency and trust arise from modular agent decomposition, explicit internal representations and inspectable reasoning workflows rather than post-hoc explanation of opaque models [55].”
Because we have LLMs, we Can and Should Pursue Agentic Interpretability
Kim, B., Hewitt, J., Nanda, N., Fiedel, N., & Tafjord, O. (2025) · arXiv · doi:10.48550/arxiv.2506.12152 · Cited by 1 · see citing works
“…is seen in steps like automated evaluation, simulated human feedback, and automated red-teaming (Bai et al., 2022; Perez et al., 2022; Song et al., 2025), automated prompt tuning (Shin et al., 2020; Zhou et al., 2023), and automated interpretability (Rott Shaham et al., 2024; Sado et al., 2023).”
From Sensor-Empowered Ubiquitous Computing to Embodied Intelligence: Architectures, Paradigm Evolution, and Emerging Challenges
Jia, A., Cai, Z.-W., Liu, X.-Y., Zheng, K.-C., & Liu, J. (2026) · Italian National Conference on Sensors · doi:10.3390/s26144352 · Cited by 1 · see citing works
“…[149] Sensor Roles Passive data collectors, peripheral or core components [29] Active, goal-driven sensors integrated with the Action module [150] Intelligence Types Data-driven, task-specific [2] Knowledge-driven, generalizable autonomous intelligence [151] Interaction Modes Indirect,…”
Explainable Artificial Intelligence (XAI) for Identifying the Integration of International Students in the Host Country and Its Culture
Vakilian, J., Din, F. U., Sadgrove, E. J., Haghighat, M., & Shafiabady, N. (2026) · Applied Informatics · doi:10.3390/ai7070238 · Cited by 1
“…providing insights into the decision-making processes of AI models, XAI enables us to identify variables associated with international students’ self-rated integration, such as language proficiency, social network composition, participation in cultural events, and academic support systems [24,25].”
Agentic AI and Explainable CADx: Trends, Methods, and Emerging Directions in Clinical Cancer Diagnosis
Xing, H., Rehman, A., Hussain, M., Mastoi, M. S., Gulzar, N., & Khan, M. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10699-0
Applications of human-machine collaborative decision-making: A review of research with recent developments
Yun, L., Hao, Y., Gong, H., & Li, B. (2026) · iScience · doi:10.1016/j.isci.2026.116056 · Cited by 1 · see citing works
Neurosymbolic Explanation Selection in Robotics: Combining the Strengths of Planning and Foundation Models for XAI
Wachowiak, L., Coles, A., Çeliktutan, O., & Canal, G. (2026) · HRI Companion · doi:10.1145/3776734.3794387 · Cited by 1
Multi-User Personalisation in Human-Robot Interaction: Using Quantitative Bipolar Argumentation Frameworks for Preferences Conflict Resolution
Civit, A., Andriella, A., Sierra, C., & Alenyà, G. (2025) · arXiv.org · doi:10.48550/arxiv.2511.03576 · Cited by 1
“Firstly, their “black-box” nature makes it difficult to explain why a particular decision was made, which is a critical factor for establishing trust and acceptance in HRI [16], [17].”
An Appraisal-Based Approach to Human-Centred Explanations
Somarathna, R., Perera, M., Gedeon, T., & Adcock, M. (2025) · arXiv.org · doi:10.48550/arxiv.2508.01388 · Cited by 1
“Incorporating normative significance into XAI frameworks is crucial for explaining whether system behaviour aligns with personal ideals, socially accepted norms, or beliefs [1], [9], [22].”
Gamification Frameworks for Behavioural Change Using Psychophysiological Feedback
Salt, A., Nasri, S. A. E. M., & Kavakli-Thorne, M. (2025) · International Conference Virtual and Augmented Reality Simulations · doi:10.1109/icvars66454.2025.11198615 · Cited by 1 · see citing works
“In gamified learning environments, NPCs can serve as social stand-ins, enhancing social connectedness and supporting task completion [38, 39].”
Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain
Wang, C., Calandra, R., & Klös, V. (2025) · IEEE RO-MAN · doi:10.1109/ro-man63969.2025.11217662 · Cited by 1 · see citing works
“[4] survey techniques that expose agent beliefs and plans to users, while [10] integrate epistemic reasoning into planning, to explain assumptions and failures in uncertain environments.”
Collision Probability Distribution Estimation via Temporal Difference Learning
Steinecker, T., Luettel, T., & Maehlisch, M. (2024) · 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10920103 · Cited by 1
“Interpretability is crucial in safety-critical robotic systems to ensure reliable and transparent decision-making [20], [21].”
Interpretation of Neural Network Players for a Generalized Divide the Dollar Game Using SHAP Values
Greenwood, G., Abbass, H., & Hussein, A. (2023) · IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci52147.2023.10371984 · Cited by 1 · see citing works
“A recent review showed that there is a scarcity of XAI studies in multi-agent systems [4].”
Background
Autonomous Conscious Humanoid Robot System
Zhang, Z., Yu, H., Wang, W.-T., & Luo, Y. (2023) · International Conference on Robotics, Intelligent Control and Artificial Intelligence · doi:10.1109/ricai60863.2023.10489092 · Cited by 1 · see citing works
“In recent years, with the development of artificial intelligence (AI), intelligent robots have been widely used in various fields [1], such as housekeeping robot [2], medical service robot [3], emotional communication [4] and so on [5].”
Background
Trust Me, I’m AI: A User-Centric Review of Human–Machine Interaction and Teaming With Explainable AI
Jirak, D., Maes, P., & Rooy, D. V. (2026) · IEEE Access · doi:10.1109/access.2026.3682940 · Cited by 1 · see citing works
“For the interested reader on this topic, [102], [103] are recent reviews on the topic of XAI employment in robotics.”
How Can AI Recognize Pain and Express EmpathyPreprint
Cao, S., Fu, D., Yang, X., Barros, P. V. A., Wermter, S., Liu, X., & Wu, H. (2021) · arXiv.org · doi:10.48550/arxiv.2110.04249 · Cited by 1 · see citing works
Self-citation
Learning Adaptive Deliberation Control for Reliable Recognition with Multi-Agent Language Models
Liu, Y., Peng, H.-Y., Hu, F.-L., Huang, L., & Cai, W.-L. (2026) · doi:10.48550/arxiv.2610.17840
“Goal-driven and collaborative frameworks define roles and workflows [10, 11], while model routing selects the models assigned to them [12].”
Effects of Transparent Robot Intentions on Human Trust and Compliance in Decision-Making: The Role of Benevolent and Ulterior Intentions of Robots
Schäfer, A., & Kubicek, B. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2732835
AI Agents and Agentic AI in plant breeding: new frontiers, opportunities and challenges
Vitale, P., Acharya, D. B., Gerard, G., Dreisigacker, S., Montesinos-López, O. A., Montesinos‐López, A., Ortiz, R., & Crossa, J. L. (2026) · Frontiers in Genetics · doi:10.3389/fgene.2026.1958764
“AI Agents are typically oriented toward specific objectives or tasks and operate within a confined environment (Sado et al., 2023).”
Behaviora - A Conceptual Architecture for External and Internal Behavior of Robots and Agents
Nyman, G. (2026) · doi:10.48550/arxiv.2609.27536
“Planning and goal-based systems organize intended action and seek to make the reasoning behind goal-directed behavior understandable [5].”
From Model Explainability to Operational Transparency in Agentic AI: A Transparency-by-Design Framework for Critical Energy Systems Under the EU AI Act
Jørgensen, B., & Ma, Z. (2026) · Algorithms · doi:10.3390/a19090733
Intelligent agents: architectures, learning paradigms, deployment models, and applications
Dritsas, E., & Trigka, M. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01259-3
Should I Say It or Write It Down? How Explanation Type and Delivery Modality Affect Human Perception of Navigating Robots
Tiourine, D., Epstein, S. L., & Korpan, R. (2026) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/ro-man72286.2026.11714594
“Explainable HRI research has explored what constitutes a “good” explanation, emphasizing layered, context-sensitive communication that aligns robot processes with human mental models [5], [6].”
Influential
Artificial Intelligence and Liability: Exploring Legal Challenges and Responsibility in AI Decision-Making and Autonomous Systems
Afuwape, K. (2026) · Digital Law Journal · doi:10.38044/2686-9136-2025-6-17
Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration
Fernandes, M., & Sardinha, A. (2026) · Adaptive Agents and Multi-Agent Systems · doi:10.65109/hdmg2174
“Explainable artificial intelligence (XAI) research offers promising approaches to bridge this gap [12], yet systematic evaluation of such techniques in collaborative contexts remains limited.”
Learning Preferences and Resolving Conflicts in Multi-User Personalisation in Human-Robot Interaction
Civit, A. (2026) · Adaptive Agents and Multi-Agent Systems · doi:10.65109/tbuz5275
“RQ.4 Contestability : Robots must explain their decisions in a manner that users can clearly understand them, so they can provide explicit feedback to influence the robot’s decisions or beliefs [27, 34].”
Towards Design Guidelines for Natural-Language-Based Human-Multi-Robot Collaboration in Domestic Environments
Wang, X.-Y., Hsia, S.-K., Liu, Z.-Y., Zhu, C., Zhu, Z.-Z., Hu, X.-Y., Ostrowski, A. K., & Ramani, K. (2026) · CHI Extended Abstracts · doi:10.1145/3772363.3798976
Unifying Explainable Agency Via a Ladder of Intentions
Giménez-Ábalos, V., Tormos, A., Vázquez-Salceda, J., Edström, F. E., Brännström, M., Lindqvist, J., Cortés, U., & Álvarez-Napagao, S. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10710-w
“These are mostly based on symbolic foundations of some sort, such as inherently explain-able planning agents (XAIP) (Chakraborti et al., 2018, 2017; Sado et al., 2023) or agents with an explicit BDI architecture (Winikoff & Sidorenko, 2023 ; Winikoff et al., 2018).”
Causal Explanations for Robot Decisions and Beliefs
Love, T., Andriella, A., & Alenyà, G. (2026) · HRI Companion · doi:10.1145/3776734.3794634
“There are many different approaches for automatically generating explanations in explainable artificial intelligence (XAI) [21] and explainable robotics [19] contexts, relying on different formulations of explanations.”
A Survey on Quantitative Modeling of Trust in Online Social Networks
Song, W., & Barber, K. (2026) · arXiv.org · doi:10.48550/arxiv.2603.11054
“There are literature reviews on trust modeling in various # elds, including agent systems [132], cybersecurity [56], the Internet of Things [5, 41], and online behavior research [54, 186].”
Towards Natural Language Environment: Understanding Seamless Natural-Language-Based Human-Multi-Robot Interactions
Liu, Z.-Y., Wang, X.-Y., Hsia, S.-K., Zhu, C., Zhu, Z.-Z., Hu, X.-Y., Ostrowski, A. K., & Ramani, K. (2026) · arXiv.org · doi:10.48550/arxiv.2601.13338
“This is supported by recent research advocating for explainable AI [31, 101], which emphasizes that transparency in decision-making is essential for fostering user trust.”
KG-Retailbot: A Knowledge Graph-Based Chatbot for Explaining Robotic Scenario Information in a Retail Setting
Xu, K., Yuan, S., Dogramadzi, S., & Corbato, C. H. (2026) · International Journal of Social Robotics · doi:10.1007/s12369-025-01335-1
The Influence of Human-Like Appearance on Expected Robot Explanations
Kopecka, H., & Such, J. M. (2025) · IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/ro-man72286.2026.11714567
“…transparency, fairness, and accountability [4], but also to help in building an appropriate mental model of the robot and its skills to support human-robot collaboration and prevent failure [5], and to help address situations in which the robot fails to achieve its goal or behaves unexpectedly [3].”
Enabling Transparent Cyber Threat Intelligence Combining Large Language Models and Domain Ontologies
Cotti, L., Rula, A., Bianchini, D., & Cerutti, F. (2025) · XAI-KRKG/UCEC-XAI@ECAI · doi:10.48550/arxiv.2509.00081
“AI Agents can be defined as autonomous software entities, typically based on LLMs, engineered for goal-directed task execution within bounded digital environments [26, 27, 28].”
A Design-Driven Taxonomy of AI Agentic Patterns
Joyce, J. E., & Maheshwari, S. (2025) · IEEE International Conference on Electronics, Computing and Communication Technologies · doi:10.1109/conecct65861.2025.11306519
“C. Goal-Oriented Agent Pattern (Planning & Decision-Making) [14] These are designed to achieve specific objectives by reasoning about possible actions and selecting the best course.”
Logistics Path Optimization and Intelligent Scheduling System Based on Deep and Reinforcement Learning
Ren, N. (2025) · 2025 International Conference on Engineering Innovations and Technologies (ICoEIT) · doi:10.1109/icoeit63558.2025.11211667
“The algorithm updates the network state by the attention weights between nodes, and is particularly suitable for dealing with complex relationships, large-scale graph data, and node importance problems [14-15].”
Quantity forecast of mobile subscribers with Time-Dilated Attention
Yao, B. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2024.103940
Towards Explainability of Affordance Learning in Robot Vision
Mirnateghi, N., Islam, S. M. S., & Shah, S. A. A. (2024) · International Conference on Digital Image Computing: Techniques and Applications · doi:10.1109/dicta63115.2024.00085
“Visual XAI approaches focus on some visual cues that represent primitive explanations of the model’s internal mechanics [7].”
Influential
Exploring Stanislavskian Performance for Agent-based Nonplayer Characters through Defeasible Logic
Mitchell, K., & McCoy, J. (2024) · International Conference on Intelligent Virtual Agents · doi:10.1145/3652988.3673953
Computational implementation of an explainable state machine-based agent
Viana, H., Tacla, C., & Simão, J. (2023) · Anais do XX Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2023) · doi:10.5753/eniac.2023.234005
Leveraging text data in healthcare: A systematic review of AI-driven emotional analysis and patient-centric innovations
Nag, P. K., Bhagat, A., Priya, R., Malviya, S., & Mishra, S. (2026) · AIP Conference Proceedings · doi:10.1063/5.0322638

Authors: Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S.
Affiliations: University of Malaya, Kuala Lumpur, Malaysia
University of Malaya (MY)
Universität Hamburg (DE)
ORCID: Sado, Fatai
Loo, Chu Kiong
Liew, Wei Shiung
Kerzel, Matthias
Wermter, Stefan
Venue: ACM Computing Surveys
Publisher: Association for Computing Machinery (ACM)
Published: 2023-02-02
Volume: 55
Issue: 10
Pages: 1-41
ISSN: 0360-0300 (ISSN-L)
1557-7341
DOI: 10.1145/3564240
OpenAlex: W4308804331
Cited by: 79
References: 237
Citations by year: 2026: 23 · 2025: 19 · 2024: 23 · 2023: 12 · 2021: 2
FWCI: 4.44
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Computing Surveys (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: Georg Forster Research Fellowship for Experienced Researchers
Alexander von Humboldt-Stiftung/Foundation and Impact Oriented Interdisciplinary Research (IIRG002C-19HWB)
University of Malaya
German Research Foundation (TRR 169)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords: explainable AI, robots, trust in AI, human-robot interaction, perceptual functions, cognitive reasoning, human-in-the-loop, transparency, comprehensibility, continual learning, safety-critical applications, black box opacity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Explainable goal-driven agents and robots-a comprehensive review

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Record

About these checks