Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
With the growing capabilities of intelligent systems, the integration of robots in our everyday life is increasing. However, when interacting in such complex human environments, the occasional failure of robotic systems is inevitable. The field of explainable AI has sought to make complex-decision making systems more interpretable but most existing techniques target domain experts. On the contrary, in many failure cases, robots will require recovery assistance from non-expert users. In this work, we introduce a new type of explanation, εerr, that explains the cause of an unexpected failure during an agent's plan execution to non-experts. In order for error explanations to be meaningful, we investigate what types of information within a set of hand-scripted explanations are most helpful to non-experts for failure and solution identification. Additionally, we investigate how such explanations can be autonomously generated, extending an existing encoder-decoder model, and generalized across environments. We investigate such questions in the context of a robot performing a pick-and-place manipulation task in the home environment. Our results show that explanations capturing the context of a failure and history of past actions, are the most effective for failure and solution identification among non-experts. Furthermore, through a second user evaluation, we verify that our model-generated explanations can generalize to an unseen office environment, and are just as effective as the hand-scripted explanations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
Explainable AI: From black box to glass box
Rai, A. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00710-5 · Cited by 1,317
HUMAN–ROBOT COLLABORATION: A SURVEY
BAUER, A., WOLLHERR, D., & BUSS, M. (2008) · International Journal of Humanoid Robotics · doi:10.1142/s0219843608001303 · Cited by 493 · cited 1×
“is often required to correct the problem [7], and co-located users – homeowners, medical lab technicians, and warehouse workers – will be frst on the scene.”
BackgroundBeyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213
Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302316 · Cited by 210 · cited 5×
“nations introduced by [20] to generate context-based history explanations within a continuous state-space.”
BackgroundMethodInfluentialA Reinforcement Learning Framework for Explainable Recommendation
Wang, X., Chen, Y., Yang, J., Wu, L., Wu, Z., & Xie, X. (2018) · 2018 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2018.00074 · Cited by 163
Recovering from failure by asking for help
Knepper, R. A., Tellex, S., Li, A., Roy, N., & Rus, D. (2015) · Autonomous Robots · doi:10.1007/s10514-015-9460-1 · Cited by 76
Cited by
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
“It is imperative to follow human-centered approaches to understand the “personal, social, and cultural aspects” [71] of the recipients of AI explanations, especially since a monolithic view of the who may inadvertently risk dehu-manization [23, 81].”
BackgroundYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85
Explainable Activity Recognition for Smart Home Systems
Das, D., Nishimura, Y., Vivek, R. P., Takeda, N., Fish, S. T., Plötz, T., & Chernova, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3561533 · Cited by 70
“Prior work demonstrates that taking a design-centric approach, in which we first understand non-experts’ preferences towards explanations in a given domain, can aid in generating more meaningful explanations for our target audience [13, 16].”
InfluentialSelf-citationCommunicating Missing Causal Information to Explain a Robot’s Past Behavior
Han, Z., & Yanco, H. (2023) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3568024 · Cited by 11
“[24] used an encoder-decoder deep learning model to generate robot failure explanations that consider causal knowledge from the current environment, i.”
Method
Metadata
- Authors
- Das, D., Banerjee, S., & Chernova, S.
- Venue
- Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction
- Publisher
- ACM
- Published
- 2021-03-08
- Pages
- 351-360
- OpenAlex
- W3134655704
- Cited by
- 107
- References
- 11
- FWCI
- 9.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery. https://app.cochat.ai/papers/10.1145/3434073.3444657
- The paper
- Das, D., Banerjee, S., & Chernova, S. (2021). Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery. Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction. https://doi.org/10.1145/3434073.3444657
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.