“Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
While the research area of artificial intelligence benefited from increasingly sophisticated machine learning techniques in recent years, the resulting systems suffer from a loss of transparency and comprehensibility, especially for end-users. In this paper, we explore the effects of incorporating virtual agents into explainable artificial intelligence (XAI) designs on the perceived trust of end-users. For this purpose, we conducted a user study based on a simple speech recognition system for keyword classification. As a result of this experiment, we found that the integration of virtual agents leads to increased user trust in the XAI system. Furthermore, we found that the user’s trust significantly depends on the modalities that are used within the user-agent interface design. The results of our study show a linear trend where the visual presence of an agent combined with a voice output resulted in greater trust than the output of text or the voice output alone. Additionally, we analysed the participants’ feedback regarding the presented XAI visualisations. We found that increased human-likeness of and interaction with the virtual agent are the two most common mention points on how to improve the proposed XAI interaction design. Based on these results, we discuss current limitations and interesting topics for further research in the field of XAI. Moreover, we present design recommendations for virtual agents in XAI systems for future projects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01228-7 · Cited by 6,909
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“This information is a crucial step in establishing appropriate trust [17] in AI systems in the future.”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“One of the first approaches for creating XAI visualisations was to use gradients with respect to the input [34] to measure how much a small change in each input part would affect the prediction.”
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 4×
“Another method to determine relevance is Layerwise-Relevance Propagation [2], which uses the intermediate results of the model during the prediction to calculate the contribution of each part of the input to the over-all prediction.”
MethodInfluentialTrust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817 · cited 2×
“Hoff and Bashir [12] presented a three-layered framework, consisting of dispositional trust, situational trust, and learned trust.”
BackgroundMethods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Cited by
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective
Bach, T. A., Khan, A., Hallock, H., Beltrão, G., & Sousa, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2138826 · Cited by 377
ŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“simultaneously, and show the interplay between multimodal interactions required for building interactive explanation interfaces followed with feedback mechanism able to incorporate change in the system [89].”
BackgroundInfluential
Metadata
- Authors
- Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E.
- Venue
- Journal on Multimodal User Interfaces
- Published
- 2020-07-09
- Volume
- 15
- Issue
- 2
- Pages
- 87-98
- ISSN
- 1783-7677 (ISSN-L)
- 1783-8738
- OpenAlex
- W3041723710
- Cited by
- 119
- References
- 20
- FWCI
- 5.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design. https://app.cochat.ai/papers/10.1007/s12193-020-00332-0
- The paper
- Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2021). “Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design. Journal on Multimodal User Interfaces. https://doi.org/10.1007/s12193-020-00332-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.