Are explanations always important?: a study of deployed, low-cost intelligent interactive systems
The DOI matches its Crossref record; 27 of the 28 references it cites were found in a registry or an index.
Abstract
Intelligent interactive systems (IIS) have great potential to improve users' experience with technology by tailoring their behaviour and appearance to users' individual needs; however, these systems, with their complex algorithms and dynamic behaviour, can also suffer from a lack of comprehensibility and transparency. We present the results of two studies examining the comprehensibility of, and desire for explanations with deployed, low-cost IIS. The first study, a set of interviews with 21 participants, reveals that i) comprehensibility is not always dependent on explanations, and ii) the perceived cost of viewing explanations tends to outweigh the anticipated benefits. Our second study, a two-week diary study with 14 participants, confirms these findings in the context of daily use, with participants indicating a desire for an explanation in only 7% of diary entries. We discuss the implications of our findings for the design of explanation facilities.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They found that although these systems were opaque and provided no reasoning behind their predictions, they were still positively perceived by the users.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [92]
In the citing paper
“Providing system explanations may also undermine user perceptions when users lack the attentional capacity to process complex explanations, for example when executing a demanding task [1,12,43,85].”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [12]
In the citing paper
“For these users who preferred the non-transparent version, it seemed that incremental feedback was providing more information than they required [12,43].”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [12]
In the citing paper
“For these users who preferred the nontransparent version, it seemed that incremental feedback was providing more information than they required [12,42].”
From Progressive disclosure (2019) · cites this paper as [12]
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
GroupLens: an open architecture for collaborative filtering of netnews
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., & Riedl, J. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.192905 · Cited by 5,058
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 3,229
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 2×
“, [2, 8]), require time and effort on behalf of the user to consume, and may even lead to lower decision quality for some users [5].”
BackgroundMethodSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“, [4, 7, 8, 11, 15, 25]).”
Intelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552
Explanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535 · cited 7×
“According to Gregor and Benbasat s survey of a number of studies in this domain [7], explanations can improve user performance, helping users make faster and better decisions.”
BackgroundInfluentialDirect manipulation vs. interface agents
Shneiderman, B., & Maes, P. (1997) · Interactions · doi:10.1145/267505.267514 · Cited by 518
Recommendation Agents for Electronic Commerce: Effects of Explanation Facilities on Trusting Beliefs
Wang, W., & Benbasat, I. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222230410 · Cited by 498
An analysis of physician attitudes regarding computer-based clinical consultation systems
Teach, R. L., & Shortliffe, E. H. (1981) · Computers and Biomedical Research · doi:10.1016/0010-4809(81)90012-4 · Cited by 328
Designing and Evaluating Explanations for Recommender Systems
Tintarev, N., & Masthoff, J. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_15 · Cited by 322
CoScripter: automating & sharing how-to knowledge in the enterprise
Leshed, G., Haber, E. M., Matthews, T., & Lau, T. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357323 · Cited by 311
Trust building with explanation interfaces
Pu, P., & Chen, L. (2006) · Proceedings of the 11th international conference on Intelligent user interfaces · doi:10.1145/1111449.1111475 · Cited by 300
Assessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246
Toward establishing trust in adaptive agents
Glass, A., McGuinness, D. L., & Wolverton, M. (2008) · Proceedings of the 13th international conference on Intelligent user interfaces · doi:10.1145/1378773.1378804 · Cited by 211 · cited 1×
“[6] performed a field study evaluating information needs and perceptions of trust of the CALO system, a complex task assistant for a range of office-related tasks.”
BackgroundInteracting meaningfully with machine learning systems: Three experiments
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., & Herlocker, J. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.03.004 · Cited by 184
An open architecture for collaborative filtering of netnews
Resnick, P. (1994) · Computer Supported Cooperative Work (CSCW) · Cited by 150
How it works: a field study of non-technical users interacting with an intelligent system
Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240630 · Cited by 137
Cheverst, K., Byun, H. E., Fitton, D., Sas, C., Kray, C., & Villar, N. (2005) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-005-1269-8 · Cited by 81 · cited 1×
“, [3, 13, 14, 24]).”
Selection criteria for expert system shells: a socio-technical framework
Stylianou, A. C., Madey, G. R., & Smith, R. D. (1992) · Communications of the ACM · doi:10.1145/135239.135240 · Cited by 61
An Evaluation of Explanations of Probabilistic Inference
Suermondt, H. J., & Cooper, G. F. (1993) · Computers and Biomedical Research · doi:10.1006/cbmr.1993.1017 · Cited by 50
Understanding and Dealing With Usability Side Effects of Intelligent Processing
Jameson, A. (2009) · AI Magazine · doi:10.1609/aimag.v30i4.2274 · Cited by 47
Taking advice from intelligent systems: the double-edged sword of explanations
Ehrlich, K., Kirk, S. E., Patterson, J., Rasmussen, J. C., Ross, S. I., & Gruen, D. M. (2011) · Proceedings of the 16th international conference on Intelligent user interfaces · doi:10.1145/1943403.1943424 · Cited by 35
Computer applications in medical care
Remoortere, P. V. (1979) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(79)90087-9 · Cited by 35
Understanding the Utility of Rationale in a Mixed-Initiative System for GUI Customization
Bunt, A., McGrenere, J., & Conati, C. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73078-1_18 · Cited by 24
Self-citation
Research works printed but not linked: 1
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
Cited by
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“[160] raised questions as to the importance of, and consequently anticipated usage of, explanation techniques within systems that support users in making low-cost decisions.”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“In the spirit of holism, Bunt et al. [92] originally raised this question in their work mainly due to the efficiency of explanation techniques deployed in systems that offer users low-cost decisions.”
BackgroundAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Metadata
- Authors
- Bunt, A., Lount, M., & Lauzon, C.
- Affiliations
- University of Manitoba (CA)
- University of British Columbia (CA)
- ORCID
- Bunt, Andrea
- Venue
- Proceedings of the 2012 ACM international conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2012-02-14
- Pages
- 169-178
- OpenAlex
- W2026805626
- Cited by
- 106
- References
- 28
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 11 · 2023: 12 · 2022: 15 · 2021: 18 · 2020: 10 · 2019: 10 · 2018: 5 · 2017: 3 · 2016: 3 · 2015: 1 · 2013: 3
- FWCI
- 0.98
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- explanation, intelligent interactive systems, comprehensibility, transparency, explanation facilities, user experience, low-cost systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.