Too much, too little, or just right? Ways explanations impact end users' mental models
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Research is emerging on how end users can correct mistakes their intelligent agents make, but before users can correctly “debug” an intelligent agent, they need some degree of understanding of how it works. In this paper we consider ways intelligent agents should explain themselves to end users, especially focusing on how the soundness and completeness of the explanations impacts the fidelity of end users' mental models. Our findings suggest that completeness is more important than soundness: increasing completeness via certain information types helped participants' mental models and, surprisingly, their perception of the cost/benefit tradeoff of attending to the explanations. We also found that oversimplification, as per many commercial agents, can be a problem: when soundness was very low, participants experienced more mental demand and lost trust in the explanations, thereby reducing the likelihood that users will pay attention to such explanations at all.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“rule-based methods have the advantage of interpretability [2,51,55,60,61].”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [51]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Kulesza et al. [31] found that people with more complete mental models were able to collaborate more effectively with a recommendation system.”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as Kulesza et al. [31]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
““Too Much, Too Little, or Just Right?,” a work proposed by Kulesza et al. [182] presented their findings to reveal how explanations are impacting end users' mental models.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [182]
The cited source says what the claim attributes to it.
In the citing paper
“Some studies found that explanations increase trust [4,8], some found no significant effect [15,42,97], while others found varying effects under different conditions [49,51].”
From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“To select specific songs , our prototype used a bagged decision tree based on Weka’s J48 implementation [7] (the bagging ensemble consisted of 100 decision trees).”
MethodMental Models: Towards a Cognitive Science of Language, Inference, and Consciousness
Ford, M., & Johnson-Laird, P. N. (1985) · Language · doi:10.2307/414498 · Cited by 4,419 · cited 1×
“These models allow people to understand, explain, and predict phenomena [9], and to then act accordingly.”
BackgroundA Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,602 · cited 1×
“Our prototype used a hybrid recommendation approach, as such approaches have been shown to out-perform more traditional types of recommenders [17] and provide more “moving parts” to explain.”
MethodAccurate methods for the statistics of surprise and coincidence
Dunning, T. E. (1993) · doi:10.5555/972450.972454 · Cited by 2,647
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“For example, when Herlocker et al. [8] evaluated 21 ways of explaining collaborative filtering systems, they investigated only low fidelity explanations because they wanted to “avoid creating a new kind of information overload by presenting too much or too confusing data”.”
BackgroundSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Grounded Theory Methodology: The Pressing Need for a Coherent Logic of Justification
Rennie, D. L. (1998) · Theory & Psychology · doi:10.1177/0959354398081006 · Cited by 1,434
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 1×
“They later showed that one of their intelligibility types can help end users better understand the output of decision trees [13].”
BackgroundThe effects of transparency on trust in and acceptance of a content-based art recommender
Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-008-9051-3 · Cited by 448 · cited 2×
“Cramer et al. found that providing explanations of why an intelligent agent believes two items are similar helped participants believe they understood how the agent worked, but their actual mental models were not measured [4].”
Background(1987) · Cited by 378
Designing and Evaluating Explanations for Recommender Systems
Tintarev, N., & Masthoff, J. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_15 · Cited by 322 · cited 1×
“Other researchers have also investigated explanation fidelity in recommender systems (Tintarev and Masthoff provide a comprehensive overview in [18]), but primarily as it relates to trust, acceptance, or satisfaction.”
BackgroundUsing neural networks for data mining
Craven, M. W., & Shavlik, J. W. (1997) · Future Generation Computer Systems · doi:10.1016/s0167-739x(97)00022-8 · Cited by 257 · cited 1×
“For the medium-soundness treatment (MM), we trained a simpler model (a single J48 decision tree) using the bagged classifier’s predicted labels for all of the training instances, and explained this derived model (a variation of the technique in [5]).”
MethodAssessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · cited 2×
“To objectively manipulate completeness, our treatments used a varying number of the intelligibility types identified by Lim and Dey [12]: inputs (features the system is aware of), model (an overview of the agent’s decision making process), why (the reasons underlying a specific decision), and…”
BackgroundMethodFirst steps in programming: a rationale for attention investment models
Blackwell, A. F. (2003) · Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments · doi:10.1109/hcc.2002.1046334 · Cited by 234 · cited 2×
“On the other hand, the Attention Investment Model predicts that people will still use high-cost explanations if they perceive the benefits will outweigh the costs (e.g., time) and risks (e.g., no reward) of doing so [1].”
BackgroundTell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · cited 4×
“As with other kinds of end-user debugging, users’ mental models of how an agent works help them decide exactly what about the agent they need to correct, and how to go about it [11].”
BackgroundInfluentialSelf-citationHow it works: a field study of non-technical users interacting with an intelligent system
Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240630 · Cited by 137 · cited 1×
“…agent that predicted office workers’ availability, finding that even when faced with contradictory evidence, users often tried to fit it into their existing mental model of the agent [19]; this suggests that it is important to help end users develop correct models as quickly as possible.”
BackgroundHarnessing curiosity to increase correctness in end-user programming
Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C., Durham, M., & Rothermel, G. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642665 · Cited by 109 · cited 1×
“One successful way to communicate the benefits of invested attention is the Surprise-Explain-Reward method [20], which leverages curiosity by surprising users (e.g., showing odd values to test spreadsheet formulas), then explains the benefits of the behavior it is trying to encourage (e.g., fewer…”
BackgroundSelf-citationAre explanations always important?: a study of deployed, low-cost intelligent interactive systems
Bunt, A., Lount, M., & Lauzon, C. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166996 · Cited by 106 · cited 1×
“For example, Bunt et al. [2] found that when users had no direct control over an agent’s reasoning, user interest in any type of explanation was very low.”
BackgroundMey, G. (2024) · Springer eBooks · doi:10.1007/978-3-658-45155-4_18 · Cited by 84
Smalltalk scaffolding: a case study of minimalist instruction
Rosson, M. B., Carrol, J. M., & Bellamy, R. K. E. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97319 · Cited by 76 · cited 1×
“…model [3]—which minimizes passive learning tasks, such as reading long explanations, favoring instead short explanations coupled with try-it-out exploration—helped programmers understand Smalltalk programs up to two orders of magnitude faster than traditional instruction techniques [15].”
BackgroundHarnessing curiosity to increase correctness in end-user programming
Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C. R., Durham, M., & Rothermel, G. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642663.642665 · Cited by 33
Self-citationWhere Are My Intelligent Assistant’s Mistakes? A Systematic Testing Approach
Kulesza, T., Burnett, M., Stumpf, S., Wong, W., Das, S., Groce, A., Shinsel, A., Bice, F., & McIntosh, K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21530-8_14 · Cited by 20 · cited 1×
“Although research has shown that certainty is a useful intelligibility type to users assessing an intelligent agent’s reliability [10], other researchers have found that certainty does not help users’ perceived understanding of how a recommender operates [4].”
BackgroundSelf-citation
Cited by
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“[161] where they presented their findings regarding how explanations impact end users’ mental models.”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
““Too Much, Too Little, or Just Right?,” a work proposed by Kulesza et al. [182] presented their findings to reveal how explanations are impacting end users’ mental models.”
BackgroundAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Hence, in addition to tuning explanation attributes such as soundness and completeness [39], or faithfulness, sensitivity, and complexity [5], explainable AI researchers should ensure that people will exert effort to attend to those explanations.”
Background
Metadata
- Authors
- Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K.
- Affiliations
- Oregon State University (US)
- City, University of London (GB)
- Oregon Institute of Technology (US)
- Venue
- 2013 IEEE Symposium on Visual Languages and Human Centric Computing
- Publisher
- IEEE
- Published
- 2013-09
- Pages
- 3-10
- ISSN
- 1943-6092 (ISSN-L)
- OpenAlex
- W1994606570
- Cited by
- 388
- References
- 22
- Citations by year
- 2026: 40 · 2025: 38 · 2024: 51 · 2023: 44 · 2022: 56 · 2021: 55 · 2020: 38 · 2019: 23 · 2018: 20 · 2017: 7 · 2016: 7 · 2015: 5 · 2014: 3
- FWCI
- 3.23
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- intelligent agents, explanation fidelity, mental demand, cost-benefit trade-off, oversimplification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.