Mental Models of Mere Mortals with Explanations of Reinforcement Learning
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
How should reinforcement learning (RL) agents explain themselves to humans not trained in AI? To gain insights into this question, we conducted a 124-participant, four-treatment experiment to compare participants’ mental models of an RL agent in the context of a simple Real-Time Strategy (RTS) game. The four treatments isolated two types of explanations vs. neither vs. both together. The two types of explanations were as follows: (1) saliency maps (an “Input Intelligibility Type” that explains the AI’s focus of attention) and (2) reward-decomposition bars (an “Output Intelligibility Type” that explains the AI’s predictions of future types of rewards). Our results show that a combined explanation that included saliency and reward bars was needed to achieve a statistically significant difference in participants’ mental model scores over the no-explanation treatment. However, this combined explanation was far from a panacea: It exacted disproportionately high cognitive loads from the participants who received the combined explanation. Further, in some situations, participants who saw both explanations predicted the agent’s next action worse than all other treatments’ participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776 · cited 2×
“When trying to solve problems (in this case, figure out an AI with the help of explanations), people can experience at least three types of cognitive load: intrinsic, germane, and extraneous [20, 24, 49, 53].”
BackgroundNASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 1×
“…physical demand, temporal demand, effort, success, and frustration, which allows researchers to understand the tradeoffs that participants experience The TLX has been used by a variety of researchers around the world, even 20 years after its creation, speaking to its effectiveness as a tool [11].”
BackgroundNew Effect Size Rules of Thumb
Sawilowsky, S. S. (2009) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1257035100 · Cited by 3,254
Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions
van Merriënboer, J. J. G., & Sweller, J. (2005) · Educational Psychology Review · doi:10.1007/s10648-005-3951-0 · Cited by 2,206
Self‐Explanations: How Students Study and Use Examples in Learning to Solve Problems
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989) · Cognitive Science · doi:10.1207/s15516709cog1302_1 · Cited by 1,714
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Self-citationGuidance during mathematical problem solving.
Tarmizi, R. A., & Sweller, J. (1988) · Journal of Educational Psychology · doi:10.1037/0022-0663.80.4.424 · Cited by 464
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · cited 2×
“looked at the impact of two explanation properties, completeness and soundness, on their participants’ mental models in a music recommendation system powered by a hybrid AI agent [26].”
BackgroundSelf-citation
Cited by
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1096257 · Cited by 272
“Compelling evidence of the value of task reflection comes from a set of studies that used methods of knowledge elicitation for the evaluation of explainable AI systems (Anderson et al., 2020; Dodge et al., 2021; Khanna et al., 2021; Tabatabai et al., 2021).”
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262
Zhang, Z., Jia, M., Lee, H.-P. (., Yao, B., Das, S., Lerner, A., Wang, D., & Li, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642385 · Cited by 95
“Anderson et al. [1] found differences in users’ mental models of reinforcement learning when prompted with varying explanations, where the excessive explanations increased cognitive load.”
Jung, J., Lee, H., Jung, H., & Kim, H. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16110 · Cited by 87
“Furthermore, although the mental model is one of the most important criteria for measurement of explanation effectiveness, particularly when using XAI techniques on users who were unfamiliar with an XAI model, it was not considered in any studies [52,55].”
MethodExploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86
“Existing explanation styles are limited in addressing the multifaceted user experience in high-stake contexts such as healthcare, which requires a further understanding of the user’s mental model and the explanation goal when generating and presenting explanations to the users [2].”
Background
Metadata
- Authors
- Anderson, A., Dodge, J., Sadarangani, A., Juozapaitis, Z., Newman, E., Irvine, J., Chattopadhyay, S., Olson, M., Fern, A., & Burnett, M.
- Venue
- ACM Transactions on Interactive Intelligent Systems
- Published
- 2020-05-30
- Volume
- 10
- Issue
- 2
- Pages
- 1-37
- ISSN
- 2160-6455 (ISSN-L)
- 2160-6463
- OpenAlex
- W2990540492
- Cited by
- 63
- References
- 11
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mental Models of Mere Mortals with Explanations of Reinforcement Learning. https://app.cochat.ai/papers/10.1145/3366485
- The paper
- Anderson, A., Dodge, J., Sadarangani, A., Juozapaitis, Z., Newman, E., Irvine, J., Chattopadhyay, S., Olson, M., Fern, A., & Burnett, M. (2020). Mental Models of Mere Mortals with Explanations of Reinforcement Learning. ACM Transactions on Interactive Intelligent Systems. https://doi.org/10.1145/3366485
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.