Automated rationale generation: a technique for explainable AI and its effects on human perceptions
Open accessAbstract
Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent XAI research has begun to explore how to 'translate' raw features used by a model into higher-level concepts that are familiar to people [30,48].
“We note that recent XAI research has begun to explore how to”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [30]
Whether an explanation fits the context is more important than its length.
“Ehsan et al. discussed that whether an explanation that fits the context is more important than its length [9].”
From Dynamic Selection of Reliance Calibration Cues With AI Reliance Model (2023) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · see citing works
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Watkins, C., & Dayan, P. (1992) · Machine-mediated learning · doi:10.1023/a:1022676722315 · Cited by 14,277 · see citing works
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Residual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073 · see citing works
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · cited 1× · see citing works
“Human-likeness, central to generating human-centered rationales, was inspired from sociability and anthropomorphization factors from HRI work on robot acceptance [[35, 33, 34].”
BackgroundTurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,978 · see citing works
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · see citing works
A Pragmatic View of Thematic Analysis
Aronson, J. (1995) · The Qualitative Report · doi:10.46743/2160-3715/1995.2069 · Cited by 1,758 · cited 1× · see citing works
“We analyzed the open-ended justifications participants provided using a combination of thematic analysis [4] and grounded theory [38].”
MethodUnderstanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 1× · see citing works
“For instance, contestability, the ability to either reject a reasoning or ask for another one, which has shown to improve user satisfactions [16, 22] can be incorporated in the future.”
BackgroundConsciousness, accessibility, and the mesh between psychology and neuroscience
Block, N. (2007) · Behavioral and Brain Sciences · doi:10.1017/s0140525x07002786 · Cited by 1,146 · see citing works
Making Use: Scenario-Based Design of Human-Computer Interactions
Carroll, J. M. (2000) · Cited by 984 · see citing works
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · see citing works
Effective Approaches to Attention-based Neural Machine TranslationPreprint
Luong, M.-T., Pham, H., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04025 · Cited by 742 · see citing works
Two neural correlates of consciousness
Block, N. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.12.006 · Cited by 713 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · see citing works
A Description of Think Aloud Method and Protocol Analysis
Fonteyn, M. E., Kuipers, B. J., & Grobe, S. J. (1993) · Qualitative Health Research · doi:10.1177/104973239300300403 · Cited by 675 · see citing works
Nass, C., Fogg, B. J., & Moon, Y. (1996) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1996.0073 · Cited by 665 · see citing works
A fundamental attribution error? Rethinking cognitive distortions †
Maruna, S., & Mann, R. E. (2006) · Legal and Criminological Psychology · doi:10.1348/135532506x114608 · Cited by 465 · see citing works
Explanations as Mechanisms for Supporting Algorithmic Transparency
Rader, E., Cotter, K., & Cho, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173677 · Cited by 388 · see citing works
Fodor, J. A. (1995) · The MIT Press eBooks · doi:10.7551/mitpress/2693.001.0001 · Cited by 280 · see citing works
To err is robot: how humans assess and act toward an erroneous social robot
Mirnig, N., Stollnberger, G., Miksch, M., Stadler, S., Giuliani, M., & Tscheligi, M. (2017) · Frontiers in Robotics and AI · doi:10.3389/frobt.2017.00021 · Cited by 244 · see citing works
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
Ehsan, U., Harrison, B., Chan, L., & Riedl, M. O. (2017) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278736 · Cited by 225 · see citing works
Self-citationCarroll, J. M. (2000) · ACM Conference on Designing Interactive Systems (DIS) · doi:10.1145/347642.347652 · Cited by 205 · see citing works
(2010) · doi:10.5555/1734454.1734544 · Cited by 163 · see citing works
Nass, C., Steuer, J., Henriksen, L., & Dryer, D. C. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1025 · Cited by 146 · see citing works
Impact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A. M., & Yanco, H. A. (2013) · doi:10.5555/2447556.2447663 · Cited by 144 · see citing works
Attitudinal and Intentional Acceptance of Domestic Robots by Younger and Older Adults
Ezer, N., Fisk, A. D., & Rogers, W. A. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02710-9_5 · Cited by 134 · see citing works
Kwon, M., Huang, S. H., & Dragan, A. D. (2018) · Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3171221.3171276 · Cited by 123 · see citing works
Designing Contestability: Interaction Design, Machine Learning, and Mental Health
Hirsch, T., Merced, K., Narayanan, S., Imel, Z. E., & Atkins, D. C. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064703 · Cited by 110 · see citing works
Lipton, P. (2001) · Springer eBooks · doi:10.1007/978-94-015-9731-9_2 · Cited by 102 · see citing works
Do You Get It? User-Evaluated Explainable BDI Agents
Broekens, J., Harbers, M., Hindriks, K., van den Bosch, K., Jonker, C., & Meyer, J. (2010) · Multiagent System Technologies · doi:10.1007/978-3-642-16178-0_5 · Cited by 86 · see citing works
Jaszczolt, K. M. (1995) · Language · doi:10.2307/415751 · Cited by 81 · see citing works
Understanding Robot Acceptance
Beer, J. M., Prakash, A., Mitzner, T. L., & Rogers, W. A. (2011) · SMARTech Repository (Georgia Institute of Technology) · Cited by 78 · see citing works
Explainable PCGML via Game Design PatternsPreprint
Guzdial, M., Reno, J., Chen, J., Smith, G., & Riedl, M. O. (2018) · AIIDE Workshops · doi:10.48550/arxiv.1809.09419 · Cited by 34 · see citing works
Self-citationRobot confidence and trust alignment
Kaniarasu, P., Steinfeld, A. M., Desai, M., & Yanco, H. A. (2013) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.5555/2447556.2447614 · Cited by 24 · see citing works
Translating NeuralesePreprint
Andreas, J., Dragan, A. D., & Klein, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06960 · Cited by 12 · see citing works
Teaching Meaningful ExplanationsPreprint
Codella, N., Hind, M., Ramamurthy, K., Campbell, M., Dhurandhar, A., Varshney, K. R., Wei, D., & Mojsilovic, A. (2018) · arXiv.org · doi:10.48550/arxiv.1805.11648 · Cited by 7 · see citing works
A Confidence-Based Approach to Multi-Robot Learning from Demonstration.
Chernova, S., & Veloso, M. (2009) · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [30]30. Tim Miller. 2017. Explanation in artificial intelligence: insights from the social sciences. arXiv preprint arXiv:1706.07269 (2017). · arXiv
- [37]37. Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research 17 (1994), 273-85.
- [40]40. Christopher Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3-4 (1992), 279-292.
Cited by
Sorted by citations
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · see citing works
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · see citing works
Self-citationHoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1096257 · Cited by 272 · see citing works
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278 · see citing works
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274 · see citing works
Self-citationA Survey of Data-driven and Knowledge-aware eXplainable AI
Li, X.-H., Cao, C. C., Shi, Y., Bai, W., Gao, H., Qiu, L., Wang, C., Gao, Y., Zhang, S., Xue, X., et al. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2983930 · Cited by 224 · see citing works
How to explain AI systems to end users: a systematic literature review and research agenda
Laato, S., Tiainen, M., Najmul Islam, A. K. M., & Mäntymäki, M. (2022) · Internet Research · doi:10.1108/intr-08-2021-0600 · Cited by 155 · see citing works
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 145 · see citing works
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144 · see citing works
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · see citing works
Self-citationDhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119 · see citing works
Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., Riener, A., & Riedl, M. O. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503727 · Cited by 117 · see citing works
Self-citationCOGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · see citing works
Exploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86 · see citing works
You Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85 · see citing works
Explainable goal-driven agents and robots-a comprehensive review
Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S. (2023) · ACM Computing Surveys · doi:10.1145/3564240 · Cited by 79 · see citing works
“ARG [59], on the other hand, is a Reactive technique for producing human-like rationalization for agents playing a game.”
BackgroundMethodInfluentialExplainable Activity Recognition for Smart Home Systems
Das, D., Nishimura, Y., Vivek, R. P., Takeda, N., Fish, S. T., Plötz, T., & Chernova, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3561533 · Cited by 70 · see citing works
Exploring the effects of human-centered AI explanations on trust and reliance
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1151150 · Cited by 57 · see citing works
Leveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52 · see citing works
Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
Ehsan, U., & Riedl, M. O. (2021) · arXiv · doi:10.48550/arxiv.2109.12480 · Cited by 99 · see citing works
Self-citationWait, But Why?: Assessing Behavior Explanation Strategies for Real-Time Strategy Games
Robertson, J., Kokkinakis, A. V., Hook, J., Kirman, B., Block, F., Ursu, M. F., Patra, S., Demediuk, S., Drachen, A., & Olarewaju, O. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450699 · Cited by 18 · see citing works
Dynamic Selection of Reliance Calibration Cues With AI Reliance Model
Fukuchi, Y., & Yamada, S. (2023) · IEEE Access · doi:10.1109/access.2023.3339548 · Cited by 9 · see citing works
“However, it has also been revealed that giving an excessive amount of information on an AI can have negative consequences [7], [8], [9].”
Background
- Authors
- Ehsan, U., Tambwekar, P., Chan, L., Harrison, B., & Riedl, M. O.
- Affiliations
- Georgia Institute of Technology (US)
- University of Kentucky (US)
- Venue
- Proceedings of the 24th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2019-03-17
- Pages
- 263-274
- OpenAlex
- W2963395533
- Cited by
- 210
- References
- 45
- Citations by year
- 2026: 27 · 2025: 25 · 2024: 38 · 2023: 30 · 2022: 22 · 2021: 42 · 2020: 21 · 2019: 5
- FWCI
- 8.61
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- ONR (N00014141000)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- autonomous agents, explainable AI, mental models, user preferences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
