The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Mental Models of AI Agents in a Cooperative Game Setting
Abstract
As more and more forms of AI become prevalent, it becomes increasingly important to understand how people develop mental models of these systems. In this work we study people's mental models of AI in a cooperative word guessing game. We run think-aloud studies in which people play the game with an AI agent; through thematic analysis we identify features of the mental models developed by participants. In a large-scale study we have participants play the game with the AI agent online and use a post-game survey to probe their mental model. We find that those who win more often have better estimates of the AI agent's abilities. We present three components for modeling AI systems, propose that understanding the underlying technology is insufficient for developing appropriate conceptual models (analysis of behavior is also necessary), and suggest future work for studying the revision of mental models over time.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it more importantly shows how the AI's confidence is a simple, yet salient feature via which users create a mental model of the AI's global behavior [23].”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [23]
In the citing paper
“It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [12]
In the citing paper
“It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [12]
Citations
Cites
Philosophical Investigations: Philosophische Untersuchungen
Burnheim, J., & Philosophical Studies, W. S. F. T. N. U. O. I. (1954) · Philosophical Studies · doi:10.5840/philstudies1954421 · Cited by 2,902
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
The University of South Florida free association, rhyme, and word fragment norms
Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195588 · Cited by 1,929
Some observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“We note that our use of the term “local behavior” is related to the team “local explanations” as used in the explainable AI literature [18].”
MethodConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03975 · Cited by 758
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“Yin et. al [29] look at the effect of stated and observed accuracy of machine learning models on people’s trust of the system, finding that the effect of stated accuracy can change depending on the observed accuracy.”
BackgroundMental models, conceptual models, and modelling
Greca, I. M., & Moreira, M. A. (2000) · International Journal of Science Education · doi:10.1080/095006900289976 · Cited by 518
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 3×
“While some HCI researchers have looked into how people develop mental models of AI systems [ Kulesza et al. , 2012; Kulesza et al. , 2013; Bansal et al. , 2019], mostly we have seen research into Explainable AI [Cheng et al. , 2019; Wang et al. , 2019; Wiegand et al. , 2019; Kunkel et al. , 2019].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Das, A., Kottur, S., Moura, J. M. F., Lee, S., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.321 · Cited by 356
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
Havrylov, S., & Titov, I. (2017) · arXiv · doi:10.48550/arxiv.1705.11192 · Cited by 195
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
I Drive - You Trust: Explaining Driving Behavior Of Autonomous Cars
Wiegand, G., Schmidmaier, M., Weber, T., Liu, Y., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312817 · Cited by 54
An HR perspective: the global hunt for talent in the digital age
Dickson, D. R., & Nusair, K. (2010) · Worldwide Hospitality and Tourism Themes · doi:10.1108/17554211011012612 · Cited by 44
Evaluating Visual Conversational Agents via Cooperative Human-AI Games
Chattopadhyay, P., Yadav, D., Prabhu, V., Chandrasekaran, A., Das, A., Lee, S., Batra, D., & Parikh, D. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13312 · Cited by 44
Considerations on explainable AI and users’ mental models
Rutjes, H., Willemsen, M. C., & IJsselsteijn, W. A. (2019) · TU/e Research Portal · Cited by 18
Inference and communication in the game of Password
Xu, Y., & Kemp, C. (2018) · doi:10.1184/r1/6615230 · Cited by 11
The Taboo Challenge Competition
Rovatsos, M., Gromann, D., & Bella, G. (2018) · AI Magazine · doi:10.1609/aimag.v39i1.2779 · Cited by 6
A Semantic Distance based Architecture for a Guesser Agent in ESSENCE's Location Taboo Challenge
Adrian, K., Bilgin, A., & Eecke, P. V. (2016) · UvA-DARE (University of Amsterdam) · Cited by 5
Research works printed but not linked: 5
- Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, and others. 2019. The Hanabi Challenge: A New Frontier for AI Research. arXiv preprint arXiv:1902.00506 (2019). · arXiv
- Virginia Braun and Victoria Clarke. 2012. Thematic analysis. In APA handbook of research methods in psychology, Vol. 2. American Psychological Association, 57--71.
- Tim Miller. 2018. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence (2018).
- Judy Hanwen Shen, Matthias Hofer, Bjarke Felbo, and Roger Levy. 2018. Comparing Models of Associative Meaning: An Empirical Investigation of Reference in Simple Language Games. arXiv preprint arXiv:1810.03717 (2018). · arXiv
- Ludwig Wittgenstein. 2009. Philosophical investigations. John Wiley & Sons.
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“” While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5], it more importantly shows how the AI’s confdence is a simple, yet salient feature via which users create a mental model of the AI’s global behavior [23].”
BackgroundDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635
“demonstrated that people can construct mental models of AI through repeated interactions with a model that help them understand the AI’s knowledge distributions, local behavior, and global behavior.[13] Using prompts to generate image evidence of AI knowledge is also a way of reducing uncertainty with AI, one of the f…”
BackgroundDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
“demonstrated that people can construct mental models of AI through repeated interactions with a model that help them understand the AI’s knowledge distributions, local behavior, and global behavior.[13] Using prompts to generate image evidence of AI knowledge is also a way of reducing uncertainty with AI, one of the f…”
BackgroundLai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“The most common objective metric is time taken on the task [1, 8, 20, 23, 36, 38, 41, 56, 60, 63, 70, 71, 79, 91, 96, 106, 108].”
BackgroundMethod
Metadata
- Authors
- Gero, K. I., Ashktorab, Z., Dugan, C., Pan, Q., Johnson, J., Geyer, W., Ruiz, M., Miller, S., Millen, D. R., Campbell, M., Kumaravel, S., & Zhang, W.
- Affiliations
- Columbia University (US)
- IBM (United States) (US)
- IBM Research - Cambridge (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-12
- OpenAlex
- W3030109276
- Cited by
- 121
- References
- 26
- Citations by year
- 2026: 11 · 2025: 30 · 2024: 21 · 2023: 23 · 2022: 20 · 2021: 15 · 2020: 1
- FWCI
- 20.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- AI agents, cooperative games, word guessing game, human-AI interaction, think-aloud study, conceptual model, model revision, human-AI collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.