Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Open accessAbstract
Decisions made by human-AI teams (e.g., AI-advised humans) are increasingly common in high-stakes domains such as healthcare, criminal justice, and finance. Achieving high team performance depends on more than just the accuracy of the AI system: Since the human and the AI may have different expertise, the highest team performance is often reached when they both know how and when to complement one another. We focus on a factor that is crucial to supporting such complementary: the human’s mental model of the AI capabilities, specifically the AI system’s error boundary (i.e. knowing “When does the AI err?”). Awareness of this lets the human decide when to accept or override the AI’s recommendation. We highlight two key properties of an AI’s error boundary, parsimony and stochasticity, and a property of the task, dimensionality. We show experimentally how these properties affect humans’ mental models of AI capabilities and the resulting team performance. We connect our evaluations to related work and propose goals, beyond accuracy, that merit consideration during model selection and optimization to improve overall human-AI team performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human-AI collaborative decision-making can achieve complementary performance, outperforming neither the AI nor the human alone.
“researchers have explored the feasibility of human-AI collaborative decision-making, in which an AI model provides humans new data-driven insights on a task for achieving complementary performance, outperforming neither of the AI or the human alone [5, 30, 42, 43].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [5]
The Introduction says AI recommendations can help human-AI teams perform better than either the human or the AI alone. That contradicts the citing sentence’s printed claim that complementary performance outperforms neither, although the paper does support the broader idea of human-AI collaboration. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPeople whose own outcome intuition is highly consistent with the AI system will benefit less from AI support because of a weak pathway 1.
“Second, we can anticipate that people whose own outcome intuition is highly consistent with the AI system will benefit less from AI support because of a weak pathway 1, which has been alluded to by prior works as complementary knowledge [3,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [3]
The Introduction says human–AI teams benefit when people can account for each partner’s strengths and weaknesses, supporting the general idea of complementary knowledge. It does not establish that people whose outcome intuitions closely match the AI’s predictions benefit less from AI support; the paper instead examines mental models of the AI’s error boundary. The cited copy provides section labels but no page numbers. (LLM, conf=0.78)
Experimental — accuracy not yet measuredTask complexity plays a significant role in determining the degree of reliance on AI advice.
“The complexity of tasks plays a signifcant role in determining the degree of reliance on AI advice, consistent with the fndings of [9, 105].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [9]
The Results (Q2: Do more parsimonious error boundaries facilitate mental model creation?) report that team performance generally decreases as the number of human-visible task features increases. The Introduction explains that people decide whether to trust or override AI recommendations, but the cited passages do not establish that task complexity determines the degree of reliance on AI advice specifically. (LLM, conf=0.82)
Experimental — accuracy not yet measuredRequiring the programmer to ask for suggestions may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities, which can require long-term interactions with the model or training.
“Requiring the programmer to ask may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities (Sarkar et al. 2022) which can require long-term interactions with the model (Bansal et al. 2019) or training (Mozannar, Satyanarayan, and Sontag 2022).”
From When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming (2024) · cites this paper as Bansal et al. 2019
Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians.
“Other works that utilize AI/ML models focused on tasks that do not require domain experts, such as judges or clinicians [5, 38, 64].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [5]
AI explanation is important to communicate an AI output to a user and to realize human-AI collaboration for a decision-making task.
“researchers have discussed the importance of an AI explanation to communicate an AI output to a user [5, 20, 29] and realize human-AI collaboration [48] for a decision-making task [15, 42].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [5]
End-users develop a mental model of trust in AI-assisted decision making.
“While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5]”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [5]
Systems with parsimonious and non-stochastic error boundaries were the easiest for humans to work with.
“Bansal et al. [3] focused on attributes of AI systems and found that systems with parsimonious (simple), and non-stochastic (predictable) error boundaries were the easiest for humans to work with.”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as Bansal et al. [3]
People decide when to rely on an AI by using their mental models of how the AI tends to behave.
“People decide when to rely on an AI by using their mental models [3, 30], or internal representations, of how the AI tends to behave: when it is most accurate, when it is most likely to fail, etc.”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as [3]
An overly simple or wrong mental model can lead to blind spots and systematic failures.
“while an overly simple or wrong mental model can lead to blind spots and systematic failures [3, 8].”
From Improving Human-AI Collaboration With Descriptions of AI Behavior (2023) · cites this paper as [3]
These findings relate to work on automation bias, algorithm aversion, and appropriate reliance.
“These fi ndings relate to work on automation bias, algorithm aversion, and 'appropriate reliance' [5, 32, 44, 87].”
From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [5]
There have been contradictory perspectives on the effect of AI explanations on user's trust in an AI output: trust is not affected by the explanation interface or can be increased by just presenting explanations.
“However, there has been contradictory perspectives on the effect of AI explanations on user's trust on an AI output: users' trust in an algorithmic decision is not affected by the explanation interface [16] or can be increased by just presenting explanations [5].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [5]
The discussion of interpretability and explanations (Q3: Do less stochastic error boundaries lead to better mental models?; page unavailable) says explanations can help understanding, but does not say that merely presenting them increases trust in an AI output. Searches for explanation presentation, trust, and explanation interfaces found no passage addressing that effect. (LLM, conf=0.74)
Experimental — accuracy not yet measuredUsing five features is appropriate for non-expert users to develop a decision-making model after seeing several task samples.
“This number of features is suggested to be appropriate for non-expert users to form a decisionmaking model by experiencing several task samples (e.g., Bansal et al. [4,5] established users' mental models of AI's error boundaries using a three-feature task).”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [4]
Users with better mental models of their AI teammates are more successful.
“It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [3]
Knowledge of the error boundary enables the user to know when to trust automation and when to override it.
“It has been empirically shown that users with better mental models of their AI teammates are more successful; in particular, knowledge of the error boundary enables the user to know when to trust automation and when to override it [3,12].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [3]
For optimal results, the human needs to understand when to listen to the computer and when to exercise their own agency.
“For optimal results, the human needs to understand when to listen to the computer and when to exercise their own agency[3].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1× · see citing works
“In fact, appropriate trust in automation is a topic that has received early attention (Lee and See 2004) as determinant factor for designing systems that require people to manage and intervene during imperfect automation.”
BackgroundDropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
The Psychology of Everyday Things
Massaro, D. W., & Norman, D. A. (1990) · The American Journal of Psychology · doi:10.2307/1423268 · Cited by 3,475 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Self-citationThe psychology of everyday things
Weld, D. S. (1989) · Artificial Intelligence · doi:10.1016/0004-3702(89)90083-0 · Cited by 2,118 · see citing works
Self-citationExplainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · see citing works
Deep Learning for Identifying Metastatic Breast CancerPreprint
Wang, D., Khosla, A., Gargeya, R., Irshad, H., & Beck, A. H. (2016) · arXiv.org · doi:10.48550/arxiv.1606.05718 · Cited by 1,028 · see citing works
Metaphor No More: A 15-Year Review of the Team Mental Model Construct
Mohammed, S., Ferzandi, L. A., & Hamilton, K. (2010) · Journal of Management · doi:10.1177/0149206309356804 · Cited by 839 · see citing works
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · see citing works
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · see citing works
(2001) · Cited by 683 · see citing works
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662 · see citing works
Fairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2× · see citing works
“…is informative for decision-making, research in human-centered machine learning (Gillies et al. 2016) and HCI shows that people have difficulties with correctly interpreting probabilistic statements (Handmer and Proudley 2007) and even system accuracy itself (Yin, Vaughan, and Wallach 2019).”
BackgroundFrom software product lines to software ecosystems
Bosch, J. (2009) · Software Product Lines · doi:10.5555/1753235.1753251 · Cited by 475 · see citing works
The role of mental models in team performance in complex systems
Rouse, W. B., Cannon‐Bowers, J. A., & Salas, E. (1992) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.199457 · Cited by 416 · see citing works
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · see citing works
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · see citing works
Self-citationKocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277 · see citing works
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · see citing works
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213 · see citing works
Grosz, B. J., & Kraus, S. (1999) · Applied Logic Series · doi:10.1007/978-94-015-9204-8_10 · Cited by 177 · cited 1× · see citing works
“Early work explored the importance of mental models for achieving high performance in group work (Grosz and Kraus 1999; Mohammed, Ferzandi, and Hamilton 2010), human-system collaboration (Rouse, Cannon-Bowers, and Salas 1992), and interface design (Carroll and Olson 1988).”
BackgroundDirections in hybrid intelligence: complementing AI systems with human intelligence
Kamar, E. S. (2016) · International Joint Conference on Artificial Intelligence · Cited by 154 · see citing works
Self-citationIdentifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Lakkaraju, H., Kamar, E. S., Caruana, R., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10821 · Cited by 146 · see citing works
Self-citationHuman-Centred Machine Learning
Gillies, M. F. P., Fiebrink, R. A., Tanaka, A., Garcia, J., Bevilacqua, F. P., Héloir, A., Nunnari, F., Mackay, W. E., Amershi, S., Lee, B., et al. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2856492 · Cited by 138 · see citing works
Please Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138 · see citing works
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · see citing works
Self-citationBayati, M., Braverman, M., Gillam, M., Mack, K. M., Ruiz, G., Smith, M. S., & Horvitz, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109264 · Cited by 122 · see citing works
Self-citationCollaborative Systems (AAAI-94 Presidential Address)
Grosz, B. J. (1996) · AI Magazine · doi:10.1609/aimag.v17i2.1223 · Cited by 119 · see citing works
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92 · see citing works
Beyond Sparsity: Tree Regularization of Deep Models for InterpretabilityPreprint
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Röth, V., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06178 · Cited by 88 · see citing works
Learning Global Additive Explanations for Neural Nets Using Model DistillationPreprint
Tan, S., Caruana, R., Hooker, G., Koch, P., & Gordo, A. (2018) · arXiv (Cornell University) · Cited by 77 · see citing works
Communicating uncertainty via probabilities: The case of weather forecasts
Handmer, J. W., & PROUDLEY, B. (2007) · Environmental Hazards · doi:10.1016/j.envhaz.2007.05.002 · Cited by 68 · see citing works
Human-in-the-Loop Interpretability PriorPreprint
Lage, I., Ross, A. S., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11571 · Cited by 55 · see citing works
Patient risk stratification with time-varying parameters: a multitask learning approach
Wiens, J., Guttag, J. V., & Horvitz, E. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 54 · see citing works
Self-citationMental Models in Human-Computer Interaction
National Research Council (1987) · National Academies Press eBooks · doi:10.17226/790 · Cited by 50 · see citing works
Ustun, B., & Rudin, C. D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098161 · Cited by 47 · see citing works
Rudin, C. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.10154 · Cited by 44 · see citing works
A Coverage-Based Utility Model for Identifying Unknown Unknowns
Bansal, G., & Weld, D. S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11493 · Cited by 32 · see citing works
Self-citationChakraborti, T., & Subbarao Kambhampati, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09854 · Cited by 29 · see citing works
What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative PlayPreprint
Feng, S., & Boyd-Graber, J. L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09648 · Cited by 28 · see citing works
Lundgard, A., Yang, Y., Foster, M. L., & Lasecki, W. S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174041 · Cited by 22 · see citing works
Self-citationMENTAL MODELS IN HUMAN-COMPUTER INTERACTION
Payne, S. (2007) · CRC Press eBooks · doi:10.1201/9781410615862-12 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [15]15. Gal, Y. 2016. Uncertainty in deep learning. Ph.D. Disserta- tion, PhD thesis, University of Cambridge.
- [18]18. Grosz, B. J., and Kraus, S. 1999. The evolution of shared- plans. In Foundations of rational agency. Springer.
- [35]35. Lundgard, A.; Yang, Y.; Foster, M. L.; and Lasecki, W. S. 2018. Bolt: Instantaneous crowdsourcing via just-in-time training. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 467. ACM.
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781 · see citing works
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · see citing works
“” While this observation empirically shows that end-users develop mental model of trust in AI-assisted decision making [5], it more importantly shows how the AI’s confdence is a simple, yet salient feature via which users create a mental model of the AI’s global behavior [23].”
BackgroundSelf-citationEvaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · see citing works
Xu, W., Dainoff, M. J., Ge, L., & Gao, Z. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2041900 · Cited by 363 · see citing works
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · see citing works
“Prior XAI literature has used a variety of tasks, including binary classification [3, 4, 8, 54] and question answering about images [12, 15] and text [4, 26].”
BackgroundWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · see citing works
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300 · see citing works
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · see citing works
Chong, L., Zhang, G., Goucher-Lambert, K., Kotovsky, K., & Cagan, J. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107018 · Cited by 262 · see citing works
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Larger and more instructable language models become less reliable
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024) · Nature · doi:10.1038/s41586-024-07930-y · Cited by 203 · see citing works
Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183 · see citing works
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · see citing works
Jacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177 · see citing works
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 145 · see citing works
“We can also anticipate that people whose own intuition outcomes are highly consistent with the AI system will benefit less from AI support in general because of a weak pathway 1, which has been alluded to by prior works on complementary knowledge [3, 78].”
BackgroundSelf-citationMa, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581058 · Cited by 143 · see citing works
Henry, K. E., Adams, R., Parent, C., Soleimani, H., Sridharan, A., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., Klein, E. Y., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01895-z · Cited by 127 · see citing works
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132 · see citing works
Self-citationSocial Dynamics of AI Support in Creative Writing
Gero, K. I., Long, T., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580782 · Cited by 129 · see citing works
Gmeiner, F., Yang, H., Yao, L., Holstein, K., & Martelaro, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580999 · Cited by 130 · see citing works
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102285 · Cited by 125 · see citing works
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · see citing works
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · see citing works
Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119 · see citing works
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · see citing works
“A leading view to explain the inferior team performance is that humans overtrust the AI [1].”
BackgroundMental Models of AI Agents in a Cooperative Game Setting
Gero, K. I., Ashktorab, Z., Dugan, C., Pan, Q., Johnson, J., Geyer, W., Ruiz, M., Miller, S., Millen, D. R., Campbell, M., et al. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376316 · Cited by 121 · see citing works
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118 · see citing works
Self-citationXAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Lopes, P., Silva, E., Braga, C., Oliveira, T., & Rosado, L. (2022) · Applied Sciences · doi:10.3390/app12199423 · Cited by 112 · see citing works
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107 · see citing works
“Topics like performance feedback [2, 55], risk perception [27, 34], uncertainty [77] and confidence [10, 79, 87] of machine learning models, impact of explanations [3, 46] have been extensively studied in human-AI decision making.”
BackgroundZhang, Z., Jia, M., Lee, H.-P. (., Yao, B., Das, S., Lerner, A., Wang, D., & Li, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642385 · Cited by 95 · see citing works
“Much of the current mental model research in AI centers on optimizing human-AI teamwork [2, 4].”
On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659051 · Cited by 91 · see citing works
“For instance, there seem to be substantial challenges with respect to achieving adequate trust in AI-based systems [3, 37, 50, 90].”
BackgroundImproving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87 · see citing works
“People decide when to rely on an AI by using their mental models [3, 30], or internal representations, of how the AI tends to behave: when it is most accurate, when it is most likely to fail, etc.”
BackgroundInfluentialYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85 · see citing works
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78 · see citing works
Levy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · see citing works
Hou, C., Zhu, G., Sudarshan, V., Lim, F. S., & Ong, Y. S. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105329 · Cited by 70 · see citing works
"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools
Adnin, R., & Das, M. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675631 · Cited by 73 · see citing works
Fok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54 · see citing works
Self-citationYou’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · see citing works
AI Trust: Can Explainable AI Enhance Warranted Trust?
de Brito Duarte, R., Correia, F., Arriaga, P., & Paiva, A. (2023) · Human Behavior and Emerging Technologies · doi:10.1155/2023/4637678 · Cited by 52 · see citing works
Artificial Intelligence and Groups: Effects of Attitudes and Discretion on Collaboration
Bezrukova, K., Griffith, T. L., Spell, C., Rice, V., & Yang, H. E. (2023) · Group & Organization Management · doi:10.1177/10596011231160574 · Cited by 51 · see citing works
My teammate is AI: understanding students’ perceptions of student-AI collaboration in drawing tasks
Kim, J., & Cho, Y. H. (2025) · Asia Pacific Journal of Education · doi:10.1080/02188791.2023.2286206 · Cited by 50 · see citing works
How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration
Cao, S., Gomez, C., & Huang, C.-M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610068 · Cited by 48 · see citing works
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48 · see citing works
Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42 · see citing works
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · see citing works
Chiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42 · see citing works
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41 · see citing works
Evaluating the Interpretability of Generative Models by Interactive Reconstruction
Ross, A., Chen, N., Hang, E. Z., Glassman, E. L., & Doshi-Velez, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445296 · Cited by 40 · see citing works
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
“…wide range of studies, researchers have consistently found that reliance on AI systems is influenced by various factors including human-related aspects [37, 51, 81, 101, 120], attributes of the AI systems [43, 79, 106, 107], and characteristics of the decision-making tasks [9, 12, 16, 46, 126].”
InfluentialHopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22 · see citing works
“…to ask may lead to sub-optimal interactions because its success would rely on programmers having an accurate mental model of Copilot abilities (Sarkar et al. 2022) which can require long-term interactions with the model (Bansal et al. 2019) or training (Mozannar, Satyanarayan, and Sontag 2022).”
BackgroundSelf-citationHe, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Self-citationHow Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)
Schlicker, N., Baum, K., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2022) · doi:10.31234/osf.io/qhwvx · Cited by 79 · see citing works
Human in the AI Loop via xAI and Active Learning for Visual Inspection
Rožanec, J. M., Montini, E., Cutrona, V., Papamartzivanos, D., Klemenčič, T., Fortuna, B., Mladenić, D., Veliou, E., Giannetsos, T., & Emmanouilidis, C. (2024) · Artificial Intelligence in Manufacturing · doi:10.1007/978-3-031-46452-2_22 · Cited by 17 · see citing works
Radensky, M., Séguin, J. A., Lim, J. S., Olson, K., & Geiger, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594043 · Cited by 14 · see citing works
“While there are certainly other factors that could be measured in relation to trust such as control [23, 26] and predictability [4, 23], these perceived facets cover a large and informative portion of what may be influenced by CRS confidence signals.”
BackgroundDo Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., & McKeown, K. (2023) · arXiv · doi:10.48550/arxiv.2307.08678 · Cited by 5 · see citing works
“…an AI system is important, as it helps humans understand what an AI system can and cannot achieve (Chandrasekaran et al., 2018), which informs humans how to improve the system or appropriately deploy the system without misuse or overtrust (Cassidy, 2009; Bansal et al., 2019; Ye and Durrett, 2022).”
BackgroundReimagining Situation Awareness and Option Awareness for Human-Machine Teaming
Drury, J. L., Klein, G. L., Booker, L., Ryall, K., & Dubrow, S. (2022) · 2022 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) · doi:10.1109/cogsima54611.2022.9830660 · Cited by 2 · see citing works
From testbeds to high-stakes work: a review of human-AI teaming domains and teaming factors
Kargarnovin, S., Hernandez, C. I., Reiners, D., Cruz-Neira, C., Bochenek, G., & Karwowski, W. (2026) · Frontiers in Robotics and AI · doi:10.3389/frobt.2026.1733942 · Cited by 2 · see citing works
Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
Ko, J., Choi, J., Morales, C., Kim, D., & Ko, M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790929 · Cited by 1 · see citing works
- Authors
- Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- University of Michigan (US)
- Venue
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Published
- 2019-10-28
- Volume
- 7
- Pages
- 2-11
- ISSN
- 2769-1330 (ISSN-L)
- 2769-1349
- OpenAlex
- W2984353433
- Semantic Scholar
- 5688b8077117b3aafd54c2e71d959284f4d5c8b9
- Cited by
- 466
- References
- 51
- Citations by year
- 2026: 88 · 2025: 90 · 2024: 78 · 2023: 77 · 2022: 62 · 2021: 47 · 2020: 23 · 2019: 1
- FWCI
- 55.08
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- Microsoft Research
- Defense Advanced Research Projects Agency
- Office of Naval Research (N00014- 18-1-2193, N00014)
- Topics
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Team Dynamics and Performance
- Keywords
- human-AI teaming, mental models, human-AI complementarity, parsimony, randomness, team performance, model selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
