Improving Human-AI Collaboration With Descriptions of AI Behavior
Open accessThe DOI matches its Crossref record; 73 of the 76 references it cites were found in a registry or an index; 23 of the 38 sources its claims cite were checked for support.
Abstract
People work with AI systems to improve their decision making, but often under- or over-rely on AI predictions and perform worse than they would have unassisted. To help people appropriately rely on AI aids, we propose showing them behavior descriptions, details of how AI systems perform on subgroups of instances. We tested the efficacy of behavior descriptions through user studies with 225 participants in three distinct domains: fake review detection, satellite image classification, and bird classification. We found that behavior descriptions can increase human-AI accuracy through two mechanisms: helping people identify AI failures and increasing people's reliance on the AI when it is more accurate. These findings highlight the importance of people's mental models in human-AI collaboration and show that informing people of high-level AI behaviors can significantly improve AI-assisted decision making.
Overview
CoChat Summary
TL;DR
This paper tests whether showing people short, subgroup-level descriptions of an AI’s behavior helps them collaborate better with the AI. In three Amazon Mechanical Turk classification studies with 225 participants, behavior descriptions improved performance in two tasks, but did not change participants’ reported trust, helpfulness, or willingness to reuse the AI.
Key findings
- Across the three tasks, behavior descriptions significantly improved overall accuracy in 2 of 3 domains: reviews and birds; the satellite task showed no significant overall improvement.
- In the reviews task, accuracy differed across conditions with F(2,64) = 9.18, p < 0.001; the only significant pairwise difference was between No AI and AI + BD (p < 0.001, 95% CI [0.06, 0.22]).
- In the birds task, accuracy differed across conditions with F(2,65) = 3.98, p = 0.023; the only significant pairwise difference was between No AI and AI + BD (p = 0.048, 95% CI [0.00, 0.16]), and this produced complementary human-AI accuracy.
- The proposed mechanism in H2 was only partially supported: in reviews, there were no accuracy differences for the behavior-description groups, while in birds, group 2 showed a significant difference (F(2,65) = 6.81, p = 0.002) and supported H2; overall, increased accuracy also came from greater reliance on the AI in the main group.
Methods
- Three between-subjects human-subject experiments were run on Amazon Mechanical Turk with 225 U.S. participants total, across fake review detection, satellite image classification, and bird classification.
- Each participant labeled 30 instances: 20 from the main group and 5 each from two behavior-description groups; in the AI + BD condition, participants saw a behavior description for those subgroup instances.
- Main analyses used ANOVA with Tukey post-hoc tests for accuracy, Mann-Whitney U tests with Bonferroni corrections for Likert items, and linear models to test learning effects.
Limitations
- The paper states that participants were Amazon Mechanical Turk workers with limited domain expertise, so results may differ for domain experts/professionals; the authors explicitly say future studies should test that.
- The study was a controlled simulated setting, so effects may differ in real-world consequential settings such as radiology; the paper explicitly notes this.
- The paper says it tested one specific type of behavior description—subgroup accuracy—so the generality to other behavior-description formats is not established; the text makes evident this is a design-scope limitation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Don Norman's mental model framework [43] describes how designers use their mental models to implement systems. End-users then interact with the systems and develop their own mental models of how they believe the systems work. While this process is similar for AI models, a key difference is that an AI is not a direct representation of a developer's intent, but a stochastic model learned from data. This means that (1) AI developers themselves have to make sense of what an AI system has learned through testing and iteration. Subsequently, they can encode these insights as (2) behavior descriptions , details of how an AI performs on subgroups of instances, that can be shown to end-users to improve human-AI collaboration.](https://figures.cochat.ai/paper-figures/8a25e79489564f9f8cbc1e6321036c0b/improving-human-ai-collaboration-with-descriptions-of-ai-behavior-figure-1.png)









Verification
Claims the paper makes
14 Valid · 3 Not valid · 12 Not checked · 84 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“First, modern AI systems are often black-box models for which humans cannot see how or why the model made a prediction [54].”
What the sources say
- Supportedcites [54] [54] Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (May 2019), 206-215. https://doi.org/10.1038/s42256-019-0048-x - source
The Introduction says that many machine-learning models used for high-stakes predictions are black boxes that do not explain their predictions in a way humans can understand. This supports the citing sentence’s paraphrase. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite the insights post-hoc model explanations can provide end-users, they have been found to have a small or even detrimental effect on the performance of AI-assisted decision making [17, 34].”
What the sources say
- Partly supportedcites [17] [17] Eric Chu, Deb Roy, and Jacob Andreas. 2020. Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction. arXiv:2007.12248 [cs, stat] (July 2020). http://arxiv.org/abs/2007.12248 arXiv: 2007.12248. - sourceExperimental — accuracy not yet measured
Chu et al. (2020) support the 'small effect' half of the claim: Section 3.1 (p. 5) states 'The addition of explanations did not improve accuracy' and that explanation-without-prediction treatments 'resulted in small and non statistically significant effects', and the Conclusion (p. 8) reports that showing explanations 'did not have a significant effect on accuracy, trust, or understanding'. However, our copy contains no finding that explanations were detrimental to performance in their own trial — the ordering explain-strong < explain-spurious < explain-random is explicitly called small and non-significant (p. 5), and even faulty explanations 'did not significantly decrease human trust'. The detrimental/hindering results the claim bundles in appear only as reports of other work in Related Work (p. 8), e.g. accuracy decreases in [49] and transparency hindering error detection in [34]. (LLM, conf=0.72)
- Partly supportedcites [34] [34] Vivian Lai and Chenhao Tan. 2019. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, Atlanta GA USA, 29-38. https://doi.org/10.1145/3287560.3287590 - sourceExperimental — accuracy not yet measured
Section 4.1 (p. 6) reports that explanations alone slightly improved human accuracy in the paper’s deception-detection task, supporting the ‘small effect’ part. It does not report a detrimental effect on decision-making performance; the Concluding Discussion (p. 9) raises a separate concern that irrelevant explanations can increase trust in machine predictions. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In another empirical study, Hopkins and Booth [23] found that practitioners would avoid or limit model updates due to concerns about breaking their models and introducing more failures.”
What the sources say
- Supportedcites Hopkins and Booth [23] [23] Aspen Hopkins and Serena Booth. 2021. Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development. In AIES 2021 - Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. Association for Computing Machinery, Inc, 134-145. https://doi.org/10.1145/3461702.3462527 - source
Section 4.2.1 (Bias Mitigation Through Diversity or Personalization) states that teams delayed introducing new, diverse data 'until user requirements deemed it strictly necessary,' and that 'By doing so, they minimized how frequently they updated-and potentially broke-their models,' describing this slow, cautious strategy as 'common in production' — directly supporting avoidance/limitation of updates for fear of breaking models. Section 4.1.3 corroborates the failure concern: R13's retraining on each new data iteration produced 'unpredictable performance drops' and they weighed decomposing the model to scale 'without the risk of compromising past performance.' Section 4.5.4 adds that less experienced organizations 'might not afford retraining and replacing models,' with R3 calling ML development 'notably slow and defensive.' (LLM, conf=0.88)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We chose this task since it has been used in previous studies of human-AI collaboration to test the effect of explanations [34] and tutorials [33].”
What the sources say
- Supportedcites [34] [34] Vivian Lai and Chenhao Tan. 2019. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, Atlanta GA USA, 29-38. https://doi.org/10.1145/3287560.3287590 - source
The paper describes using deceptive-review detection to study human decisions assisted by machine-learning models (Related Work, p. 3). It tests explanations of machine predictions as a form of assistance (ACM Reference Format, p. 2). The tutorials portion of the citing sentence has a separate citation and is not attributed to this source. (LLM, conf=0.93)
- Supportedcites [33] [33] Vivian Lai, Han Liu, and Chenhao Tan. 2020. "Why is 'Chicago' Deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376873 - source
The citation to this paper is attached to “tutorials,” not “explanations.” The Introduction (p. 2) says the authors used deceptive review detection as a testbed for experiments on whether tutorials improve human performance, and the Abstract (p. 1) reports that the tutorials did improve it. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ML practitioners interviewed by Holstein et al. [22] would often try to fix one problem that would cause the model to start failing in other unrelated ways.”
What the sources say
- Partly supportedcites Holstein et al. [22] [22] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1-16. https://doi.org/10.1145/3290605.3300830 - sourceExperimental — accuracy not yet measured
In “Addressing Detected Issues” (p. 11), interviewees describe fairness fixes that sometimes produced unexpected harms or other forms of unfairness. This supports the idea that fixing one issue could create another, but not the claim that practitioners often caused models to fail in unrelated ways. (LLM, conf=0.85)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
54 more claims · 3 with their cited sources checked
Create a free account to see all 113 claims and to verify the 6 references still open.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 24
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“LIME [50] is one such method, which fits linear models to a neighborhood of instances to highlight which features most influenced a model’s prediction.”
MethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“Glass-box models can be easier to debug and trace, improving people’s ability to understand AI predictions [54].”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 4×
“A detailed and accurate mental model allows a person to effectively complement an AI system by appropriately relying [37] on its output, while an overly simple or wrong mental model can lead to blind spots and systematic failures [3, 8].”
BackgroundInfluentialKuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.00982 · Cited by 3,653
Remote Sensing Image Scene Classification: Benchmark and State of the Art
Cheng, G., Han, J., & Lu, X. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2675998 · Cited by 2,926 · cited 2×
“The satellite images come from the NWPU-RESISC45 dataset [16], which has 31,500 satellite images across 45 classes.”
MethodHuman trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 1×
“Although unexpected, the effect is corroborated by studies on trust in AI systems that found that low reliability, AI output that violates people’s over time, decreases trust and reliance in an AI [20].”
ResultSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 1×
“Despite the growing capability of NLP models, many state-of-the-art systems still encode serious biases and harms [13] and fail basic logical tests [52] that developers have been aware of for years, but have not been able to fix.”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Instead of using all input features to explain a prediction, follow-up techniques such as Anchors [51] and MUSE [35] learn sets of rules on features that are sufficient to explain predictions for a subset of instances.”
MethodThreat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021 · cited 1×
“Second, black-box models are also often stochastic and can provide different outputs for slightly different inputs without human-understandable reasons [1].”
BackgroundElement Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load
Sweller, J. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9128-5 · Cited by 2,020 · cited 1×
“Additionally, showing people more information increases their cognitive load, which can lead to decreased learning and performance [55, 58].”
BackgroundProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01316-z · Cited by 1,738
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · cited 1×
“Behavioral analysis enables developers to go beyond aggregate metrics, such as accuracy, to discover specific and important behaviors [11, 49].”
BackgroundBirch, A., & Osborne, M. (2011) · The Association for Computational Linguistics · Cited by 1,200
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 1×
“often expect AI models to behave as humans do, which often does not match their actual behavior [40] and makes people unaware of how an AI may fail [25, 65].”
BackgroundBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Finding Deceptive Opinion Spam by Any Stretch of the Imagination
Ott, M., Choi, Y., Cardie, C., & Hancock, J. T. (2011) · arXiv · doi:10.48550/arxiv.1107.4557 · Cited by 1,008
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · cited 4×
“For this work, we used behavior descriptions stating the model accuracy for subgroups of the data for which the model performs significantly worse than average [10, 18, 44, 52, 62].”
BackgroundMethodInfluentialImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 1×
“[22] would often try to fix one problem that would cause the model to start failing in other unrelated ways.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“Explanations have been shown to potentially increase people’s reliance on an AI, whether it is right or wrong, leading to worse team performance than a human or AI alone [5, 26, 63, 66].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 2×
“Separately, Poursabzi-Sangdeh et al. [48] found that showing end-users more details about an AI reduced their ability to detect AI errors due to information overload.”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“Human-AI collaboration is finding real-world use in tasks ranging from diagnosing prostate cancer [12] to screening calls to child welfare hotlines [19, 27].”
BackgroundVan Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., & Belongie, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298658 · Cited by 592 · cited 1×
“The task was inspired by numerous apps and products for classifying bird species [59].”
BackgroundACT-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention
Anderson, J. R., Matessa, M., & Lebiere, C. (1997) · Human–Computer Interaction · doi:10.1207/s15327051hci1204_5 · Cited by 569 · cited 1×
“This work could take inspiration from studies in HCI and psychology on cognitive modeling, mathematical models of human cognition [2].”
BackgroundKaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 1×
“Explanations have been shown to potentially increase people’s reliance on an AI, whether it is right or wrong, leading to worse team performance than a human or AI alone [5, 26, 63, 66].”
BackgroundUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“Existing work has explored the impact of stated versus observed accuracy and found that there is a small decrease in agreement with the AI the lower the observed accuracy [64].”
BackgroundCognitive Tutor: Applied research in mathematics education
Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194060 · Cited by 483 · cited 1×
“Cognitive models have been used in education by simulating how people learn math to dynamically teach students [53].”
MethodBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 4×
“People decide when to rely on an AI by using their mental models [3, 30], or internal representations, of how the AI tends to behave: when it is most accurate, when it is most likely to fail, etc.”
BackgroundInfluentialThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“At worst, people can perform worse than they would have unassisted, such as clinicians who made more errors than average when shown incorrect AI predictions [7, 24].”
BackgroundToo much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 3×
“We chose this task since it has been used in previous studies of human-AI collaboration to test the effect of explanations [34] and tutorials [33].”
BackgroundMethodResultInfluential(2013) · Cited by 353
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 3×
“This could be an important property for AI systems that are updated often, as new behavior descriptions could be used to directly update end-users mental models and avoid failures from incompatible updates [4].”
BackgroundInfluentialKocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277 · cited 1×
“Other factors such as stated and perceived accuracy [29], confidence values [66], and model personas [28] can also influence people’s mental models and their reliance on AI.”
BackgroundFaithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314229 · Cited by 256 · cited 1×
“Instead of using all input features to explain a prediction, follow-up techniques such as Anchors [51] and MUSE [35] learn sets of rules on features that are sufficient to explain predictions for a subset of instances.”
MethodSome Observations on Mental Models
Norman, D. A. (2014) · Psychology Press eBooks · doi:10.4324/9781315802725-5 · Cited by 253
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 1×
“At worst, people can perform worse than they would have unassisted, such as clinicians who made more errors than average when shown incorrect AI predictions [7, 24].”
BackgroundAdaptive trust calibration for human-AI collaboration
Okamura, K., & Yamada, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229132 · Cited by 239 · cited 1×
“Recent methods have explored improving people’s mental models, including tutorials explaining a task [33], tuning a model to better match human expectations [41], or adaptive trust calibration [45].”
MethodTell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · cited 2×
“People decide when to rely on an AI by using their mental models [3, 30], or internal representations, of how the AI tends to behave: when it is most accurate, when it is most likely to fail, etc.”
BackgroundZhang (2021) · doi:10.1145/3432945 · Cited by 225 · cited 1×
“Lastly, people often expect AI models to behave as humans do, which often does not match their actual behavior [40] and makes people unaware of how an AI may fail [25, 65].”
BackgroundA Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 1×
“Human-AI collaboration is finding real-world use in tasks ranging from diagnosing prostate cancer [12] to screening calls to child welfare hotlines [19, 27].”
BackgroundHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 1×
“Explanations have been shown to potentially increase people’s reliance on an AI, whether it is right or wrong, leading to worse team performance than a human or AI alone [5, 26, 63, 66].”
BackgroundErrudite: Scalable, Reproducible, and Testable Error Analysis
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1073 · Cited by 190 · cited 1×
“There are also tools tailored to specific domains, for example, Errudite [61], a visual analytics system for discovering and testing behaviors in NLP models.”
MethodRapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · cited 1×
“There are various methods for encouraging appropriate reliance of AI systems, often called trust calibration [57] techniques.”
MethodConceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183 · cited 1×
“Other factors such as stated and perceived accuracy [29], confidence values [66], and model personas [28] can also influence people’s mental models and their reliance on AI.”
BackgroundKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 1×
“Human-AI collaboration is finding real-world use in tasks ranging from diagnosing prostate cancer [12] to screening calls to child welfare hotlines [19, 27].”
BackgroundSelf-citationTowards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · cited 2×
“More recently, crowd-based systems such as Pandora [44], Deblinder [9], and P-BTM [38] have shown how crowdsourcing can augment behavioral analysis tools by surfacing previously unknown model failures.”
BackgroundMethodCash, J. J. (2009) · American Journal of Health-System Pharmacy · doi:10.2146/ajhp090181 · Cited by 123 · cited 1×
“When alerts or messages are shown continuously, people can suffer from alert fatigue and begin to ignore the messages [14].”
Background"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
Structure-invariant testing for machine translation
He, P., Meister, C., & Su, Z. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380339 · Cited by 118 · cited 1×
“A common technique is metamorphic testing [21], which verifies the relationship between changes to an input and expected changes in an output.”
MethodEffects of concurrent monitoring on cognitive load and performance as a function of task complexity
van Gog, T., Kester, L., & PAAS, F. G. W. C. (2010) · Applied Cognitive Psychology · doi:10.1002/acp.1726 · Cited by 110 · cited 1×
“Additionally, showing people more information increases their cognitive load, which can lead to decreased learning and performance [55, 58].”
BackgroundSlice Finder: Automated Data Slicing for Model Validation
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00139 · Cited by 83 · cited 2×
“These include systems such as FairVis [10], a visual analytics system for discovering intersectional biases, AnchorViz [15], a visualization for semantically exploring datasets, and Slice Finder [18], a method for automatically surfacing subsets of data with large model loss.”
MethodDiscovering and Validating AI Errors With Crowdsourced Failure Reports
Cabrera, Á. A., Druck, A. J., Hong, J., & Perer, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479569 · Cited by 83 · cited 2×
“More recently, crowd-based systems such as Pandora [44], Deblinder [9], and P-BTM [38] have shown how crowdsourcing can augment behavioral analysis tools by surfacing previously unknown model failures.”
BackgroundSelf-citationWhat Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 62 · cited 1×
“Behavioral analysis enables developers to go beyond aggregate metrics, such as accuracy, to discover specific and important behaviors [11, 49].”
BackgroundSelf-citationAnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Chen, N.-C., Suh, J., Verwey, J., Ramos, G., Drucker, S. M., & Simard, P. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172950 · Cited by 49 · cited 1×
“These include systems such as FairVis [10], a visual analytics system for discovering intersectional biases, AnchorViz [15], a visualization for semantically exploring datasets, and Slice Finder [18], a method for automatically surfacing subsets of data with large model loss.”
MethodMonotasking or Multitasking: Designing for Crowdworkers' Preferences
Lascau, L., Gould, S. J. J., Cox, A. L., Karmannaya, E., & Brumby, D. P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300649 · Cited by 47 · cited 1×
“This is likely due to the way AMT workers complete tasks, as they often take breaks or search for new HITs while working on a task [36].”
BackgroundHopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40 · cited 1×
“Booth [23] found that practitioners would avoid or limit model updates due to concerns about breaking their models and introducing more failures.”
BackgroundWu, T., Weld, D. S., & Heer, J. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3319616 · Cited by 39 · cited 1×
“For this work, we used behavior descriptions stating the model accuracy for subgroups of the data for which the model performs significantly worse than average [10, 18, 44, 52, 62].”
MethodTowards Hybrid Human-AI Workflows for Unknown Unknown Detection
Liu, A., Guerra, S., Fung, I. P.-W., Matute, G., Kamar, E. S., & Lasecki, W. S. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380306 · Cited by 37 · cited 1×
“More recently, crowd-based systems such as Pandora [44], Deblinder [9], and P-BTM [38] have shown how crowdsourcing can augment behavioral analysis tools by surfacing previously unknown model failures.”
BackgroundLogar, T., Aylett-Bullock, J., Nemni, E., Bromley, L., Quinn, J. A., & Luengo-Oroz, M. A. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i09.7101 · Cited by 34 · cited 1×
“This task was inspired by real-world human-AI systems for labeling and segmenting satellite images [39].”
MethodAre Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
Chu, E., Roy, D., & Andreas, J. (2020) · arXiv · doi:10.48550/arxiv.2007.12248 · Cited by 34
Sentiment analysis tools should take account of the number of exclamation marks!!!
Teh, P. L., Rayson, P., Pak, I., & Piao, S. (2015) · Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services · doi:10.1145/2837185.2837216 · Cited by 17 · cited 1×
“Group 2 - Reviews with more than 3 exclamation marks [56].”
BackgroundFAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · cited 2×
“These include systems such as FairVis [10], a visual analytics system for discovering intersectional biases, AnchorViz [15], a visualization for semantically exploring datasets, and Slice Finder [18], a method for automatically surfacing subsets of data with large model loss.”
MethodSelf-citationTeaching Humans When to Defer to a Classifier via Exemplars
Mozannar, H., Satyanarayan, A., & Sontag, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20469 · Cited by 5 · cited 1×
“[42]’s exemplar-based teaching of model behavior.”
Personalization in Human-AI Teams: Improving the Compatibility-Accuracy TradeoffPreprint
Martinez, J., Gal, K., Kamar, E. S., & Lelis, L. H. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02289 · Cited by 3
Teaching Humans When To Defer to a Classifier via Exemplars
Mozannar, H., Satyanarayan, A., & Sontag, D. (2021) · arXiv · doi:10.48550/arxiv.2111.11297
Logar, T., Bullock, J., Nemni, E., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.10685
Research works printed but not linked: 5
- Ruogu Kang, Laura Dabbish, Nathaniel Fruchter, and Sara Kiesler. 2015. My Data Just Goes Everywhere: User Mental Models of the Internet and Implications for Privacy and Security. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015). USENIX Association, Ottawa, 39--52. https://www.usenix.org/conference/soups2015/proceedings/presentation/kang
- 10.1038/s41586-019--1138-y · doi:10.1038/s41586-019--1138-y
- 10.18653/v1 · doi:10.18653/v1
- 10.1007/s10648-010--9128--5 · doi:10.1007/s10648-010--9128--5
- P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. 2010. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001. California Institute of Technology.
Cited by
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Williamson, S. M., & Prybutok, V. (2024) · Information · doi:10.3390/info15060299 · Cited by 146
The Impact of Imperfect XAI on Human-AI Decision-Making
Morrison, K., Spitzer, P., Turri, V., Feng, M., Kühl, N., & Perer, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641022 · Cited by 60
Self-citationMa, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2403.09552 · Cited by 56
Metadata
- Authors
- Cabrera, Á. A., Perer, A., & Hong, J. I.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-04-14
- Volume
- 7
- Issue
- CSCW1
- Pages
- 1-21
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4317464833
- Cited by
- 87
- References
- 76
- Citations by year
- 2026: 37 · 2025: 30 · 2024: 18 · 2023: 2
- FWCI
- 25.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (IIS-2040942)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, behavioral description, AI-assisted decision-making, trust in AI, mental models, fake review detection, satellite image classification, bird classification, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.