The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Open accessAbstract
AI systems are being deployed to support human decision making in high-stakes domains such as healthcare and criminal justice. In many cases, the human and AI form a team, in which the human makes decisions after reviewing the AI’s inferences. A successful partnership requires that the human develops insights into the performance of the AI system, including its failures. We study the influence of updates to an AI system in this setting. While updates can increase the AI’s predictive performance, they may also lead to behavioral changes that are at odds with the user’s prior experiences and confidence in the AI’s inferences. We show that updates that increase AI performance may actually hurt team performance. We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams. Empirical results on three high-stakes classification tasks show that current machine learning algorithms do not produce compatible updates. We propose a re-training objective to improve the compatibility of an update by penalizing new errors. The objective offers full leverage of the performance/compatibility tradeoff across different datasets, enabling more compatible yet accurate updates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This number of features is suggested to be appropriate for non-expert users to form a decisionmaking model by experiencing several task samples (e.g., Bansal et al. [4,5] established users' mental models of AI's error boundaries using a three-feature task).”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“First, the decision performance was linked to monetary bonus, with a reward of 5 cents if the final prediction was correct and a loss of 2 cents if otherwise (in addition to a base pay of $3). Prior research showed that such a reward design is effective in motivating participants to optimize the decision outcome [2, 31]”
From Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“frequently updating the AI algorithm may cause confusion to the human decision-maker, who may accept or reject the AI's recommendations at a wrong time, even if the algorithm's overall performance improved [2].”
From Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“A key to success in AI-assisted decision making is to form a correct mental model of the model's error boundaries [2].”
From Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“well-calibrated AI confdence scores can represent the model's actual correctness likelihood (CL)”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [5]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 2×
“2016), in-hospital mortality prediction (Will a patient die in the hospital?) (Johnson et al. 2016; Harutyunyan et al. 2017), and credit risk assessment (Will a borrower fail to pay back?)3.”
BackgroundThe Psychology of Everyday Things
Massaro, D. W., & Norman, D. A. (1990) · The American Journal of Psychology · doi:10.2307/1423268 · Cited by 3,466
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · cited 1×
“Unfortunately, confidence scores of ML classifiers are often not calibrated (Nguyen, Yosinski, and Clune 2015) or a meaningful confidence definition may not exist due to the complexity of the task.”
BackgroundImplicit Learning and Tacit Knowledge
Reber, A. S. (2003) · Essential Sources in the Scientific Study of Consciousness · doi:10.7551/mitpress/2834.003.0047 · Cited by 2,001
The foundations of cost-sensitive learning
Elkan, C. (2001) · Cited by 1,945
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“…by real-world AI applications for reducing patient readmissions and other costly outcomes in healthcare (Bayati et al. 2014; Wiens, Guttag, and Horvitz 2016; Caruana et al. 2015), and motivates the need for reducing the cost of disruption caused by updates that violate users’ mental models.”
BackgroundMethodSome observations on mental models
Norman, D. A. (1983) · Medical Entomology and Zoology · Cited by 1,549
Do Deep Nets Really Need to be Deep?
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,335 · cited 2×
“…the log loss between the prediction probabilities of h1 and h2: D′(x, y, h1, h2) = L(x, h1, h2) (4) Equation 4 is used in model distillation (Ba and Caruana 2014; Hinton, Vinyals, and Dean 2015), where the aim is to train a shallower, less expensive model by imitating the probabilities of…”
MethodMultitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · cited 2×
“2016), in-hospital mortality prediction (Will a patient die in the hospital?) (Johnson et al. 2016; Harutyunyan et al. 2017), and credit risk assessment (Will a borrower fail to pay back?)3.”
Background
Research works printed but not linked: 9
- Carroll, J. M., and Olson, J. R. 1988. Mental models in human-computer interaction. In Handbook of HumanComputer Interaction . Elsevier. 45-65.
- DeChurch, L. A., and Mesmer-Magnus, J. R. 2010. The cognitive underpinnings of effective teamwork: A metaanalysis. Journal of Applied Psychology 95(1):32.
- Grosz, B. J., and Kraus, S. 1999. The evolution of sharedplans. In Foundations of Rational Agency . Springer.
- Grosz, B. J. 1996. Collaborative systems (AAAI-94 presidential address). AI Magazine 17(2):67.
- Hoff, K. A., and Bashir, M. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors 57(3):407-434.
- Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 201611835.
- Lasecki, W.; Miller, C.; Sadilek, A.; Abumoussa, A.; Borrello, D.; Kushalnagar, R.; and Bigham, J. 2012a. Real-time captioning by groups of non-experts. In UIST , 23-34. ACM.
- Wang, D.; Khosla, A.; Gargeya, R.; Irshad, H.; and Beck, A. H. 2016. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718 . · arXiv
- Zhou, D.; Bousquet, O.; Lal, T. N.; Weston, J.; and Sch¨ olkopf, B. 2004. Learning with local and global consistency. In NIPS , 321-328.
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
“ter. For papers with multiple domains or experiments, we took one sample with the most comparable human and AI performance. In this paper, we focus explicitly on AI-assisted decision making scenarios [3], where an AI assistant (e.g., a classification model) makes recommendations to a human (e.g., a judge), who is resp…”
BackgroundSelf-citationZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
“[2] to test mental modeling of error boundaries.”
BackgroundInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“ter. For papers with multiple domains or experiments, we took one sample with the most comparable human and AI performance. In this paper, we focus explicitly on AI-assisted decision making scenarios [3], where an AI assistant (e.g., a classification model) makes recommendations to a human (e.g., a judge), who is resp…”
BackgroundSelf-citationChallenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638
“[15] introduced the notion of compatibility of an AI update.”
BackgroundWhen combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
Metadata
- Authors
- Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- University of Michigan (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 2429-2437
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2905034244
- Cited by
- 284
- References
- 21
- Citations by year
- 2026: 31 · 2025: 49 · 2024: 53 · 2023: 50 · 2022: 48 · 2021: 29 · 2020: 20 · 2019: 4
- FWCI
- 9.66
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- Microsoft Research
- Office of Naval Research (N00014- 18-1-2193, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human-AI teaming, human-AI decision-making, user experience
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. https://app.cochat.ai/papers/10.1609/aaai.v33i01.33012429
- The paper
- Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019). Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v33i01.33012429
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.