Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user’s domain expertise, mental models of an AI system, trust in recommendations, and more. This article reports on a study that examines users’ interactions with three simulated algorithmic models, all with equivalent accuracy rates but each tuned differently in terms of true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Users completed 140 trials across multiple stages, first without an AI and then with recommendations from an AI-Assistant. Although all users had prior experience with the task, their levels of proficiency varied widely. Our results demonstrated that while recommendations from an AI-Assistant can aid in users’ decision making, several underlying factors, including user base expertise and complementary human-AI tuning, significantly impact the overall team performance. First, users’ base performance matters, particularly in comparison to the performance level of the AI. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. Second, tuning an AI algorithm to complement users’ strengths and weaknesses also significantly impacted users’ performance. For example, users in our study were better at detecting flowing blood vessels, so when the AI was tuned to reduce false negatives (at the expense of increasing false positives), users were able to reject those recommendations more easily and improve in accuracy. Finally, users’ perception of the AI’s performance relative to their own performance had an impact on whether users’ accuracy improved when given recommendations from the AI. Overall, this work reveals important insights on the complex interplay of factors influencing Human-AI collaboration and provides recommendations on how to design and tune AI algorithms to complement users in decision-making tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As clinicians gain experience, they develop a deeper understanding of clinical contexts and decision-making processes, which may influence both trust in and reliance on CDSSs [ 8 , 15 - 17 ].”
From Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists (2025) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Collaborative systems have shown that humans can correct model predictions in specific instances1, and that humans, in collaboration with a Machine Learning (ML) model, can achieve higher performance than either the expert or the model alone3.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331 · cited 1×
“Such adjustments can be implemented in practice either by oversampling positive or negative examples [26], assigning diferent costs to diferent examples during learning [20], or simply by assigning diferent conidence thresholds for detection depending on whether the goal is to reduce the number of false positives or f…”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Proceedings of the International Joint Conference on Artificial Intelligence 2007
Bundy, A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 2,761
Proceedings of the 16th international joint conference on Artificial Intelligence - IJCAI '99
Colton, S., Bundy, A., & Walsh, T. (1999) · Cited by 2,500
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Previous work has emphasized the importance of providing such information in order to set expectations on performance [3, 33, 61].”
BackgroundSelf-citationConfirmation, disconfirmation, and information in hypothesis testing
Klayman, J., & Ha, Y.-W. (1987) · Psychological Review · doi:10.1037/0033-295x.94.2.211 · Cited by 2,094 · cited 1×
“Klayman & Ha also find that the most critical way to test hypotheses is a "positive test strategy," where participants examine cases with the target characteristic and retrieve knowledge from their memory related to the assumption provided by an anchor [32].”
BackgroundAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Research works printed but not linked: 2
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
Cited by
Items 101–133 of 133
Future-Facing Pedagogy: Empowering Students Through AI Literacies
Wallace, J. J., Perino, J., & Katzarska-Miller, I. I. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3000-6.ch004 · Cited by 1
Hanák, R., & Rybjanský, D. (2026) · Inproforum ... · doi:10.32725/978-80-7694-143-4.18
Perceptions of AI-Driven EdTech: Nationwide Survey and Focus Group Insights from Key End Users
Min, A., Dickerson, K. L., Park, S., Dotch, E., Han, A., Rubin, J. D., Lombard, E. J., Chen, K., Divanji, R. A., Odgers, C. L., et al. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3798278 · Cited by 3
“Furthermore, prior work suggests that users’ trust in AI varies depending on their level of expertise and on how they perceive the AI’s role ( e.g., as a colleague, assistant, tool, or distractor)[1, 40].”
Exploring a Collaborative Gamified Approach to Vision-Language Model Evaluation
Gao, F., Wang, J. R. M., Sayuda, T., Pennington, M., & Sugano, Y. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799410
“Recent work further suggests that human-AI disagreement constitutes a valuable signal for diagnosing model blind spots and guiding iterative system improvement, positioning comparison not merely as evaluation but as a tool for discovery and sensemaking [22, 25, 26].”
InterPilot: Exploring the Design Space of AI-assisted Job Interview Support for HR Professionals
Xu, Z., Xia, Z., Zhu, Z., Boonprakong, N., Chen, Y.-A., Zbib, R., Carrino, C. P., & Lee, Y. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798373 · Cited by 1
Luo, Y., Gkouskos, D., & Russo, N. L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790672
The Evolution of Real-Time Collaborative AI Workspace
Kadu, N., Khanna, P., Joshi, A., Jail, D., & Jaiswal, V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19690-3_21
The Current Intensity of AI Use and the Perception of AI Applications in Strategic Decision-Making
Grzesik, K., Brajer-Marczak, R., Nadolny, M., Liwarska-Fulczyk, K., Chomiak-Orsa, I., & Marciszewska, A. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-21184-2_17
AI Error Difficulty Modulates the Effectiveness of Explainability in Decision Support Systems
Rieger, T., Schindler, H., Koch, K., & Onnasch, L. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3817603 · Cited by 1
Jones, L., Higgins, B. E., Devraj, K., Crabb, D. P., Thomas, P. B. M., & Moosajee, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02817-w
For Those About to Rely—A Taxonomy of Experimental Studies on AI Reliance: Completed Research Paper
Schaschek, M., Spatscheck, N., & Winkelmann, A. (2026) · Lecture notes in information systems and organisation · doi:10.1007/978-3-032-08480-4_2
Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling
Adeseye, A., Isoaho, J., Virtanen, S., & Tahir, M. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536612 · Cited by 1
“However, for effective communication, AI systems must understand the user’s expertise level to adapt communication style [6], so that it provides meaningful support that builds user trust [7].”
Multimodal perceptual curation for cognitive autonomy in Generation Z decision making
Zhu, Z., & Wang, K. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1868429
Designing cognition-driven Human-AI teaming: A Three-Layer Cognitive Framework (TriC)
Yin, Y., Jin, E., Pan, H., (324865), X. J., Zhang, Y., Liu, Y., Shen, M., & Gao, Z. (2026) · Computers in Human Behavior Reports · doi:10.2139/ssrn.6890561
Morality Meets Machine: How Evaluator and Agent Gender Shape Human-AI Moral Judgment
He, Y., Deng, G., Liu, C., Gan, T., Cui, F., Luo, Y.-J., & Gu, R. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2676767
Dang, N. Y. (2026) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet69664.2026.11561598
“PER also plays an important role in optimizing the balance between system compatibility (CM) and accuracy of AI systems in BL environments [15,16].”
Zhao, J. Y., & Shi, J. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107295 · Cited by 3
Tariq, S., Singh, R., Chhetri, M. B., Nepal, S., & Paris, C. (2026) · Research Square · doi:10.2139/ssrn.7002953
Understanding User Experience in AI-Based Evaluation Systems: A Systematic Literature Review
Park, H., Chung, G., Kim, C. Y., & Kim, H. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29653-5_21
Co-Rationality Mechanisms in AI Decision Support
Hao, X. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2694374
Bai, L., Han, D. Y., & Hennessy, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10309041/v1
Gong, L., & Gong, Y. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1886229 · Cited by 2
“In contrast, users with higher domain self-efficacy may depend less on such perceptual cues because of their stronger independent evaluative capabilities (Inkpen et al., 2023).”
Fu, N., Li, J., Hallin, A., Ivory, C. J., Piscicelli, L., Moreno, A., Menasalvas, E., & Andersson, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7162963
Bias in Algorithm‐Assisted Decision‐Making: A Theory‐Based Intervention
Li, A., & Fan, S. (2026) · Journal of Applied Social Psychology · doi:10.1111/jasp.70083
Legitimizing AI Through Environmental CSR: Effects on Public Attitudes and IntentionsPreprint
Buckley, C., Troy, C. L. C., Ryan, K., & Cardona, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7381189
Rethinking Creative Ideation: Toward Seamless Synergetic AI in Human–AI Design Collaboration
Zhang, C., Feng, Z., Zhou, T., & Han, T. (2025) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid68789.2025.00079
Enhancing AI Use: How Complementary System Information Drives Delegation Frequency and Effectiveness
Taudien, A., Fügener, A., Gupta, A., & Ketter, W. (2026) · Information Systems Research · doi:10.1287/isre.2024.1330
González, O., & Alvarado-Rodríguez, F. J. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37438-7_50
Tuo, B., Wang, B. T., Miller, T., & Demartini, G. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3848504
“These conditions are consistent with prior work on human–AI complementarity, including studies on differentiated information [25, 55], the role of strong human expertise [33, 68], and interface designs that surface rich yet balanced information to support independent judgment [15, 22, 48].”
Machain, J. D. J. R., Lizárraga, D. M., García, V. M. M., Romero, Y. D., Ramírez, M. I. T., & Lizarraga, P. A. V. (2026) · SAP Engineering and Industry · doi:10.62486/ei202670
When AI joins the team: The impact of human-AI diversity on willingness to work with teammates
Lu, J., Zhang, J., & Lu, Y. (2026) · Decision Support Systems · doi:10.1016/j.dss.2026.114786
Jin, Y., Sun, Y., & Zhang, H. L. (2026) · Legal and Criminological Psychology · doi:10.1111/lcrp.70071
Performance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2024) · arXiv · doi:10.48550/arxiv.2409.16708 · Cited by 71
“Fundamental biases, such as overtrust and overreliance, impair performance [46] up to the point that the interaction decreases overall performance as compared to having no AI at all [7, 85].”
Metadata
- Authors
- Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G.
- Affiliations
- Microsoft (United States) (US)
- University of Virginia (US)
- University of Washington (US)
- University of Michigan (US)
- Human Computation Institute (US)
- Ludwig-Maximilians-Universität München (DE)
- Western Washington University (US)
- ORCID
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-09-23
- Volume
- 30
- Issue
- 5
- Pages
- 1-29
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4323841539
- Cited by
- 132
- References
- 58
- Citations by year
- 2026: 48 · 2025: 44 · 2024: 36 · 2023: 4
- FWCI
- 39.07
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, shared decision-making, user expertise, false positive and false negative rates, user proficiency, trust in AI recommendations, team performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.