Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user’s domain expertise, mental models of an AI system, trust in recommendations, and more. This article reports on a study that examines users’ interactions with three simulated algorithmic models, all with equivalent accuracy rates but each tuned differently in terms of true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Users completed 140 trials across multiple stages, first without an AI and then with recommendations from an AI-Assistant. Although all users had prior experience with the task, their levels of proficiency varied widely. Our results demonstrated that while recommendations from an AI-Assistant can aid in users’ decision making, several underlying factors, including user base expertise and complementary human-AI tuning, significantly impact the overall team performance. First, users’ base performance matters, particularly in comparison to the performance level of the AI. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. Second, tuning an AI algorithm to complement users’ strengths and weaknesses also significantly impacted users’ performance. For example, users in our study were better at detecting flowing blood vessels, so when the AI was tuned to reduce false negatives (at the expense of increasing false positives), users were able to reject those recommendations more easily and improve in accuracy. Finally, users’ perception of the AI’s performance relative to their own performance had an impact on whether users’ accuracy improved when given recommendations from the AI. Overall, this work reveals important insights on the complex interplay of factors influencing Human-AI collaboration and provides recommendations on how to design and tune AI algorithms to complement users in decision-making tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As clinicians gain experience, they develop a deeper understanding of clinical contexts and decision-making processes, which may influence both trust in and reliance on CDSSs [ 8 , 15 - 17 ].”
From Psychological Factors Influencing Appropriate Reliance on AI-enabled Clinical Decision Support Systems: Experimental Web-Based Study Among Dermatologists (2025) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Collaborative systems have shown that humans can correct model predictions in specific instances1, and that humans, in collaboration with a Machine Learning (ML) model, can achieve higher performance than either the expert or the model alone3.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Proceedings of the International Joint Conference on Artificial Intelligence 2007
Bundy, A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 2,761
Proceedings of the 16th international joint conference on Artificial Intelligence - IJCAI '99
Colton, S., Bundy, A., & Walsh, T. (1999) · Cited by 2,500
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Previous work has emphasized the importance of providing such information in order to set expectations on performance [3, 33, 61].”
BackgroundSelf-citationConfirmation, disconfirmation, and information in hypothesis testing
Klayman, J., & Ha, Y.-W. (1987) · Psychological Review · doi:10.1037/0033-295x.94.2.211 · Cited by 2,094 · cited 1×
“Klayman & Ha also find that the most critical way to test hypotheses is a "positive test strategy," where participants examine cases with the target characteristic and retrieve knowledge from their memory related to the assumption provided by an anchor [32].”
BackgroundAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,966
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 2×
“This strategy suggests that individuals are more likely to use and rely on algorithms when they have the opportunity to possibly correct and minimize errors in their decision-making judgement, even when the algorithmic-assistance is imperfect [15].”
BackgroundThe eBird enterprise: An integrated approach to development and application of citizen science
Sullivan, B. L., Aycrigg, J. L., Barry, J. H., Bonney, R. E., Bruns, N., Cooper, C. B., Damoulas, T., Dhondt, A. A., Dietterich, T., Farnsworth, A., et al. (2014) · Biological Conservation · doi:10.1016/j.biocon.2013.11.003 · Cited by 1,155
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3×
“This inding is further supported by a series of work at the intersection of human-computer interaction and machine learning interpretability, which highlights concerns around the phenomenon of increased, but not appropriate, reliance on automation [6, 9, 34, 46, 58].”
BackgroundSelf-citationDeep Learning for Identifying Metastatic Breast Cancer
Wang, D., Khosla, A., Gargeya, R., Irshad, H., & Beck, A. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05718 · Cited by 756
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Self-citationManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“This inding is further supported by a series of work at the intersection of human-computer interaction and machine learning interpretability, which highlights concerns around the phenomenon of increased, but not appropriate, reliance on automation [6, 9, 34, 46, 58].”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Similarities and differences between human–human and human–automation trust: an integrative review
Madhavan, P., & Wiegmann, D. A. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500337708 · Cited by 577
Do as AI say: susceptibility in deployment of clinical decision-aids
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00385-9 · Cited by 543 · cited 1×
“ran a studywhere physicians and radiologists were given chest X-rays and diagnostic advice [23].”
MethodUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“Previous work has emphasized the importance of providing such information in order to set expectations on performance [3, 33, 61].”
BackgroundStubborn Reliance on Intuition and Subjectivity in Employee Selection
Highhouse, S. (2008) · Industrial and Organizational Psychology · doi:10.1111/j.1754-9434.2008.00058.x · Cited by 498
Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., & Stumpe, M. C. (2018) · American Journal of Surgical Pathology · doi:10.1097/pas.0000000000001151 · Cited by 489 · cited 3×
“[52] explored the impact of computer assistance in the ield of pathology to improve interpretation of images and clinical care [52].”
BackgroundMethodBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 2×
“[4, 5] show that other AI properties beyond AI accuracy also impact team performance, including the predictability of errors and whether the model errors remain backward compatible over updates.”
BackgroundSelf-citationMaking sense of recommendations
Yeomans, M., Shah, A., Mullainathan, S., & Kleinberg, J. (2019) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2118 · Cited by 459
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
The Perceived Utility of Human and Automated Aids in a Visual Detection Task
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494856 · Cited by 386 · cited 2×
“Dzinolet et al. conducted a series of studies to determine whether people rely on automated and human group members differently [18].”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377 · cited 2×
“This inding is further supported by a series of work at the intersection of human-computer interaction and machine learning interpretability, which highlights concerns around the phenomenon of increased, but not appropriate, reliance on automation [6, 9, 34, 46, 58].”
BackgroundWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 1×
“This inding is further supported by a series of work at the intersection of human-computer interaction and machine learning interpretability, which highlights concerns around the phenomenon of increased, but not appropriate, reliance on automation [6, 9, 34, 46, 58].”
BackgroundCombining human and machine intelligence in large-scale crowdsourcing
Kamar, E. S., Hacker, S., & Horvitz, E. (2012) · doi:10.5555/2343576.2343643 · Cited by 318
Understanding algorithm aversion: When is advice from automation discounted?
Prahl, A., & Van Swol, L. (2017) · Journal of Forecasting · doi:10.1002/for.2464 · Cited by 308 · cited 1×
“People want to understand where a recommendation is coming from, and ind that they have more in common with human-based recommendations as opposed to algorithms [12, 47, 60].”
BackgroundProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 3×
“[4, 5] show that other AI properties beyond AI accuracy also impact team performance, including the predictability of errors and whether the model errors remain backward compatible over updates.”
BackgroundMethodSelf-citationKocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
Comparing Is Believing: A Selective Accessibility Model of Judgmental Anchoring
Mussweiler, T., & Strack, F. (1999) · European Review of Social Psychology · doi:10.1080/14792779943000044 · Cited by 267
Zooniverse: observing the world's largest citizen science platform
Simpson, R., Page, K. R., & De Roure, D. (2014) · Proceedings of the 23rd International Conference on World Wide Web · doi:10.1145/2567948.2579215 · Cited by 239
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 1×
“, who showed that humans are less likely to follow the AI recommendation when the stated score is an incorrect estimate of risk [13].”
BackgroundDeciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · cited 1×
“At the same time, fully decoupling anchoring bias efects from intentional agreement [48] would require experimentation with other types of worklows where the human takes a temporary decision irst and then has a chance to revise it after getting assistance from the AI.”
BackgroundEffect of Robot Performance on Human–Robot Trust in Time-Critical Situations
Robinette, P., Howard, A. M., & Wagner, A. R. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2648849 · Cited by 205
Wang, G., Liu, X., Shen, J., Wang, C., Li, Z., Ye, L., Wu, X., Chen, T., Wang, K., Zhang, X., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00704-1 · Cited by 190
Accepting Error to Make Less Error
Einhorn, H. J. (1986) · Journal of Personality Assessment · doi:10.1207/s15327752jpa5003_8 · Cited by 179
Why trust an algorithm? Performance, cognition, and neurophysiology
Alexander, V., Blinder, C., & Zak, P. J. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.026 · Cited by 175
Persuasiveness of expert systems
Dijkstra, J. J., Liebrand, W. B. G., & Timminga, E. (1998) · Behaviour & Information Technology · doi:10.1080/014492998119526 · Cited by 161 · cited 1×
“Djikstra et al. found that individuals believe algorithmic systems represent a more objective and rational perspective than a human being [16, 17].”
BackgroundNugent, J. (2018) · Science Scope · doi:10.2505/4/ss18_041_07_12 · Cited by 138
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
User agreement with incorrect expert system advice
Dijkstra, J. J. (1999) · Behaviour and Information Technology · doi:10.1080/014492999118832 · Cited by 124
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118
“Who goes first? Influences of human-AI workflow on decision making in clinical imaging,”
Fogliato, R., Chappidi, S., Lungren, M., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K., & Nushi, B. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533193 · Cited by 103
Self-citationWhat can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92
Bracko, O., Vinarcsik, L. K., Cruz Hernández, J. C., Ruiz-Uribe, N. E., Haft-Javaherian, M., Falkenhain, K., Ramanauskaite, E. M., Ali, M., Mohapatra, A., Swallow, M. A., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-65908-y · Cited by 85
Self-citationYou’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · cited 1×
“This behavior is salient when an individual’s trust surpasses the algorithm’s real capabilities [5, 9, 11, 35].”
BackgroundThe effects of automatic speech recognition quality on human transcription latency
Gaur, Y., Lasecki, W. S., Metze, F., & Bigham, J. P. (2016) · International Web for All Conference (W4A) · doi:10.1145/2899475.2899478 · Cited by 48
Falkenhain, K., Ruiz‐Uribe, N. E., Haft‐Javaherian, M., Ali, M. A., contributors, S. C., Michelucci, P. E., Schaffer, C. B., & Bracko, O. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235691 · Cited by 26
Self-citationInvestigating Human + Machine Complementarity for Recidivism Predictions
Tan, S., Adebayo, J., Inkpen, K. M., & Kamar, E. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09123 · Cited by 25
Self-citationDeciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-makingPreprint
Rastogi, C., Zhang, Y., Wei, D., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07938 · Cited by 9
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
Gonzalez, A. V., Bansal, G., Fan, A., Jia, R., Mehdad, Y., & Iyer, S. (2020) · arXiv · doi:10.48550/arxiv.2012.15075 · Cited by 7
The People and Serendipity of the EyesOnALZ project
Michelucci, P. E. (2019) · Narrative Inquiry in Bioethics · doi:10.1353/nib.2019.0013 · Cited by 2
Self-citationCitizen Science: Accelerating Alzheimer’s Research With Stall Catchers
Nugent, J. (2021) · The Science Teacher · doi:10.1080/00368555.2021.12293593 · Cited by 1
Research works printed but not linked: 2
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
Cited by
From explainable to interactive AI: A literature review on current trends in human-AI interaction
Raees, M., Meijerink, I., Lykourentzou, I., Khan, V.-J., & Papangelis, K. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103301 · Cited by 180
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
Complementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 237
“…the story: On the one hand, various studies have demonstrated that human-AI teams are able to outperform human individuals (Alufaisan et al., 2021; Inkpen et al., 2023; Liu et al., 2021; Sarkar et al., 2023)—often not analyzing, though, whether they also surpass the AI model’s individual…”
AI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 185
“Fundamental biases, such as overtrust and overreliance, impair performance [46] up to the point that the interaction decreases overall performance as compared to having no AI at all [7, 85].”
A cognitive approach to human–AI complementarity in dynamic decision-making
González, C., & Heidari, H. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00499-x · Cited by 92
“Data-driven AI systems play an essential part in supporting dynamic decision-making in many domains 4,5 .”
Metadata
- Authors
- Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G.
- Affiliations
- Microsoft (United States) (US)
- University of Virginia (US)
- University of Washington (US)
- University of Michigan (US)
- Human Computation Institute (US)
- Ludwig-Maximilians-Universität München (DE)
- Western Washington University (US)
- ORCID
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-09-23
- Volume
- 30
- Issue
- 5
- Pages
- 1-29
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4323841539
- Cited by
- 132
- References
- 58
- Citations by year
- 2026: 48 · 2025: 44 · 2024: 36 · 2023: 4
- FWCI
- 39.07
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI collaboration, shared decision-making, user expertise, false positive and false negative rates, user proficiency, trust in AI recommendations, team performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.