Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support
Open accessAbstract
As AI-based clinical decision support (AI-CDS) is introduced in more and more aspects of healthcare services, HCI research plays an increasingly important role in designing for complementarity between AI and clinicians. However, current evaluations of AI-CDS often fail to capture when AI is and is not useful to clinicians. This position paper reflects on our work and influential AI-CDS literature to advocate for moving beyond evaluation metrics like Trust, Reliance, Acceptance, and Performance on the AI's task (what we term the "trap" of human-AI collaboration). Although these metrics can be meaningful in some simple scenarios, we argue that optimizing for them ignores important ways that AI falls short of clinical benefit, as well as ways that clinicians successfully use AI. As the fields of HCI and AI in healthcare develop new ways to design and evaluate CDS tools, we call on the community to prioritize ecologically valid, domain-appropriate study setups that measure the emergent forms of value that AI can bring to healthcare professionals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work shows an evolution toward more rigorous, multifaceted trust assessments, in contrast to many earlier XAI studies that demonstrated technical feasibility and interpretability but did not empirically evaluate user trust or integration into real clinical workflows.
“The evolution toward more rigorous, multifaceted trust assessments in recent work [53,63-65] stands in contrast to many earlier XAI studies [11,45,46,66,67], which often demonstrated technical feasibility and interpretability but did not empirically evaluate user trust or integration into real clinical workflows.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [63]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 1
Research works printed but not linked: 19
- Roy Adams, Katharine E. Henry, Anirudh Sridharan, Hossein Soleimani, Andong Zhan, Nishi Rawat, Lauren Johnson, David N. Hager, Sara E. Cosgrove, Andrew Markowski, Eili Y. Klein, Edward S. Chen, Mustapha O. Saheed, Maureen Henley, Sheila Miranda, Katrina Houston, Robert C. Linton, Anushree R. Ahluwalia, Albert W. Wu, and Suchi Saria. 2022. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nature Medicine 28, 7 (July 2022), 1455-1460. doi:10. 1038/s41591-022-01894-0 Publisher: Nature Publishing Group. · DOI · Paper page
- Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). doi:10.1145/3411764.3445717 · DOI · Paper page
- Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis. 2020. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. Conference on Human Factors in Computing Systems - Proceedings (2020), 1-12. doi:10.1145/3313831.3376718 · DOI · Paper page
- Francisco Maria Calisto, João Fernandes, Margarida Morais, Carlos Santiago, João Maria Abrantes, Nuno Nunes, and Jacinto C Nascimento. 2023. Assertiveness- based agent communication for a personalized medicine on medical imaging diagnosis. In Proceedings of the 2023 CHI conference on human factors in computing systems. 1-20.
- Lorenzo Corti, Rembrandt Oltmans, Jiwon Jung, Agathe Balayn, Marlies Wijsen- beek, and Jie Yang. 2024. "It Is a Moving Process": Understanding the Evolution of Explainability Needs of Clinicians in Pulmonary Medicine. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-21.
- Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Adam Rodman, and Jonathan H. Chen. 2024. Large Language Model Influence on Diagnostic Reasoning: A Ran- domized Clinical Trial. JAMA Network Open 7, 10 (Oct. 2024), e2440969-e2440969. doi:10.1001/jamanetworkopen.2024.40969 · DOI · Paper page
- Maia Jacobs, Melanie F. Pradier, Thomas H. McCoy, Roy H. Perlis, Finale Doshi- Velez, and Krzysztof Z. Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant se- lection. Translational Psychiatry 11, 1 (2021). doi:10.1038/s41398-021-01224-x Publisher: Springer US. · DOI · Paper page
- Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. 2024. "I'm Not Sure, But...": Examining the Im- pact of Large Language Models' Uncertainty Expression on User Reliance and Trust. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24). 822-835. doi:10.1145/3630106.3658941 · DOI · Paper page
- Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In 2023 ACM Conference on Fairness, Accountability, and Transparency. ACM, Chicago IL USA, 1369-1385. doi:10.1145/3593013.3594087 · DOI · Paper page
- John D. Lee and Katrina A. See. 2004. Trust in automation: Designing for appro- priate reliance. Human Factors 46, 1 (2004), 50-80. doi:10.1518/hfes.46.1.50_30392 · DOI · Paper page
- Drew Prinster, Amama Mahmood, Suchi Saria, Jean Jeudy, Cheng Ting Lin, Paul H. Yi, Chien-Ming Huang, and Shannyn Wolfe. 2024. Care to Explain? AI Explanation Types Differentially Impact Chest Radiograph Diagnostic Per- formance and Physician Trust in AI. Radiology 313, 2 (Nov. 2024), e233261. doi:10.1148/radiol.233261 · DOI
- Venkatesh Sivaraman, Leigh A Bukowski, Joel Levin, Jeremy M. Kahn, and Adam Perer. 2023. Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). doi:10.1145/ 3544548.3581075 · DOI · Paper page
- Venkatesh Sivaraman, Yejun Kwak, Courtney Kuza, Qingnan Yang, Kayleigh Adamson, Katie Suda, Lu Tang, Walid Gellad, and Adam Perer. 2025. Static Algo- rithm, Evolving Epidemic: Understanding the Potential of Human-AI Risk Assess- ment to Support Regional Overdose Prevention. arXiv preprint arXiv:2502.10542 (2025). · arXiv
- Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, John Paoli, Susana Puig, Cliff Rosendahl, H. Peter Soyer, Iris Zalaudek, and Harald Kittler. 2020. Human-computer collaboration for skin cancer recog- nition. Nature Medicine 26, 8 (2020), 1229-1234. doi:10.1038/s41591-020-0942-0 Publisher: Springer US. · DOI · Paper page
- Violet Turri, Katelyn Morrison, Katherine-Marie Robinson, Collin Abidi, Adam Perer, Jodi Forlizzi, and Rachel Dzombak. 2024. Transparency in the Wild: Navigat- ing Transparency in a Deployed AI System to Broaden Need-Finding Approaches. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. 1494- 1514.
- Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. 2024. When com- binations of humans and AI are useful: A systematic review and meta-analysis. arXiv preprint arXiv:2405.06087 (2024). · arXiv
- Qian Yang, Aaron Steinfeld, and John Zimmerman. 2019. Unremarkable AI: Fitting intelligent decision support into critical, clinical decision-making pro- cesses. Conference on Human Factors in Computing Systems -Proceedings (2019). doi:10.1145/3290605.3300468 · DOI · Paper page
- Nur Yildirim, Susanna Zlotnikov, Deniz Sayar, Jeremy M. Kahn, Leigh A Bukowski, Sher Shah Amin, Kathryn A. Riman, Billie S. Davis, John S. Minturn, Andrew J. King, Dan Ricketts, Lu Tang, Venkatesh Sivaraman, Adam Perer, Sarah M. Preum, James McCann, and John Zimmerman. 2024. Sketching AI Concepts with Capa- bilities and Examples: AI Innovation in the Intensive Care Unit. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 1-18. doi:10.1145/3613904.3641896 · DOI
- Zelun Tony Zhang, Cara Storath, Yuanting Liu, and Andreas Butz. 2023. Re- silience Through Appropriation: Pilots' View on Complex Decision Support. In Proceedings of the 28th International Conference on Intelligent User Interfaces. ACM, Sydney NSW Australia, 397-409. doi:10.1145/3581641.3584056 · DOI
- Authors
- Sivaraman, V., Morrison, K., Epperson, W., & Perer, A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.07423
- OpenAlex
- W6948262408
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- human-AI collaboration, AI evaluation metrics, AI reliance, trust in AI, AI acceptance, clinical benefit, artificial intelligence in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
