Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
Clinician skepticism toward opaque AI hinders adoption in high-stakes healthcare. We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making. By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations and LLM-driven diagnostic recommendations. Through a within-subjects counterbalanced study with 16 clinicians across nephrology and obstetrics, we comprehensively evaluated AICare using objective measures (task completion time and error rate), subjective assessments (NASA-TLX, SUS, and confidence ratings), and semi-structured interviews. Our findings indicate AICare’s reduced cognitive workload. Beyond performance metrics, qualitative analysis reveals that trust is actively constructed through verification, with interaction strategies diverging by expertise: junior clinicians used the system as cognitive scaffolding to structure their analysis, while experts engaged in adversarial verification to challenge the AI’s logic. This work offers design implications for creating AI systems that function as transparent partners, accommodating diverse reasoning styles to augment rather than replace clinical judgment.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citation"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 1×
“Common techniques include feature importance methods like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations), which highlight the factors driving a prediction [40, 54].”
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 1×
“We assess cognitive workload using the NASA Task Load Index (NASA-TLX) [26, 27] (see Appendix F.3).”
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“On one hand, algorithmic performance continues to break records in controlled experiments [19, 51, 65, 82, 84].”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“On one hand, algorithmic performance continues to break records in controlled experiments [19, 51, 65, 82, 84].”
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Research works printed but not linked: 5
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321–1330.
- Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017).
- John D Manning Beatriz E Marciano and James J Cimino. 2013. Visualizing the data–using lifelines2 to gain insights from data drawn from a clinical data repository. AMIA summits on translational science proceedings 2013 (2013) 168.
- National Academy of Medicine; The Learning Health System Series. 2023. Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. National Academies Press (US), Washington (DC).
- Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. 2019. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference. PMLR, 359–380.
Cited by
Choube, A., Shrestha, S., Le, H., Li, J.-C., Swain, V. D., & Mishra, V. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3832033 · Cited by 3
(2000) · Time to knit · doi:10.1145/3772318.3808884
OneEHR: Reproducible and AI Agent-Ready Longitudinal EHR Analysis Toolkit
Zhu, Y., Wang, Z., Gu, L., Sui, D., Wang, Y., Harrison, E. M., Fu, T., Gao, J., Yu, L., & Ma, L. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3816474
Self-citationArtificial Intelligence–Driven Decision Fatigue in Healthcare Management Systems
Abbas, K. D., Fahmi, A. M., & Kanbar, A. B. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39711-9_18
Metadata
- Authors
- Zhu, Y., Sui, D., Wang, Z., Hu, X., Gu, L., Qi, Y., Wu, T., Wang, L., Wei, Y., Tang, W., Cui, Z., Wang, Y., Yu, L., Harrison, E. M., Gao, J., & Ma, L.
- Affiliations
- Peking University (CN)
- Hong Kong University of Science and Technology (HK)
- Peking University Third Hospital (CN)
- Xuzhou Medical College (CN)
- University of Hong Kong (HK)
- University of Edinburgh (GB)
- Health Data Research UK (GB)
- ORCID
- Venue
- Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2026-04-13
- Pages
- 1-26
- OpenAlex
- W7154143872
- Semantic Scholar
- 13365a9a9f9d796af8dc1d69460c8fab660fa915
- Cited by
- 3
- References
- 69
- Citations by year
- 2026: 3
- FWCI
- 25.86
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (62402017, 82470774, 82561128248, 82504426)
- Research Grants Council of Hong Kong (27206123, 17200125, C5055-24G, T45-401/22-N)
- Hong Kong Innovation and Technology Fund (GHP/318/22GD)
- Beijing Natural Science Foundation (QY25224, L244063, L244025)
- Peking University (Clinical Medicine Plus X Pilot Project) (2024YXXLHGG007)
- Peking University (\"TengYun\" Clinical Research Program) (TY2025015)
- Beijing Traditional Chinese Medicine Science and Technology Development Fund (BJZYZD-2025-13)
- Beijing Municipal Health Commission Research Ward Excellence Clinical Research Program (BRWEP2024W032150205)
- Xuzhou Scientific Technological Projects (KC23143)
- Health Data Research UK-The Alan Turing Institute Wellcome PhD Programme in Health Data Science (218529/Z/19/Z)
- Baidu Scholarship (/)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- AI copilots, explainable AI, clinical decision-making, clinician trust in AI, mental workload, nephrology, obstetrics, cognitive scaffolding, adversarial verification, AI adoption in healthcare, NASA-TLX, System Usability Scale
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics. https://app.cochat.ai/papers/10.1145/3772318.3791458
- The paper
- Zhu, Y., Sui, D., Wang, Z., Hu, X., Gu, L., Qi, Y., Wu, T., Wang, L., Wei, Y., Tang, W., Cui, Z., Wang, Y., Yu, L., Harrison, E. M., Gao, J., & Ma, L. (2026). Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3772318.3791458
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.