From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis
Open accessAbstract
Early studies of large language models (LLMs) in clinical settings have largely treated artificial intelligence (AI) as a tool rather than an active collaborator. As LLMs now demonstrate expert-level diagnostic performance, the focus shifts from whether AI can offer valuable suggestions to how it can be effectively integrated into physicians’ diagnostic workflows. We conducted a randomized controlled trial (n=70 clinicians) to evaluate the value of employing a custom GPT system designed to engage collaboratively with clinicians on diagnostic reasoning challenges. The collaborative design began with independent diagnostic assessments from both the clinician and the AI. These were then combined in an AI-generated synthesis that integrated the two perspectives, highlighting points of agreement and disagreement and offering commentary on each. We evaluated two workflow variants: one where the AI provided an initial opinion (AI-first), and another where it followed the clinician’s assessment (AI-second). Clinicians using either collaborative workflow outperformed those using traditional tools, achieving average accuracies of 85% (AI-first) and 82% (AI-second), compared to 75% with traditional resources (p < 0.0004 and p < 0.00001; mean differences = 9.8% and 6.8%; 95% CI = 4.6%–15% and 4.0%–9.6%). Performance did not differ significantly between workflows or from the AI-alone score of 90%. These results underscore the value of collaborative AI systems that complement clinician expertise and foster effective coordination between human and machine reasoning in diagnostic decision-making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evidence indicates that the depletion mechanism operates conditionally rather than universally.
“Evidence that the depletion mechanism operates conditionally rather than universally (Everett et al., 2025;Hyman et al., 2025;Manning & Aguirre, 2026) reinforces the point”
From The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (2026) · cites this paper as Everett et al., 2025
The Discussion reports that AI engagement lowered scores on clinically actionable decisions in 8% of cases, while other passages report performance gains. This supports the narrower point that adverse performance effects were not universal, but the paper does not identify a depletion mechanism or establish when one operates. (LLM, conf=0.76)
Experimental — accuracy not yet measuredWorkflow designs that preserve developmental engagement, worker adaptive capacity, and heterogeneity in retrainability across occupations and labor market conditions all indicate that the depletion mechanism operates conditionally rather than universally.
“Workflow designs that preserve developmental engagement (Everett et al., 2025), worker adaptive capacity (Manning & Aguirre, 2026), and heterogeneity in retrainability across occupations and labor market conditions (Hyman et al., 2025) all indicate that the depletion mechanism operates conditionally rather than universally.”
From The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (2026) · cites this paper as Everett et al., 2025
The Methods (Two workflows; page unavailable) describe a design in which clinicians first reason independently, then review and discuss an AI second opinion. The Discussion (page unavailable) says the comparative display supports reflection and deeper diagnostic insight. This supports the workflow-engagement portion attributed to Everett et al., but the paper does not establish that the design preserves clinicians’ development over time. (LLM, conf=0.79)
Experimental — accuracy not yet measuredA trial of physician-AI collaboration found that workflows requiring active clinician engagement with AI reasoning, rather than passive acceptance of AI outputs, preserved and even improved diagnostic performance.
“Everett et al.'s (2025) trial of physician-AI collaboration found that structuring workflows to require active clinician engagement with AI reasoning, rather than passive acceptance of AI outputs, preserved and even improved diagnostic performance.”
From The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (2026) · cites this paper as Everett et al.'s (2025)
The Abstract and Discussion report that both collaborative workflows improved diagnostic accuracy compared with conventional resources, with no significant performance difference from AI alone. The Methods (Two workflows) describe clinician assessment and review of an AI-generated synthesis, but further dialogue was optional; the Results (Workflow influences physician-AI dialogue) report that some clinicians did not interact beyond required inputs. The trial therefore does not establish that requiring active engagement, rather than passive acceptance, caused the performance gains. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCollaborative AI workflows requiring active clinical engagement have improved diagnostic accuracy rather than degrading it.
“collaborative AI workflows requiring active clinical engagement have improved diagnostic accuracy rather than degrading it (Everett et al., 2025).”
From The Tragedy of the Cognitive Commons: How AI Could Disrupt the Regeneration of Professional Expertise (2026) · cites this paper as Everett et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724 · see citing works
Self-citationTowards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 132 · see citing works
Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Safavi-Naini, S. A. A., Soroush, A., & Chen, J. H. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae254 · Cited by 92 · see citing works
Self-citationWhen to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Self-citationWhen to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 16
- [1]Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on Medical Challenge Problems. 2023. doi: 10.48550/arXiv.2303.13375 · DOI · Paper page
- [2]Cabral S, Restrepo D, Kanjee Z, Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physicians. JAMA Intern Med. 84(5): 581– 583. 2024. doi: 10.1001/jamainternmed.2024.0295 PMC10985627 38557971 · DOI · Paper page
- [5]Horvitz E. Principles of mixed-initiative user interfaces. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI ‘99). Association for Computing Machinery, New York, NY, USA, 159– 166. 1999. doi: 10.1145/302979.303030 · DOI · Paper page
- [6]Amershi S, Weld D, Vorvoreanu M, Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19). Association for Computing Machinery, New York, NY, USA, Paper 3, 1– 13. 2019. doi: 10.1145/3290605.3300233 · DOI · Paper page
- [7]Wilder B, Horvitz E, Kamar E. Learning to complement humans. IJCAI’20: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 212: 1526 – 1533. 2020. doi: 10.24963/ijcai.2020/212 · DOI · Paper page
- [8]Fogliato R, Chappidi S, Lungren M, Who goes first? Influences of human-AI workflow on decision making in clinical imaging. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ‘22). Association for Computing Machinery, New York, NY, USA, 1362– 1374. 2022. doi: 10.1145/3531146.3533193 · DOI
- [11]Salecha A, Ireland ME, Subrahmanya S, Sedoc J, Ungar LH, Eichstaedt JC. Large language models display human-like social desirability biases in Big Five personality surveys. PNAS Nexus. 3(12): pgae533. 2024. doi: 10.1093/pnasnexus/pgae533. 39691446 PMC11650498 · DOI · Paper page
- [12]Sellen A, Horvitz E. The rise of the AI co-pilot: Lessons for design from aviation and beyond. Communications of the ACM. 67(7): 18– 23. 2024. doi: 10.1145/3637865 · DOI
- [13]Drosos I, Sarkar A, Toronto N. “ It makes you think”: Provocations Help Restore Critical Thinking to AI-Assisted Knowledge Work. ArXiv Prepr. Published online 2025. doi: 10.48550/arXiv.2501.17247 · DOI
- [14]Mozannar H, Satyanarayan A, Sontag D. Teaching Humans When to Defer to a Classifier via Exemplars. Proceedings of the AAAI Conference on Artificial Intelligence. 36, 5323– 5331. 2022. doi: 10.1609/aaai.v36i5.20469. · DOI · Paper page
- [16]Shaikh O, Mozannar H, Bansal G, Fourney A, Horvitz E. Navigating Rifts in Human-LLM Grounding: Study and Benchmark. ACL 2025: Proceedings of 63rd Annual meeting of the Association for Computational Linguistics. 2025. doi: 10.48550/arXiv.2503.13975 · DOI · Paper page
- [18]Bansal G, Nushi B, Kamar E, Lasecki WS, Weld DS, Horvitz E. Beyond accuracy: The role of mental models in human-AI team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1), 2– 11. 2019. doi: 10.1609/hcomp.v7i1.5285 · DOI · Paper page
- [19]Langlotz CP, Shortliffe EH. Adapting a consultation system to critique user plans. Int J Man-Mach Stud. 19(5): 479– 496. 1983. doi: 10.1016/S0020-7373(83)80067-4 · DOI
- [20]Miller PL. ATTENDING: Critiquing a physician’s management plan. IEEE Trans Pattern Anal Mach Intell. (5): 449– 461. 1983. doi: 10.1109/TPAMI.1983.4767424 21869130 · DOI
- [21]Bejnordi BE, Veta M, Van Diest PJ, Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 318(22): 2199– 2210. 2017. doi: 10.1001/jama.2017.14585 29234806 PMC5820737 · DOI · Paper page
- [23]Balachandran V, Chen J, Joshi N, Eureka: Evaluating and understanding large foundation models. ArXiv Prepr. Published online 2024. doi: 10.48550/arXiv.2409.10566 · DOI
Cited by
Sorted by citations
Lovett, N. (2026) · Human Resource Development Review · doi:10.1177/15344843261470602 · Cited by 7 · see citing works
- Authors
- Everett, S. S., Bunning, B. J., Jain, P., Lopez, I., Agarwal, A., Desai, M., Gallo, R., Goh, E., Kadiyala, V. B., Kanjee, Z., Koshy, J. M., Olson, A., Rodman, A., Schulman, K., Strong, E., Chen, J. H., & Horvitz, E.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Cambridge Health Alliance (US)
- VA Palo Alto Health Care System (US)
- Beth Israel Deaconess Medical Center (US)
- University of Minnesota (US)
- Microsoft (United States) (US)
- Corresponding author
- Horvitz, Eric
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2025-06-08
- PMC
- PMC12155023
- OpenAlex
- W4411128392
- Cited by
- 18
- References
- 9
- Citations by year
- 2026: 15 · 2025: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- Stanford Bio-X
- U.S. National Library of Medicine (T15LM007033)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, GPT, clinician-AI collaboration, diagnostic reasoning, diagnostic accuracy, clinical decision support, human-AI coordination, diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
