The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial
Open accessAbstract
While large language models (LLMs) have shown promise in diagnostic reasoning, their impact on management reasoning, which involves balancing treatment decisions and testing strategies while managing risk, is unknown. This prospective, randomized, controlled trial assessed whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources. From November 2023 to April 2024, 92 practicing physicians were randomized to use either GPT-4 plus conventional resources or conventional resources alone to answer five expert-developed clinical vignettes in a simulated setting. All cases were based on real, de-identified patient encounters, with information revealed sequentially to mirror the nature of clinical environments. The primary outcome was the difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case. Physicians using the LLM scored significantly higher compared to those using conventional resources (mean difference = 6.5%, 95% confidence interval (CI) = 2.7 to 10.2, P < 0.001). LLM users spent more time per case (mean difference = 119.3 s, 95% CI = 17.4 to 221.2, P = 0.02). There was no significant difference between LLM-augmented physicians and LLM alone (−0.9%, 95% CI = −9.0 to 7.2, P = 0.8). LLM assistance can improve physician management reasoning in complex clinical vignettes compared to conventional resources and should be validated in real clinical practice. ClinicalTrials.gov registration: NCT06208423. In a prospective study involving 92 physicians from multiple institutions, access to large language model assistance on top of conventional resources increased a score expressing the quality of their reasoning in addressing patient care while assessing five clinical vignettes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Qualitative Research: Consensus methods for medical and health services research
Jones, J., & Hunter, D. (1995) · BMJ · doi:10.1136/bmj.311.7001.376 · Cited by 3,471
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Educational Strategies to Promote Clinical Diagnostic Reasoning
Bowen, J. L. (2006) · New England Journal of Medicine · doi:10.1056/nejmra054782 · Cited by 931
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · cited 1×
“As an exploratory evaluation, a single blinded reviewer (A.R.) rated all responses for potential harm using a similar methodology to a previously published assessment 34 .”
Self-citationLarge Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724
Self-citationPrompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709
Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 281
Cited by
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209
A Multimodal Vision Foundation Model for Clinical Dermatology
Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., Tschandl, P., Hu, M., Ju, L., Tan, G., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03747-y · Cited by 117
“…to clinicians with PanDerm assistance in skin cancer diagnosis and even outperformed human–AI collaboration in differential diagnosis, similar to observations in a previous study 58 showing ‘no significant difference between large language model (LLM)-augmented physicians and LLM alone’.”
Automating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30
Implications of integrating large language models into clinical decision making
Christof, M., & Armoundas, A. A. (2025) · Communications Medicine · doi:10.1038/s43856-025-01216-8 · Cited by 16
Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians
Qazi, I. A., Ali, A., Khawaja, A. U., Akhtar, M. J., Sheikh, A. Z., & Alizai, M. H. (2025) · medRxiv · doi:10.1101/2025.08.23.25334280 · Cited by 16
Metadata
- Authors
- Goh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A.
- Venue
- Nature Medicine
- Published
- 2025-02-05
- Volume
- 31
- Issue
- 4
- Pages
- 1233-1238
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4407150885
- Cited by
- 167
- References
- 15
- FWCI
- 33.99
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. https://app.cochat.ai/papers/10.1038/s41591-024-03456-y
- The paper
- Goh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., Ahuja, N., Horvitz, E., Yang, D., Milstein, A., Olson, A. P. J., Hom, J., Chen, J. H., & Rodman, A. (2025). GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nature Medicine. https://doi.org/10.1038/s41591-024-03456-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.