Use of gpt-4 to diagnose complex clinical cases
Open accessAbstract
We assessed the performance of the newly released AI GPT-4 in diagnosing complex medical case challenges and compared the success rate to that of medical-journal readers. GPT-4 correctly diagnosed 57% of cases, outperforming 99.98% of simulated human readers generated from online answers. We highlight the potential for AI to be a powerful supportive tool for diagnosis; however, further improvements, validation, and addressing of ethical considerations are needed before clinical implementation. (No funding was obtained for this study.).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have demonstrated medical ability through tasks such as medical licensing exams, answering clinical questions from patient records, and reasoning about clinical case studies.
“LLMs have demonstrated medical ability through tasks such as medical licensing exams (Nori et al., 2023), answering clinical questions from patient records (Pampari et al., 2018;Patrick and Li, 2012;Bai et al., 2024), and reasoning about clinical case studies (Eriksen et al., 2024;Chen et al., 2025).”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Eriksen et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Haug CJ, Drazen JM. Artificial intelligence and machine learning in clinical medicine, 2023. N Engl J Med 2023;388:1201-1208. DOI: 10.1056/NEJMra2302038. · DOI · Paper page
- Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health 2023;5:e107-e108. DOI: 10.1016/S2589- 7500(23)00021-3. · DOI · Paper page
- The New England Journal of Medicine. Case challenges (https:// www.nejm.org/case-challenges).
- Kanjee Z, Crowe B, Rodman A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA 2023; 330:78-80. DOI: https://doi.org/10.1001/jama.2023.8288. · DOI · Paper page
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
Monitoring performance of clinical artificial intelligence in health care: a scoping review
Andersen, E. S., Birk-Korch, J. B., Hansen, R. S., Fly, L. H., Röttger, R., Arcani, D. M. C., Brasen, C. L., Brandslund, I., & Madsen, J. S. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00042 · Cited by 73 · see citing works
Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/52758 · Cited by 50 · see citing works
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1504805 · Cited by 25 · see citing works
Multimodal Large Language Models are Generalist Medical Image Interpreters
Han, T., Adams, L. C., Nebelung, S., Kather, J. N., Bressem, K. K., & Truhn, D. (2023) · doi:10.1101/2023.12.21.23300146 · Cited by 18 · see citing works
Generative artificial intelligence performs rudimentary structural biology modeling
Ille, A. M., Markosian, C., Burley, S. K., Mathews, M. B., Pasqualini, R., & Arap, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-69021-2 · Cited by 16 · see citing works
Large Language Model Uncertainty Measurement and Calibration for Medical Diagnosis and Treatment
Savage, T., Wang, J., Gallo, R., Boukil, A., Patel, V., Ahmad Safavi-Naini, S. A., Soroush, A., & Chen, J. H. (2024) · doi:10.1101/2024.06.06.24308399 · Cited by 11 · see citing works
Harnessing Moravec’s paradox in health care: a new era of collaborative intelligence
Loaiza-Bonilla, A., & Penberthy, S. (2025) · NEJM AI · doi:10.1056/aip2500005 · Cited by 4 · see citing works
Huang, H., DeLucia, A., Tiyyala, V. M., & Dredze, M. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.693 · Cited by 2 · see citing works
“…have demonstrated medical ability through tasks such as medical licensing exams (Nori et al., 2023), answering clinical questions from patient records (Pampari et al., 2018; Patrick and Li, 2012; Bai et al., 2024), and reasoning about clinical case studies (Eriksen et al., 2024; Chen et al., 2025).”
Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?
Yuan, G. C., Zhang, X., Kim, S. E., & Rajpurkar, P. (2026) · Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026) · doi:10.18653/v1/2026.healing-1.1 · Cited by 1 · see citing works
“Clinical LLMs and Vendor Differences Single LLM systems have demonstrated impressive capabilities in medical reasoning, often matching or even exceeding average physician performance (Eriksen et al., 2024; Singhal et al., 2025; Son-oda et al., 2024).”
- Authors
- Eriksen, A. V., Möller, S., & Ryg, J.
- Affiliations
- University of Southern Denmark (DK)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2024-01
- Volume
- 1
- Issue
- 1
- Pages
- AIp2300031-AIp2300031
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4388525110
- Cited by
- 200
- References
- 1
- Citations by year
- 2026: 48 · 2025: 69 · 2024: 77 · 2023: 5
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- GPT-4, large language models, diagnosis, complex clinical cases, diagnostic accuracy, AI-assisted diagnosis, clinical implementation, ethical considerations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
