CoChat
Sign Up
Cited by 378Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.

Towards accurate differential diagnosis with large language models

Open access
McDuff, Daniel, Schaekermann, Mike, Tu, Tao, Palepu, Anil, Wang, Amy, Garrison, Jake, Singhal, Karan, Sharma, Yash, Azizi, Shekoofeh, Kulkarni, Kavita, Hou, Le, Cheng, Yong, Liu, Yun, Mahdavi, S. Sara, Prakash, Sushant, Pathak, Anupam, Semturs, Christopher, Patel, Shwetak, Webster, Dale R., Dominowska, Ewa, Gottweis, Juraj, Barral, Joelle, Chou, Katherine, Corrado, Greg S., Matias, Yossi, Sunshine, Jake, Karthikesalingam, Alan, Natarajan, Vivek·2025·Nature
DOI 10.1038/s41586-025-08869-4PMC PMC12158753
Open this paper in CoChat — free account

Abstract

A comprehensive differential diagnosis is a cornerstone of medical care that is often reached through an iterative process of interpretation that combines clinical history, physical examination, investigations and procedures. Interactive interfaces powered by large language models present new opportunities to assist and automate aspects of this process 1 . Here we introduce the Articulate Medical Intelligence Explorer (AMIE), a large language model that is optimized for diagnostic reasoning, and evaluate its ability to generate a differential diagnosis alone or as an aid to clinicians. Twenty clinicians evaluated 302 challenging, real-world medical cases sourced from published case reports. Each case report was read by two clinicians, who were randomized to one of two assistive conditions: assistance from search engines and standard medical resources; or assistance from AMIE in addition to these tools. All clinicians provided a baseline, unassisted differential diagnosis prior to using the respective assistive tools. AMIE exhibited standalone performance that exceeded that of unassisted clinicians (top-10 accuracy 59.1% versus 33.6%, P = 0.04). Comparing the two assisted study arms, the differential diagnosis quality score was higher for clinicians assisted by AMIE (top-10 accuracy 51.7%) compared with clinicians without its assistance (36.1%; McNemar’s test: 45.7, P < 0.01) and clinicians with search (44.4%; McNemar’s test: 4.75, P = 0.03). Further, clinicians assisted by AMIE arrived at more comprehensive differential lists than those without assistance from AMIE. Our study suggests that AMIE has potential to improve clinicians’ diagnostic reasoning and accuracy in challenging cases, meriting further real-world evaluation for its ability to empower physicians and widen patients’ access to specialist-level expertise.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41586-025-08869-4

Cites

37 of 38 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Evaluation of the quality of DDx lists from generalist physicians. a, DDx quality score based on the question: “How close did the differential diagnoses (DDx) come to including the final diagnosis?” b, DDx comprehensiveness score based on the question: “Using your DDx list as a benchmark/gold standard, how comprehensive are the differential lists from each of the experts’?” c, DDx appropriateness score based on the question: “How appropriate was each of the DDx lists from the different medical experts compared to the differential list that you just produced?” The colours correspond to experiment arms, and the shade of the colour corresponds to different levels on the rating scales. In all cases, AMIE and clinicians assisted by AMIE scored highest overall. Numbers reflect the number of cases (out of 302). Note that the clinicians had the option of answering “I am not sure” in response to these questions; they used this option in a very small number (less than 1%) of cases.
Fig. 1 Evaluation of the quality of DDx lists from generalist physicians. a, DDx quality score based on the question: “How close did the differential diagnoses (DDx) come to including the final diagnosis?” b, DDx comprehensiveness score based on the question: “Using your DDx list as a benchmark/gold standard, how comprehensive are the differential lists from each of the experts’?” c, DDx appropriateness score based on the question: “How appropriate was each of the DDx lists from the different medical experts compared to the differential list that you just produced?” The colours correspond to experiment arms, and the shade of the colour corresponds to different levels on the rating scales. In all cases, AMIE and clinicians assisted by AMIE scored highest overall. Numbers reflect the number of cases (out of 302). Note that the clinicians had the option of answering “I am not sure” in response to these questions; they used this option in a very small number (less than 1%) of cases.CC BY · PubMed Central
Fig. 2 Top-n accuracy in DDx lists through human and automated evaluations. The percentage accuracy of DDx lists with the final diagnosis through human evaluation (left) or automated evaluation (right). Points reflect the mean; shaded areas show ±1 s.d. from the mean across 10 trials.
Fig. 2 Top-n accuracy in DDx lists through human and automated evaluations. The percentage accuracy of DDx lists with the final diagnosis through human evaluation (left) or automated evaluation (right). Points reflect the mean; shaded areas show ±1 s.d. from the mean across 10 trials.CC BY · PubMed Central
Fig. 3 Sankey diagram showing effect of assistance. a, In the AMIE arm, the final correct diagnosis appeared in the DDx list only after assistance in 73 cases. b, In the Search arm, the final correct diagnosis appeared in the DDx list only after assistance in 37 cases. In a small minority of cases in both arms (AMIE arm: 11 (a); Search arm: 12 (b)), the final diagnosis appeared in the DDx list before assistance but was not in the list after assistance.
Fig. 3 Sankey diagram showing effect of assistance. a, In the AMIE arm, the final correct diagnosis appeared in the DDx list only after assistance in 73 cases. b, In the Search arm, the final correct diagnosis appeared in the DDx list only after assistance in 37 cases. In a small minority of cases in both arms (AMIE arm: 11 (a); Search arm: 12 (b)), the final diagnosis appeared in the DDx list before assistance but was not in the list after assistance.CC BY · PubMed Central
Fig. 4 Top-n accuracy in DDx lists from different LLMs. Comparison of the percentage of DDx lists that included the final diagnosis for AMIE versus GPT-4 for 70 cases. We used Med-PaLM 210, GPT-46 and AMIE as the raters—all resulted in similar trends. Points reflect the mean; shaded areas show ±1 s.d. from the mean across 10 trials.
Fig. 4 Top-n accuracy in DDx lists from different LLMs. Comparison of the percentage of DDx lists that included the final diagnosis for AMIE versus GPT-4 for 70 cases. We used Med-PaLM 210, GPT-46 and AMIE as the raters—all resulted in similar trends. Points reflect the mean; shaded areas show ±1 s.d. from the mean across 10 trials.CC BY · PubMed Central
Extended Data Fig. 1 NEJM Clinicopathological Conference Case Reports. History of Present Illness, Admission Labs and Admission Imaging sections were included in the redacted version presented to generalist clinicians for producing a DDx. The LLM had access to only the History of Present Illness. Specialist clinicians evaluating the quality of the DDx had access to the full (unredacted) case report including the expert differential discussion.
Extended Data Fig. 1 NEJM Clinicopathological Conference Case Reports. History of Present Illness, Admission Labs and Admission Imaging sections were included in the redacted version presented to generalist clinicians for producing a DDx. The LLM had access to only the History of Present Illness. Specialist clinicians evaluating the quality of the DDx had access to the full (unredacted) case report including the expert differential discussion.CC BY · PubMed Central
Extended Data Fig. 2 The AMIE User Interface. The history of the present illness (text only) was pre-populated in the user interface (A) with an initial suggested prompt to query the LLM (B). Following this prompt and response, the user was free to enter any additional follow-up questions (C). The case shown in this figure is a mock case selected for illustrative purposes only.
Extended Data Fig. 2 The AMIE User Interface. The history of the present illness (text only) was pre-populated in the user interface (A) with an initial suggested prompt to query the LLM (B). Following this prompt and response, the user was free to enter any additional follow-up questions (C). The case shown in this figure is a mock case selected for illustrative purposes only.CC BY · PubMed Central
Extended Data Fig. 3 Experimental Design. To evaluate the LLM’s ability to generate DDx lists and aid clinicians with their DDx generation, we designed a two-stage reader study. First, clinicians with access only to the case presentation completed DDx lists without using any assistive tools. Second, the clinicians completed DDx lists with access either to Search engines and other resources (Condition I), or to LLM in addition to these tools (Condition II). Randomization was employed such that every case was reviewed by two different clinicians, one with LLM assistance and one without. In Condition II the clinician was given a suggested initial prompt to use in the LLM interface and was then free to try any other questions. These DDx lists were then evaluated by a specialist who had access to the full case and expert commentary on the differential diagnosis, but who was blinded to whether and what assistive tool was used.
Extended Data Fig. 3 Experimental Design. To evaluate the LLM’s ability to generate DDx lists and aid clinicians with their DDx generation, we designed a two-stage reader study. First, clinicians with access only to the case presentation completed DDx lists without using any assistive tools. Second, the clinicians completed DDx lists with access either to Search engines and other resources (Condition I), or to LLM in addition to these tools (Condition II). Randomization was employed such that every case was reviewed by two different clinicians, one with LLM assistance and one without. In Condition II the clinician was given a suggested initial prompt to use in the LLM interface and was then free to try any other questions. These DDx lists were then evaluated by a specialist who had access to the full case and expert commentary on the differential diagnosis, but who was blinded to whether and what assistive tool was used.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 2

  1. Hong, J., Levine, S. & Dragan, A. Zero-shot goal-directed dialogue via RL on imagined conversations. In Proc. Foundation Models for Decision Making Workshop at NeurIPS 2023 (eds Yang, S. et al.) 77 (NeurIPS, 2023).
  2. Umapathi, L. K., Pal, A. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. 27th Conference on Computational Natural Language Learning (CoNLL) (eds Jiang, J. et al.) 314–334 (Association for Computational Linguistics, 2023).

Cited by

Sorted by citations

Related

See all

Metadata

Authors
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., Hou, L., Cheng, Y., Liu, Y., Mahdavi, S. S., Prakash, S., Pathak, A., Semturs, C., Patel, S., Webster, D. R., Dominowska, E., Gottweis, J., Barral, J., Chou, K., Corrado, G. S., Matias, Y., Sunshine, J., Karthikesalingam, A., & Natarajan, V.
Affiliations
  • Google (United States) (US)
  • Google (Switzerland) (CH)
  • Google (United Kingdom) (GB)
Corresponding author
  • McDuff, Daniel
  • Schaekermann, Mike
  • Karthikesalingam, Alan
  • Natarajan, Vivek
ORCID
Venue
Nature
Publisher
Springer Science and Business Media LLC
Published
2025-04-09
Volume
642
Issue
8067
Pages
451-457
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
DOI
10.1038/s41586-025-08869-4
PMC
PMC12158753
OpenAlex
W4409283616
Cited by
311
References
34
Citations by year
2026: 172 · 2025: 108
FWCI
67.40
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Massachusetts Medical Society
  • DeepMind
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
differential diagnosis, large language models, diagnostic reasoning, clinical decision support, clinician performance, AI-assisted diagnosis, medical knowledge retrieval
MeSH terms
Case Reports as Topic, Large Language Models, Artificial Intelligence, Diagnosis, Differential, Humans, Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review