The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools
Open accessAbstract
Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5,213 case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Foundation models such as large language models are unable to achieve the diagnostic accuracy of traditional rare-disease decision support tools.
“Although foundation models, such as large language models, have been leveraged for clinical applications45,46, they are unable to achieve the diagnostic accuracy of traditional rare-disease decision support tools47.”
From Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases (2025) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
How many rare diseases are there?
Haendel, M., Vasilevsky, N., Unni, D., Bologa, C., Harris, N., Rehm, H., Hamosh, A., Baynam, G., Groza, T., McMurry, J., et al. (2020) · Nature Reviews Drug Discovery · doi:10.1038/d41573-019-00180-y · Cited by 554 · cited 1×
“12,13 Here, we assess the diagnostic ability of LLMs in a large set of cases of Mendelian or chromosomal diseases, leveraging a collection of computational case reports recorded using the Global Alliance for Genomics and Health (GA4GH) Phenopacket Schema.”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 31
- 2019, Information needs of physicians regarding the diagnosis of rare diseases: a questionnaire-based study in Belgium, Orphanet J. Rare Dis, 14, 99, 10.1186/s13023-019-1075-8 · DOI
- 10.1186/s13073-015-0199-2 · DOI
- 10.1016/j.ajhg.2009.09.003 · DOI · Paper page
- 10.1093/bioinformatics/bts471 · DOI
- Yuan, X. , Wang, J. , Dai, B. , Sun, Y. , Zhang, K. , Chen, F. , Peng, Q. , Huang, Y. , Zhang, X. , Chen, J. , et al. (2022). Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Brief. Bioinform. 23,. · DOI
- 2025, Leveraging clinical intuition to improve accuracy of phenotype-driven prioritization, Genet. Med, 27, 101292, 10.1016/j.gim.2024.101292 · DOI
- 10.1038/s41586-023-06291-2 · DOI · Paper page
- Kanjee, Z. , Crowe, B. , and Rodman, A. (2023). Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA. · DOI · Paper page
- 2024, Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models, JMIR Med Educ, 10, e51391, 10.2196/51391 · DOI · Paper page
- 2024, Evaluation of large language models as a diagnostic aid for complex medical cases, Front. Med, 11, 1380148, 10.3389/fmed.2024.1380148 · DOI · Paper page
- 10.2196/53724 · DOI
- 10.1148/radiol.231040 · DOI
- Milad, D. , Antaki, F. , Milad, J. , Farah, A. , Khairy, T. , Mikhail, D. , Giguère, C.-É. , Touma, S. , Bernstein, A. , Szigiato, A.-A. , et al. (2024). Assessing the medical reasoning skills of GPT-4 in complex ophthalmology cases. Br. J. Ophthalmol. · DOI
- 2021, Case 2-2021: A 26-Year-Old Pregnant Woman with Ventricular Tachycardia and Shock, N. Engl. J. Med, 384, 272, 10.1056/NEJMcpc2027086 · DOI
- Reese, J.T. , Danis, D. , Caufield, J.H. , Groza, T. , Casiraghi, E. , Valentini, G. , Mungall, C.J. , and Robinson, P.N. (2024). On the limitations of large language models in clinical diagnosis. medRxiv. · DOI
- 2021, Case 16-2021: A 37-Year-Old Woman with Abdominal Pain and Aortic Dilatation, N. Engl. J. Med, 384, 2054, 10.1056/NEJMcpc2100278 · DOI
- 10.1016/j.ajhg.2024.08.010 · DOI · Paper page
- 2024, Evaluating large language models on medical, lay-language, and self-reported descriptions of genetic conditions, Am. J. Hum. Genet, 111, 1819, 10.1016/j.ajhg.2024.07.011 · DOI · Paper page
- Yang, J. , Shu, L. , Duan, H. , and Li, H. (2024). RDguru: A conversational intelligent agent for rare diseases. IEEE J. Biomed. Health Inform. PP,. · DOI
- 10.1038/s41587-022-01357-4 · DOI
- 10.1093/nar/gku1205 · DOI · Paper page
- Danis, D. , Bamshad, M.J. , Bridges, Y. , Cacheiro, P. , Carmody, L.C. , Chong, J.X. , Coleman, B. , Dalgleish, R. , Freeman, P.J. , Graefe, A.S.L. , et al. (2024). A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discovery. · DOI
- 2024, Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study, JMIR Form Res, 8, e53985, 10.2196/53985 · DOI
- 2025, A generalist medical language model for disease diagnosis assistance, Nat. Med, 31, 932, 10.1038/s41591-024-03416-6 · DOI · Paper page
- 10.1056/NEJMoa2035790 · DOI · Paper page
- 10.1093/nar/gkz997 · DOI
- Bridges, Y.S. , de Souza, V. , Cortes, K.G. , Haendel, M. , Harris, N.L. , Korn, D.R. , Marinakis, N.M. , Matentzoglu, N. , McLaughlin, J.A. , Mungall, C.J. , et al. (2024). Towards a standard benchmark for variant and gene prioritisation algorithms: PhEval - Phenotypic inference Evaluation framework. · DOI
- 2024, OpenAI o1-Preview vs. ChatGPT in healthcare: A new frontier in medical AI reasoning, Cureus, 16, e70640
- 10.1038/s41598-024-58760-x · DOI
- 10.1002/humu.24380 · DOI
- 2023, Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation, J. Med. Internet Res, 25, e48763, 10.2196/48763 · DOI
Cited by
Sorted by citations
Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases
Alsentzer, E., Li, M. M., Kobren, S. N., Noori, A., Undiagnosed Diseases Network, Kohane, I. S., & Zitnik, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01749-1 · Cited by 54
Metadata
- Authors
- Reese, J. T., Chimirri, L., Bridges, Y., Danis, D., Caufield, J. H., Gargano, M. A., Kroll, C., Schmeder, A., Liu, F., Wissink, K., McMurry, J. A., Graefe, A. S., Niyonkuru, E., Korn, D. R., Casiraghi, E., Valentini, G., Jacobsen, J. O., Haendel, M., Smedley, D., Mungall, C. J., & Robinson, P. N.
- Affiliations
- Lawrence Berkeley National Laboratory (US)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Queen Mary University of London (GB)
- University of North Carolina at Chapel Hill (US)
- University of Milan (IT)
- Corresponding author
- Robinson, Peter N
- ORCID
- Reese, Justin T
- Chimirri, Leonardo
- Bridges, Yasemin
- Danis, Daniel
- Caufield, J Harry
- Gargano, Michael A.
- Schmeder, Andrew
- Wissink, Kyran
- McMurry, Julie A
- Graefe, Adam SL
- Niyonkuru, Enock
- Korn, Daniel R
- Casiraghi, Elena
- Valentini, Giorgio
- Jacobsen, Julius OB
- Haendel, Melissa
- Smedley, Damian
- Mungall, Christopher J
- Robinson, Peter N
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-07-22
- PMC
- PMC11302616
- OpenAlex
- W4400898355
- Cited by
- 8
- References
- 3
- Citations by year
- 2026: 3 · 2025: 5
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- U.S. Department of Energy (-AC02-05CH11231, 05CH11231, DE AC02 05CH11231, DE-AC02-)
- Alexander von Humboldt-Stiftung (DE-AC02-05CH11231)
- National Institutes of Health (R24OD011883, DE-AC02–05CH11231, U24TR002306)
- Office of Science (DEAC02–05CH11231, AC02-05CH11231, DE-AC02)
- National Human Genome Research Institute (RM1HG010860)
- National Institute of Child Health and Human Development
- Basic Energy Sciences (DE-AC02, DE-AC02-05CH11231., AC02–05CH11231)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD103805)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- large language models, rare diseases, genetic disease diagnosis, differential diagnosis, diagnostic accuracy, Human Phenotype Ontology, Mondo Disease Ontology, Exomiser, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
