RareBench: Can LLMs Serve as Rare Diseases Specialists?
Open accessAbstract
Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rare disease evaluations probe unusual content, whereas mARC-QA probes unusual reasoning.
“Rare disease evaluations probe unusual content68, whereas mARC QA probes unusual reasoning.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [68]
Cites
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791 · see citing works
node2vec: Scalable Feature Learning for Networks
Grover, A., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939754 · Cited by 10,926 · see citing works
DeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
The Human Phenotype Ontology in 2021
Köhler, S., Gargano, M. A., Matentzoglu, N., Carmody, L., Lewis‐Smith, D., Vasilevsky, N., Daniš, D., Balagura, G., Baynam, G. S., Brower, A., et al. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa1043 · Cited by 1,149 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
The Human Phenotype Ontology in 2017
Köhler, S., Vasilevsky, N., Engelstad, M., Foster, E. D., McMurry, J. A., Aymé, S., Baynam, G. S., Bello, S. M., Boerkoel, C. F., Boycott, K. M., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw1039 · Cited by 757 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Clinical Diagnostics in Human Genetics with Semantic Similarity Searches in Ontologies
Köhler, S., Schulz, M. H., Krawitz, P., Bauer, S., Dölken, S., Ott, C. E., Mundlos, C., Horn, D., Mundlos, S., & Robinson, P. N. (2009) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2009.09.003 · Cited by 590 · see citing works
How many rare diseases are there?
Haendel, M., Vasilevsky, N., Unni, D., Bologa, C., Harris, N., Rehm, H., Hamosh, A., Baynam, G., Groza, T., McMurry, J., et al. (2020) · Nature Reviews Drug Discovery · doi:10.1038/d41573-019-00180-y · Cited by 554 · see citing works
Ethics of large language models in medicine and medical research
Li, H., Moon, J. T., Purkayastha, S., Celi, L. A., Trivedi, H., & Gichoya, J. W. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00083-3 · Cited by 286 · see citing works
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.348 · Cited by 269 · see citing works
Deep Phenotyping on Electronic Health Records Facilitates Genetic Diagnosis by Clinical Exomes
Son, J. H., Xie, G., Yuan, C., Ena, L., Li, Z., Goldstein, A., Huang, L., Wang, L., Shen, F., Liu, H., et al. (2018) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2018.05.010 · Cited by 133 · see citing works
Self-citationInterpretable Clinical Genomics with a Likelihood Ratio Paradigm
Robinson, P. N., Ravanmehr, V., Jacobsen, J. O. B., Daniš, D., Zhang, X., Carmody, L., Gargano, M. A., Thaxton, C., Karlebach, G., Reese, J., et al. (2020) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2020.06.021 · Cited by 110 · see citing works
Li, Q., Zhao, K., Bustamante, C. D., Ma, X., & Wong, W. H. (2019) · Genetics in Medicine · doi:10.1038/s41436-019-0439-8 · Cited by 106 · see citing works
Deisseroth, C. A., Birgmeier, J. A., Bodle, E. E., Kohler, J. N., Matalon, D. R., Nazarenko, Y., Genetti, C. A., Brown, D. M., Schmitz‐Abe, K., Schoch, K., et al. (2018) · Genetics in Medicine · doi:10.1038/s41436-018-0381-1 · Cited by 105 · see citing works
Phen2Gene: rapid phenotype-driven gene prioritization for rare diseases
Zhao, M., Havrilla, J. M., Li, F., Chen, Y., Peng, J. J., Liu, C., Wu, C., Sarmady, M., Botas, P., Isla, J., et al. (2020) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqaa032 · Cited by 87 · see citing works
PhenoTagger: a hybrid method for phenotype concept recognition using human phenotype ontology
Luo, L., Yan, S., Lai, P., Veltri, D., Oler, A. J., Xirasagar, S., Ghosh, R., Similuk, M. N., Robinson, P. N., & Lu, Z. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab019 · Cited by 80 · see citing works
Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders
Vaid, A., Landi, I., Nadkarni, G., & Nabeel, I. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00202-9 · Cited by 45 · see citing works
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42 · see citing works
Self-citationPhenoBERT: A Combined Deep Learning Method for Automated Recognition of Human Phenotype Ontology
Feng, Y., Qi, L., & Tian, W. (2023) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3170301 · Cited by 38 · see citing works
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 29 · see citing works
Rare Disease Discovery: An Optimized Disease Ranking System
Piñol, M., Alves, R., Teixidó, I., Mateo, J., Solsona, F., & Vilaprinyó, E. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2686380 · Cited by 16 · see citing works
Measuring phenotype semantic similarity using Human Phenotype Ontology
Peng, J., Xue, H., Shao, Y., et al. (2016) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822617 · Cited by 15 · see citing works
DDXPlus: A New Dataset For Automatic Medical DiagnosisPreprint
Tchango, A. F., Goel, R., Wen, Z., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09148 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?
Yuan, G. C., Zhang, X., Kim, S. E., & Rajpurkar, P. (2026) · Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026) · doi:10.18653/v1/2026.healing-1.1 · Cited by 1 · see citing works
“To verify that the observed advantages of the mixed-vendor configuration are not artifacts of a specific supervisor model, we conduct supervisor-vendor ablations on the MME dataset, the most challenging subset of RareBench while remaining compact in size.”
Influential
- Authors
- Chen, X., Mao, X., Guo, Q., Wang, L., Zhang, S., & Chen, T.
- Affiliations
- Tsinghua University (CN)
- Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
- Venue
- Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2024-08-24
- Pages
- 4850-4861
- OpenAlex
- W4401863414
- Cited by
- 41
- References
- 28
- Citations by year
- 2026: 16 · 2025: 20 · 2024: 5
- FWCI
- 11.04
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2021YFF1201303, 2022YFC2703103)
- Guoqiang Institute, Tsinghua University (2020GQG0003)
- Beijing National Research Center For Information Science And Technology
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- rare diseases, large language models, differential diagnosis, benchmark, few-shot prompting, GPT-4, diagnosis, physician comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
