Rare disease diagnosis using knowledge guided retrieval augmentation for ChatGPT
Open accessAbstract
Although rare diseases individually have a low prevalence, they collectively affect nearly 400 million individuals around the world. On average, it takes five years for an accurate rare disease diagnosis, but many patients remain undiagnosed or misdiagnosed. As machine learning technologies have been used to aid diagnostics in the past, this study aims to test ChatGPT's suitability for rare disease diagnostic support with the enhancement provided by Retrieval Augmented Generation (RAG). RareDxGPT, our enhanced ChatGPT model, supplies ChatGPT with information about 717 rare diseases from an external knowledge resource, the RareDis Corpus, through RAG. In RareDxGPT, when a query is entered, the three documents most relevant to the query in the RareDis Corpus are retrieved. Along with the query, they are returned to ChatGPT to provide a diagnosis. Additionally, phenotypes for thirty different diseases were extracted from free text from PubMed's Case Reports. They were each entered with three different prompt types: "prompt", "prompt + explanation" and "prompt + role play." The accuracy of ChatGPT and RareDxGPT with each prompt was then measured. With "Prompt", RareDxGPT had a 40 % accuracy, while ChatGPT 3.5 got 37 % of the cases correct. With "Prompt + Explanation", RareDxGPT had a 43 % accuracy, while ChatGPT 3.5 got 23 % of the cases correct. With "Prompt + Role Play", RareDxGPT had a 40 % accuracy, while ChatGPT 3.5 got 23 % of the cases correct. To conclude, ChatGPT, especially when supplying extra domain specific knowledge, demonstrates early potential for rare disease diagnosis with adjustments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In [16], retrieval augmentation enhanced diagnostic accuracy from 37% to 40% by grounding model predictions in verified biomedical literature.
“In [16] retrieval augmentation enhanced the diagnostic accuracy from 37% to 40% by grounding model predictions in verified biomedical literature, which is not provided by classical rare-disease tools and standalone LLMs.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [16]
The Results (page unavailable) report 40% accuracy for RareDxGPT versus 37% for standalone ChatGPT 3.5 with the basic prompt. The Introduction and Methods say RareDxGPT retrieved documents from the RareDis Corpus, but do not establish that the retrieved material was verified biomedical literature. Verification described in Methods concerns the diagnoses in the test cases, not the retrieval corpus. (LLM, conf=0.85)
Experimental — accuracy not yet measuredRetrieval-based long-tail diagnostic support outperformed Clinical BERT on rare classes.
“and retrievalbased long-tail diagnostic support outperformed Clinical BERT on rare classes [16].”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [16]
The Abstract (page not provided) reports that the retrieval-augmented RareDxGPT achieved higher diagnostic accuracy than ChatGPT 3.5 on cases involving rare diseases. It does not report a comparison with Clinical BERT or results stratified by rare classes, so the specific performance claim is unsupported. (LLM, conf=0.88)
Experimental — accuracy not yet measuredTransparency in multi-agent and RAG-based systems is enhanced mainly by output associated with evidence and auditing intermediate reasoning.
“Transparency was enhanced mainly thanks to output associated with evidence and auditing intermediate reasoning, components generally observed in multi-agent and RAGbased systems.[16]”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [16]
Material and Methods (page not supplied) says RareDxGPT identifies the sources and specific document used in its output. The Discussion (page not supplied) examines an explanation to understand an incorrect diagnosis. The paper does not establish that these features mainly enhance transparency, nor does it study multi-agent systems. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRare disease diagnosis improved from 37% to 40-43% with retrieval augmentation.
“Rare disease diagnosis improved from 37% to 40-43% with retrieval augmentation [16]”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [16]
The Results report that retrieval-augmented RareDxGPT achieved 40–43% accuracy, versus 37% for unaugmented ChatGPT 3.5 with the basic prompt. The 43% result used a different prompt (“Prompt + Explanation”), for which unaugmented ChatGPT scored 23%; the claim therefore combines results from different prompt conditions as though they were one comparison. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSeveral studies flagged biases in training data, including that the rare-disease dataset in [16] contained only 30 conditions, leading to representativeness and fairness issues.
“Several studies flagged biases in training data, including in [16] the rare-disease dataset contained only 30 conditions, leading to issues of representativeness and fairness”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [16]
The Discussion says the study tested thirty diseases with one sample per disease and identifies the small sample size as a limitation. But thirty describes the evaluation cases, not a training dataset: the Discussion says the external knowledge resource covered 717 diseases and raises a generalizability concern. It does not attribute fairness issues or training-data bias to the thirty-case sample. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
How many rare diseases are there?
Haendel, M., Vasilevsky, N., Unni, D., Bologa, C., Harris, N., Rehm, H., Hamosh, A., Baynam, G., Groza, T., McMurry, J., et al. (2020) · Nature Reviews Drug Discovery · doi:10.1038/d41573-019-00180-y · Cited by 554 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- [2]Bick D, Bick SL, Dimmock DP, Fowler TA, Caulfield MJ, Scott RH, An online compendium of treatable genetic disorders, Am. J. Med. Genet. C Semin. Med. Genet 187 (2021) 48– 54. 33350578 PMC7986124
- [3]Wilson ML, Atun R, DeStigter K, Flanigan J, Fleming KA, Horton S, Kleinert S, Sayed S, Commissioners of The Lancet Commission on diagnostics, The Lancet Commission on diagnostics: advancing equitable access to diagnostics, Lancet 393 (2019) 2018– 2020. 31106735
- [4]Marwaha S, Knowles JW, Ashley EA, A guide for the diagnosis of rare and undiagnosed disease: beyond the exome, Genome Med. 14 (2022) 23 35220969 PMC8883622
- [5]Walkowiak D, Domaradzki J, Are rare diseases overlooked by medical education? Awareness of rare diseases among physicians in Poland: an explanatory study, Orphanet J. Rare Dis. 16 (2021) 400. 34583737 PMC8479904
- [6]Kopeć G, Podolec P, Establishing a curriculum on rare diseases for medical students, J. Rare Cardiovasc. Dis 2 (2015). 10.20418/jrcd.vol2no3.194. · DOI
- [7]Domaradzki J, Walkowiak D, Medical students’ knowledge and opinions about rare diseases: A case study from Poland, Intractable Rare Dis Res 8 (2019) 252– 259. 31890452 PMC6929592
- [8]Bauskis A, Strange C, Molster C, Fisher C, The diagnostic odyssey: insights from parents of children living with an undiagnosed condition, Orphanet J. Rare Dis 17 (2022) 233. 35717227 PMC9206122
- [10]Graungaard AH, Skov L, Why do we need a diagnosis? A qualitative study of parents’ experiences, coping and needs, when the newborn child is severely disabled, Child Care Health Dev. 33 (2007) 296– 307. 17439444
- [11]Visibelli A, Roncaglia B, Spiga O, Santucci A, The Impact of Artificial Intelligence in the Odyssey of Rare Diseases, Biomedicines 11 (2023). 10.3390/biomedicines11030887. PMC10045927 36979866 · DOI
- [12]Blank IA, What are large language models supposed to model?, Trends Cogn. Sci 27 (2023) 987– 989. 37659920
- [13]Biswas S, Logan NS, Davies LN, Sheppard AL, Wolffsohn JS, Assessing the utility of ChatGPT as an artificial intelligence-based large language model for information to answer questions on myopia, Ophthalmic Physiol. Opt 43 (2023) 1562– 1570. 37476960
- [14]Mehnen L, Gruarin S, Vasileva M, Knapp B, ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases, bioRxiv (2023). 10.1101/2023.04.20.23288859. · DOI · Paper page
- [15]Hu X, Ran AR, Nguyen TX, Szeto S, Yam JC, Chan CKM, Cheung CY, What can GPT-4 do for Diagnosing Rare Eye Diseases? A Pilot Study, Ophthalmol Ther 12 (2023) 3395– 3402. 37656399 PMC10640532
- [16]See A, Manning C, Understanding and predicting user dissatisfaction in a neural generative chatbot, in: Li H, Levow G-A, Yu Z, Gupta C, Sisman B, Cai S, Vandyke D, Dethlefs N, Wu Y, Li JJ (Eds.), Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Singapore and Online, (2021)
- [17]Yao J, Xu W, Lian J, Wang X, Yi X, Xie X, Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations, arXiv [cs.IR] (2023). http://arxiv.org/abs/2311.10779. · arXiv
- [18]Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T, Riedel S, Kiela D, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv [cs.CL] (2020). http://arxiv.org/abs/2005.11401. · arXiv
- [19]Ge J, Sun S, Owens J, Galvez V, Gologorskaya O, Lai JC, Pletcher MJ, Lai K, Development of a liver disease-specific large language model chat interface using retrieval-augmented generation, Hepatology (2024). 10.1097/HEP.0000000000000834. PMC11706764 38451962 · DOI
- [20]Ke YH, Jin L, Elangovan K, Abdullah H, Liu N, Sia ATH, Soh CR, Tung JYM, Ong JCL, Ting DSW, Development and Testing of Retrieval Augmented Generation in Large Language Models, (2024). 10.2139/ssrn.4719185. · DOI
- [21]Jin Q, Leaman R, Lu Z, PubMed and beyond: biomedical literature search in the age of artificial intelligence, EBioMedicine 100 (2024) 104988. 38306900 PMC10850402
- [22]Xiong G, Jin Q, Lu Z, Zhang A, Benchmarking Retrieval-Augmented Generation for Medicine, arXiv [cs.CL] (2024). http://arxiv.org/abs/2402.13178. · arXiv
- [23]Martínez-deMiguel C, Segura-Bedmar I, Chacón-Solano E, Guerrero-Aspizua S, The RareDis corpus: A corpus annotated with rare diseases, their signs and symptoms, J. Biomed. Inform 125 (2022) 103961. 34879250
Cited by
Sorted by citations
Liu, S., McCoy, A. B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf008 · Cited by 200 · see citing works
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81 · see citing works
Self-citationGenerative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Mugambiwa, T., & Ndlovu, B. (2026) · Journal of Applied Informatics and Computing · doi:10.30871/jaic.v10i1.11900 · Cited by 4 · see citing works
“In [16] retrieval augmentation enhanced the diagnostic accuracy from 37% to 40% by grounding model predictions in verified biomedical literature, which is not provided by classical rare-disease tools and standalone LLMs. [20] CliniqIR showed that retrieval-based decision support outpaced Clinical…”
Influential
- Authors
- Zelin, C., Chung, W. K., Jeanne, M., Zhang, G., & Weng, C.
- Affiliations
- Ryogoku High School (JP)
- Boston Children's Hospital (US)
- Columbia University (US)
- Corresponding author
- Weng, Chunhua
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2024-09
- Volume
- 157
- Pages
- 104702
- ISSN
- 1532-0464 (ISSN-L)
- PMC
- PMC11402564
- OpenAlex
- W4401100256
- Cited by
- 47
- References
- 1
- Citations by year
- 2026: 24 · 2025: 23
- FWCI
- 4.39
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 365 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NIH
- Department of Biomedical Engineering, McKelvey School of Engineering
- Sarepta Therapeutics
- Nihon Kohden America (R01LM012895, R01HG013031)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- rare diseases, rare disease diagnosis, retrieval-augmented generation, ChatGPT, prompt engineering, phenotype extraction, diagnostic support, large language models
- MeSH terms
- Machine Learning, Algorithms, Artificial Intelligence, Diagnosis, Computer-Assisted, Humans, Databases, Factual, Information Storage and Retrieval, Decision Support Systems, Clinical, Rare Diseases, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
