Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Purpose: Phenotyping is critical for informing rare disease diagnosis and treatment, but disease phenotypes are often embedded in unstructured text. While natural language processing (NLP) can automate extraction, a major bottleneck is developing annotated corpora. Recently, prompt learning with large language models (LLMs) has been shown to lead to generalizable results without any (zero-shot) or few annotated samples (few-shot), but none have explored this for rare diseases. Our work is the first to study prompt learning for identifying and extracting rare disease phenotypes in the zero- and few-shot settings. Methods: We compared the performance of prompt learning with ChatGPT and fine-tuning with BioClinicalBERT. We engineered novel prompts for ChatGPT to identify and extract rare diseases and their phenotypes (e.g., diseases, symptoms, and signs), established a benchmark for evaluating its performance, and conducted an in-depth error analysis. Results: Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.610 in the zero- and few-shot settings, respectively). However, ChatGPT achieved higher accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725). Conversational, sentence-based prompts generally achieved higher accuracy than structured lists. Conclusion: Prompt learning using ChatGPT has the potential to match or outperform fine-tuning BioClinicalBERT at extracting rare diseases and signs with just one annotated sample. Given its accessibility, ChatGPT could be leveraged to extract these entities without relying on a large, annotated corpus. While LLMs can support rare disease phenotyping, researchers should critically evaluate model outputs to ensure phenotyping accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database
Nguengang Wakap, S., Lambert, D. M., Olry, A., Rodwell, C., Gueydan, C., Lanneau, V., Murphy, D., Le Cam, Y., & Rath, A. (2020) · European Journal of Human Genetics · doi:10.1038/s41431-019-0508-0 · Cited by 1,752
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 6×
“Recently, Fabregat et al. (2018) and Segura-Bedmar et al. (2022) leveraged deep learning techniques, including Bidirectional Long Short Term Memory networks and BERT-based models, to recognize rare diseases and their clinical manifestations from texts.”
BackgroundMethodInfluentialClinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · cited 3×
“Agrawal et al. (2022) and Hu et al. (2023) used a similar approach to extract medications and clinical entities, respectively.”
BackgroundMethodInfluentialTemplate-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.01760 · Cited by 342 · cited 1×
“Last, though manually created prompts are intuitive and interpretable, evidence suggests that small changes can lead to variations in performance (Cui et al., 2021).”
BackgroundImproving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314
Self-citationChen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56989-2 · Cited by 219
Self-citationChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60 · cited 2×
“While some explored the potential of ChatGPT for diagnosing rare diseases with human-provided suggestions (Lee et al., 2023; Mehnen et al., 2023), none have studied its performance for NER in the zero- or few-shot settings.”
Background
Cited by
Unleashing the potential of prompt engineering for large language models
Chen, B., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 490
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection
Peng, X., Zhou, Y., He, B., Sun, L., & Sun, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.644 · Cited by 16
Chase, A., Most, A., Sikora, A., Smith, S. E., Devlin, J. W., Xu, S., Liu, T., & Murray, B. (2026) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf168 · Cited by 11
LongHealth: A Question Answering Benchmark with Long Clinical Documents
Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, J. N., Truhn, D., & Bressem, K. (2024) · arXiv · doi:10.48550/arxiv.2401.14490 · Cited by 10
Metadata
- Authors
- Shyr, C., Hu, Y., Bastarache, L., Cheng, A., Hamid, R., Harris, P., & Xu, H.
- Venue
- Journal of Healthcare Informatics Research
- Published
- 2024-01-05
- Volume
- 8
- Issue
- 2
- Pages
- 438-461
- ISSN
- 2509-4971 (ISSN-L)
- 2509-498X
- OpenAlex
- W4390607929
- Cited by
- 58
- References
- 9
- FWCI
- 6.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models. https://app.cochat.ai/papers/10.1007/s41666-023-00155-0
- The paper
- Shyr, C., Hu, Y., Bastarache, L., Cheng, A., Hamid, R., Harris, P., & Xu, H. (2024). Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models. Journal of Healthcare Informatics Research. https://doi.org/10.1007/s41666-023-00155-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.