CoChat
Sign Up
Cited by 74Artificial Intelligence in Healthcare and EducationTopic ModelingMachine Learning in Healthcare

Medical foundation large language models for comprehensive text analysis and beyond

Open access
Xie, Qianqian, Chen, Qingyu, Chen, Aokun, Peng, Cheng, Hu, Yan, Lin, Fongci, Peng, Xueqing, Huang, Jimin +10 more·2025·npj Digital Medicine
DOI 10.1038/s41746-025-01533-1PMC PMC11882967

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Recent advancements in large language models (LLMs) like ChatGPT and LLaMA have shown significant potential in medical applications, but their effectiveness is limited by a lack of specialized medical knowledge due to general-domain training. In this study, we developed Me-LLaMA, a new family of open-source medical LLMs that uniquely integrate extensive domain-specific knowledge with robust instruction-following capabilities. Me-LLaMA comprises foundation models (Me-LLaMA 13B and 70B) and their chat-enhanced versions, developed through comprehensive continual pretraining and instruction tuning of LLaMA2 models using both biomedical literature and clinical notes. Me-LLaMA utilized the largest and most comprehensive medical data, including 129B pre-training tokens and 214K instruction tuning samples from diverse biomedical and clinical data sources. Training the 70B models required substantial computational resources, exceeding 100,000 A100 GPU hours. We applied Me-LLaMA to six medical text analysis tasks and evaluated its performance on 12 benchmark datasets. To further assess Me-LLaMA's potential clinical utility, we evaluated its performance on complex clinical case diagnosis compared with other commercial LLMs, using both automatic and human evaluations. Me-LLaMA models outperform LLaMA, and other existing open-source medical LLMs in both zero-shot and supervised learning settings for most text analysis tasks. With task-specific instruction tuning, Me-LLaMA models also surpass leading commercial LLMs, outperforming ChatGPT on 7 out of 8 datasets and GPT-4 on 5 out of 8 datasets. Moreover, Me-LLaMA's performance is comparable to ChatGPT and GPT-4 for diagnosing complex clinical cases. Our findings underscore combining domain-specific continual pretraining with instruction tuning is essential for developing effective domain-specific large language models in healthcare, significantly enhancing performance across diverse medical text analysis tasks and applications. By publicly releasing our models and resources under appropriate user agreements, we aim to foster innovation and facilitate advancements in medical AI, benefiting researchers and practitioners within the community.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01533-1

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models.

    “Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”

    From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [52]

    The Results section (“Overall performance of medical text analysis”; page unavailable) compares zero-shot and task-specifically fine-tuned Me-LLaMA models and reports stronger results for the fine-tuned models against ChatGPT and GPT-4. The Methods section (“Evaluation settings”; page unavailable) identifies zero-shot and supervised learning as the two settings evaluated. Our copy does not report a few-shot comparison, so it does not support that part of the claim. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. [1]Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on Medical Challenge Problems. Preprint at 10.48550/arXiv.2303.13375 (2023). · DOI · Paper page
  2. [2]Wu, C. et al. PMC-LLaMA: toward building open-source language models for medicine. J. Am. Med Inf. Assoc. 31, 1833–1843 (2024). PMC11639126 38613821 · DOI · Paper page
  3. [3]Han, T. et al. MedAlpaca - An Open-Source Collection of Medical Conversational AI Models and Training Data. Preprint at 10.48550/arXiv.2304.08247 (2023). · DOI
  4. [4]Achiam, O. J. et al. GPT-4 Technical Report. Preprint at 10.48550/arXiv.2303.08774 (2023). · DOI · Paper page
  5. [5]Touvron, H. et al. LLaMA: Open and Efficient Foundation Language Models. Preprint at 10.48550/arXiv.2302.13971 (2023). · DOI · Paper page
  6. [6]Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. Preprint at 10.48550/arXiv.2307.09288 (2023). · DOI · Paper page
  7. [7]Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2022). PMC10396962 37438534 · DOI · Paper page
  8. [8]Peng, C. A. I. et al. A study of generative large language model for medical research and healthcare. NPJ Digital Medicine. 6 (2023). PMC10654385 37973919
  9. [9]Chen, Z. et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Preprint at 10.48550/arXiv.2311.16079 (2023). · DOI · Paper page
  10. [10]Li, Y. et al. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. Cureus. 15 (2023). PMC10364849 37492832
  11. [11]Zhang, X. et al. AlpaCare: Instruction-tuned Large Language Models for Medical Application. Preprint at 10.48550/arXiv.2310.14558 (2023). · DOI
  12. [12]Gema, A., Minervini, P., Daines, L., Hope, T. & Alex, B. Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain. In Proceedings of the 6th Clinical Natural Language Processing Workshop. 91-104. Mexico City, Mexico (Association for Computational Linguistics, 2024).
  13. [13]Lin, C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. 74-81. Barcelona, Spain (Annual Meeting of the Association for Computational Linguistics, 2004).
  14. [15]Hu, Y. et al. Improving large language models for clinical named entity recognition via prompt engineering. J. Am. Med. Inform. Assoc. JAMIA 31, 1812–1820 (2023). PMC11339492 38281112 · DOI · Paper page
  15. [17]Stiennon, N. et al. Learning to summarize with human feedback. Advances in neural information processing systems. (2020).
  16. [18]Chen, Y. et al. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. Preprint at 10.48550/arXiv.2309.12307 (2023). · DOI · Paper page
  17. [19]Gao, L. et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. Preprint at 10.48550/arXiv.2101.00027 (2020). · DOI · Paper page
  18. [20]Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data. 3 (2016). PMC4878278 27219127
  19. [21]Johnson, A. E. W. et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci. Data. 10 (2023). PMC9810617 36596836
  20. [22]Johnson, A. E. W. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data. 6 (2019). PMC6908718 31831740
  21. [23]Weber, M. et al. RedPajama: an Open Dataset for Training Large Language Models. Advances in neural information processing systems. (2025).
  22. [24]Rasley, J., Rajbhandari, S., Ruwase, O. & He, Y. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. (2020).
  23. [26]Kanjee, Z., Crowe, B. & Rodman, A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA. (2023). PMC10273128 37318797
  24. [28]Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. (2019).
  25. [32]Abacha, A. B., Shivade, C. P. & Demner-Fushman, D. Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. In Proceedings of the 18th BioNLP Workshop and Shared Task. 370-379. Florence, Italy (Association for Computational Linguistics, 2019).
  26. [33]Abacha, A. B. et al. Bridging the gap between consumers’ medication questions and trusted answers. Stud. health Technol. Inform. 264, 25–29 (2019). 31437878
  27. [34]Abacha, A. B., Agichtein, E., Pinter, Y. & Demner-Fushman, D. Overview of the Medical Question Answering Task at TREC 2017 LiveQA. In (Text Retrieval Conference, 2017).
  28. [36]Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2567-2577. Hong Kong, China (Association for Computational Linguistics, 2019).
  29. [37]Zhang, X., Wu, J., He, Z., Liu, X. & Su, Y. Medical Exam Question Answering with Large-scale Reading Comprehension. In Proceedings of the AAAI conference on artificial intelligence. (2018).
  30. [38]Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In Proceedings of the Conference on Health, Inference, and Learning. 248-260. (Proceedings of Machine Learning Research, 2022).
  31. [39]Pampari, A., Raghavan, P., Liang, J. J. & Peng, J. emrQA: A Large Corpus for Question Answering on Electronic Medical Records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2357-2368. Brussels, Belgium (2018).
  32. [40]Uzuner, Ö., South, B. R., Shen, S. & Duvall, S. L. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J. Am. Med. Inform. Assoc. 18 5, 552–556 (2011). 21685143 PMC3168320 · DOI
  33. [41]Segura-Bedmar, I., Martínez, P. & Herrero-Zazo, M. SemEval-2013 Task 9: Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). 341-350. Atlanta, Georgia, USA (Association for Computational Linguistics, 2013).
  34. [42]Baker, S. et al. Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinformatics 32, 432–440 (2016). 26454282 · DOI
  35. [43]Cohan, A. et al. A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 615-621. New Orleans, Louisiana (Association for Computational Linguistics, 2018).
  36. [44]Bastan, M., Surdeanu, M. & Balasubramanian, N. BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples. In Findings of the Association for Computational Linguistics: EMNLP 2022. 5093-5104. Abu Dhabi, United Arab Emirates (Association for Computational Linguistics, 2022).
  37. [45]Romanov, A. & Shivade, C. P. Lessons from Natural Language Inference in the Clinical Domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1586-1596. Brussels, Belgium (Association for Computational Linguistics, 2018).

Cited by

Sorted by citations

Authors
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., Zhou, X., Qian, L., He, H., Shung, D., Ohno-Machado, L., Wu, Y., Xu, H., & Bian, J.
Affiliations
  • Yale University (US)
  • University of Florida (US)
  • The University of Texas Health Science Center at Houston (US)
  • University of Florida Health (US)
Corresponding author
  • Xu, Hua
  • Bian, Jiang
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-03-05
Volume
8
Issue
1
Pages
141-141
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-025-01533-1
PMC
PMC11882967
OpenAlex
W4408165783
Cited by
74
References
7
Citations by year
2026: 42 · 2025: 32
FWCI
17.81
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by-sa)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-sa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (1RF1AG072799, 1R01AG078154, R01AG073435, R01LM013519, RF1AG084178, R01AG083039, R01CA284646, R01AI172875, R01AG080991, R01AG080624, R01AG080429, 1K99LM01402, 1K99LM014614-01,NIH/NCATS UL1 TR001427, CDC U18 DP006512)
  • Patient-Centered Outcomes Research Institute (PCORI RI-FLORIDA-01-PS1, PCORI ME-2018C3-14754)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
medical large language models, Llama 2, continual pre-training, instruction tuning, biomedical literature, clinical notes, clinical text mining, diagnosis, zero-shot performance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review