PMC-LLaMA: toward building open-source language models for medicine
Open accessAbstract
Objective: Recently, large language models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering (QA) situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this article, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Materials and methods: We adapt a general-purpose LLM toward the medical domain, involving data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive domain-specific instruction fine-tuning, encompassing medical QA, rationale for reasoning, and conversational dialogues with 202M tokens. Results: While evaluating various public medical QA benchmarks and manual rating, our lightweight PMC-LLaMA, which consists of only 13B parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, and datasets for instruction tuning will be released to the research community. Discussion: Our contributions are 3-fold: (1) we build up an open-source LLM toward the medical domain. We believe the proposed PMC-LLaMA model can promote further development of foundation models in medicine, serving as a medical trainable basic generative language backbone; (2) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component, demonstrating how different training data and model scales affect medical LLMs; (3) we contribute a large-scale, comprehensive dataset for instruction tuning. Conclusion: In this article, we systematically investigate the process of building up an open-source medical-specific LLM, PMC-LLaMA.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have shown the ability to perform accurate single-turn medical question-answering.
“Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [11]
The 13B parameter PMC-LLaMA model demonstrates superior performance on various public medical QA benchmarks, even surpassing ChatGPT in certain evaluations.
“The 13B parameter PMC-LLaMA model demonstrates superior performance on various public medical QA benchmarks, even surpassing ChatGPT in certain evaluations.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [11]
Current benchmarks largely focus on multiple-choice problems.
“Current benchmarks, which largely focus on multiple-choice problems2,7,8, fail to adequately reflect the practical utility of LLMs in real-world clinical scenarios.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [7]
All results were obtained using a 3-shot prompting strategy except for MCQA tasks, where a zero-shot prompting setting was used to align with previous studies.
“All results presented here were obtained using a 3-shot prompting strategy (more details in the “Evaluation Settings” section in Supplementary), except for MCQA tasks, where we used a zero-shot prompting setting to align with previous studies7,8,19.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [7]
Current benchmarks fail to adequately reflect the practical utility of large language models in real-world clinical scenarios.
“Current benchmarks, which largely focus on multiple-choice problems2,7,8, fail to adequately reflect the practical utility of LLMs in real-world clinical scenarios.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [7]
Further auto-regressive training on medical-related corpus can inject medical knowledge into models, allowing them to perform better in different downstream tasks.
“We take the same approach as our previous work7,19, which have shown that further auto-regressive training on medical-elated corpus can inject medical knowledge into the models, thus allowing them to perform better in different downstream tasks.”
From Towards evaluating and building versatile large language models for medicine (2025) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationGPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
The Unified Medical Language System
Humphreys, B. L., McCray, A. T., & Lindberg, D. A. B. (1993) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1637976 · Cited by 512 · see citing works
S2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Self-citationTowards Generalist Foundation Model for RadiologyPreprint
Wu, C., Zhang, X., Zhang, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.02463 · Cited by 236 · see citing works
Self-citationCognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C. B., Martin, C. E., Flores, M. G., Zhang, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.03540 · Cited by 58 · see citing works
Self-citationLMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
Diao, S., Pan, R. N., Dong, H., Shum, K., Zhang, J., Xiong, W., & Zhang, T. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-demo.12 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 25
- OpenAI. OpenAI. Introducing ChatGPT. OpenAI; 2023. Accessed April 2, 2024. https://openai.com/blog/chatgpt/ 02. OpenAI. GPT-4 technical report. arXiv:2303.08774. 2023, pre- print: not peer reviewed. · arXiv
- Nori H, King N, et al. Capabilities of GPT-4 on medical challenge problems. arXiv 230313375. 2023, preprint: not peer reviewed.
- Touvron H, Lavril T, Izacard G, Martinet , et al. LLaMA: open and efficient foundation language models. arXiv 230213971. 2023, preprint: not peer reviewed.
- Han T, Adams LC, et al. MedAlpaca -an open-source collection of medical conversational AI models and training data. arXiv 230408247. 2023, preprint: not peer reviewed.
- Yunxiang L, et al. Chatdoctor: a medical chat model fine-tuned on LLaMA model using medical domain knowledge. arXiv 230314070. 2023, preprint: not peer reviewed.
- Anil R, Dai AM, et al. PaLM 2 technical report. arXiv abs/ 2305.10403. 2023, preprint: not peer reviewed. https://api.seman- ticscholar.org/CorpusID:258740735
- Du Z, Qian Y, Liu X, et al. GLM: General Language Model Pre- training with Autoregressive Blank Infilling. ACL; 2022. Accessed April 2, 2024. https://aclanthology.org/2022.acl-long.26
- Peng B, et al. Instruction tuning with GPT-4. arXiv 230403277. 2023, preprint: not peer reviewed.
- Chung HW, et al. Scaling instruction-finetuned language models. arXiv 221011416. 2022, preprint: not peer reviewed.
- Iyer S, et al. OPT-IML: scaling language model instruction meta learning through the lens of generalization. arXiv 221212017. 2022, preprint: not peer reviewed.
- Zhang K, et al. BiomedGPT: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multi- modal tasks. arXiv 230517100. 2023, preprint: not peer reviewed.
- Jin Q, et al. PubMedQA: a dataset for biomedical research ques- tion answering. ACL. 2019:2567-2577.
- Pal A, Umapathi LK, et al. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning. PMLR; 2022:248-260.
- Loshchilov I, Hutter F. Decoupled Weight Decay Regularization. ICLR; 2018.
- Chen T, et al. Training deep nets with sublinear memory cost. arXiv 160406174. 2016, preprint: not peer reviewed.
- Hu EJ, Shen Y, et al. LoRA: Low-Rank Adaptation of Large Lan- guage Models. ICLR; 2021.
- Liu X, Ji K, Fu Y, et al. P-Tuning: Prompt Tuning Can Be Compa- rable to Fine-Tuning Across Scales and Tasks. ACL; 2022:61-68. Accessed April 2, 2024. https://aclanthology.org/2022.acl-short.8
- Kaplan J, McCandlish S, Henighan TJ, et al. Scaling laws for neural language models. arXiv abs/2001.08361. 2020, preprint: not peer reviewed. https://api.semanticscholar.org/CorpusID:210861095 · arXiv
- Chen Z, Cano AH, Romanou A, et al. MEDITRON-70B: scaling medical pretraining for large language models. arXiv abs/ 2311.16079. 2023, preprint: not peer reviewed. https://api.seman- ticscholar.org/CorpusID:265456229
- Moor M, Huang Q, Wu S, et al. Med-Flamingo: a multimodal medical few-shot learner. In: Proceedings of the 3rd Machine Learning for Health Symposium. PMLR; 2023:225:353-367. https://proceedings.mlr.press/v225/moor23a.html
- Li C, Wong C, Zhang S, et al. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. NIPS. 2023;36;28541-28564.
- Liu J, Jin J, Wang Z, Cheng J, Dou Z, Rong Wen J. RETA-LLM: a retrieval-augmented large language model toolkit. arXiv abs/ 2306.05212. 2023, preprint: not peer reviewed. https://api.seman- ticscholar.org/CorpusID:259108339
- Schick T, Dwivedi-Yu J, Dess� ı R, et al. Toolformer: language mod- els can teach themselves to use tools. NIPS. 2024;36:68539-68551.
- Yang R, Song L, Li Y, et al. GPT4Tools: teaching large language model to use tools via self-instruction. NIPS. 2023;36:71995-72007.
- All rights reserved. For permissions, please email: journals.permissions@oup.com Journal of the American Medical Informatics Association, 2024, 00, 1-11 https://doi.org/10.1093/jamia/ocae045 Research and Applications · DOI
Cited by
Sorted by citations
Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56989-2 · Cited by 219 · see citing works
“An accuracyof 0.7790 on PubMedQA wasalsoreported for the fi ne-tuned PMC LLaMA 13B (combined multiple question answering datasets for fi ne-tuning) 34 ; our study also reported a similar accuracy of 0.7680 using the PubMedQA training set only.”
InfluentialTowards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147 · see citing works
Self-citationEvaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68 · see citing works
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63 · see citing works
Towards evaluating and building versatile large language models for medicine
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01390-4 · Cited by 52 · see citing works
Self-citationQuantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53 · see citing works
Self-citationBenchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35 · see citing works
Rossettini, G., Bargeri, S., Cook, C., Guida, S., Palese, A., Rodeghiero, L., Pillastrini, P., Turolla, A., Castellini, G., & Gianola, S. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1574287 · Cited by 31 · see citing works
Meddeb, A., Ebert, P., Bressem, K. K., Desser, D., Dell'Orco, A., Bohner, G., Kleine, J. F., Siebert, E., Grauhan, N., Brockmann, M. A., et al. (2025) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2024-022078 · Cited by 27 · see citing works
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.266 · Cited by 13 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
Feng, H., Ronzano, F., LaFleur, J., Garber, M., de Oliveira, R., Rough, K., Roth, K., Nanavati, J., El Abidine, K. Z., & Mack, C. (2024) · doi:10.1101/2024.05.17.24307411 · Cited by 9 · see citing works
Enhancing LLM Reasoning Capabilities Through Brokered Multi-Expert Reflection
Sheokand, T., Jain, G., Bahga, A., & Madisetti, V. K. (2025) · IEEE Access · doi:10.1109/access.2025.3561693 · Cited by 7 · see citing works
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 6 · see citing works
LLM-driven medical document analysis: Enhancing trustworthy pathology and differential diagnosis
Kang, L., Fu, X., Terrades, O. R., Vazquez-Corral, J., Valveny, E., & Karatzas, D. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-04624-6_36 · Cited by 1 · see citing works
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
- Authors
- Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Wang, Yanfeng
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-04-13
- Volume
- 31
- Issue
- 9
- Pages
- 1833-1843
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11639126
- OpenAlex
- W4394782456
- Cited by
- 300
- References
- 16
- Citations by year
- 2026: 83 · 2025: 160 · 2024: 57
- FWCI
- 44.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, other-oa)
- PubMed Central (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Funders
- National Key R&D Program of China (2022ZD0160702)
- Science and Technology Commission of Shanghai Municipality (22511106101, 18DZ2270700, 21DZ1100100)
- Higher Education Discipline Innovation Project 111 (BP0719010)
- State Key Laboratory of UHD Video and Audio Production and Presentation.
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- open-source language models, medical large language models, instruction tuning, medical question answering, domain-specific fine-tuning, medical textbooks, biomedical literature, conversational dialogue, medical reasoning, instruction-tuning dataset
- MeSH terms
- Humans, Natural Language Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
