Evaluation of Large Language Model Performance on the Biomedical Language Understanding and Reasoning Benchmark: Comparative Study
Open accessAbstract
Background: The availability of increasingly powerful large language models (LLMs) has attracted substantial interest in their potential for interpreting and generating human-like text for biomedical and clinical applications. However, there are often demands for high accuracy, concerns about balancing generalizability and domain-specificity, and questions about prompting robustness when considering the adoption of LLMs for specific use cases. There also is a lack of a framework or method to help choose which LLMs (or prompting strategies) should be adopted for specific biomedical or clinical tasks. Objective: To address the speculations on applying LLMs for biomedical applications, this study aims to 1) propose a framework to comprehensively evaluate and compare the performance of a range of LLMs and prompting techniques on a suite of biomedical natural language processing (NLP) tasks; 2) use the framework to benchmark several general-purpose LLMs and biomedical domain-specific LLMs. Methods: We evaluated and compared six general-purpose LLMs (GPT-4, GPT-3.5-Turbo, Flan-T5-XXL, Llama-3-8B-Instruct, Yi-1.5-34B-Chat, and Zephyr-7B-Beta) and three healthcare-specific LLMs (Medicine-Llama3-8B, Meditron-7B, and MedLLaMA-13B) on a set of 13 datasets – referred to as the Biomedical Language Understanding and Reasoning Benchmark (BLURB) – covering six commonly needed medical natural language processing tasks: named entity recognition (NER); relation extraction (RE); population, interventions, comparators, and outcomes (PICO); sentence similarity (SS); document classification (Class.); and question-answering (QA). All models were evaluated without further training or fine-tuning. Model performance was assessed according to a range of prompting strategies (formalized as a systematic, reusable prompting framework) and relied on the standard, task-specific evaluation metrics defined by BLURB. Results: Across all tasks, GPT-4 outperformed other LLMs, achieving a score of 64.6 on the benchmark, though other models, such as Flan-T5-XXL and Llama-3-8B-Instruct, demonstrated competitive performance on multiple tasks. We found that general-purpose models achieved better overall scores than domain-specific models, sometimes by significant margins. We observed a substantial impact of strategically editing the prompt describing the task and a consistent improvement in performance when including examples semantically similar to the input text. Additionally, the most performant prompts for nearly half the models outperformed the previously reported best results for the PubMedQA dataset from the BLURB leaderboard. Conclusions: These results provide evidence of the potential LLMs may have for biomedical applications and highlight the importance of robust evaluation before adopting LLMs for any specific use cases. Notably, performant open-source LLMs such as Llama-3-8B-Instruct and Flan-T5-XXL show promise for use cases where trustworthiness and data confidentiality are concerns, as these models can be hosted locally, offering better security, transparency, and explainability. Continuing to explore how these emerging technologies can be adapted for the healthcare setting, paired with human expertise, and enhanced through quality control measures will be important research to allow responsible innovation with LLMs in the biomedical area.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even LLMs that perform well on medical benchmarks such as the United States Medical Licensing Examination may not generalize across all medical tasks.
“Even LLMs that perform well on medical benchmarks such as the United States Medical Licensing Examination 32 may not generalize across all medical tasks. 33”
From Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems (2025) · cites this paper as [33]
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
Creation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · see citing works
Evaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
Self-Alignment Pretraining for Biomedical Entity Representations
Liu, F., Shareghi, E., Meng, Z., Basaldella, M., & Collier, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.334 · Cited by 397 · see citing works
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.551 · Cited by 374 · see citing works
PMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · see citing works
GPT-NER: Named Entity Recognition via Large Language Models
Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F. Q., Zhang, T., Li, J., Wang, G., & Chen, G. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.239 · Cited by 287 · see citing works
GPT-NER: Named Entity Recognition via Large Language ModelsPreprint
Wang, S., Sun, X., Li, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.10428 · Cited by 144 · see citing works
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · see citing works
Zephyr: Direct Distillation of LM AlignmentPreprint
Tunstall, L., Beeching, E., Lambert, N., Rajani, N. F., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16944 · Cited by 80 · see citing works
Adapting Large Language Models to Domains via Reading ComprehensionPreprint
Cheng, D., Huang, S.-H., & Wei, F. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.09530 · Cited by 58 · see citing works
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.30 · Cited by 45 · see citing works
Unveiling the Generalization Power of Fine-Tuned Large Language Models
Yang, H., Zhang, Y., Xu, J., Lu, H., Heng, P., & Lam, W. P. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.51 · Cited by 44 · see citing works
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., & Wei, F. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.148 · Cited by 34 · see citing works
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., & Wei, F. (2024) · arXiv · doi:10.48550/arxiv.2406.14491 · Cited by 20 · see citing works
Two-stage LLM Fine-tuning with Less Specialization and More GeneralizationPreprint
Wang, Y., Si, S., Li, D., Łukasik, M., Yu, F., Hsieh, C., Dhillon, I. S., & Kumar, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.00635 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251348850 · Cited by 51 · see citing works
“33 Finally, an LLM cannot provide responses to novel medical questions whose answer is in content created after the LLM ’ s most recent training completion date.”
Kizilkaya, D., Sajja, R., Sermet, Y., & Demir, İ. (2025) · Environmental Data Science · doi:10.1017/eds.2025.10006 · Cited by 19 · see citing works
de Oliveira, R., Garber, M., Gwinnutt, J. M., Rashidi, E., Hwang, J.-H. (., Gilmour, W., Nanavati, J., Abidine, K. Z. E., & Mack, C. D. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf058 · Cited by 17 · see citing works
Self-citationTurón, G., Mulubwa, M., Montaner, A., Njoroge, M., Chibale, K., & Duran‐Frigola, M. (2025) · Nature Communications · doi:10.1038/s41467-025-64304-2 · Cited by 7 · see citing works
Zhu, J., Li, J., Zhao, S., Deng, Y., Miao, Y., & Xu, J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-025-08182-x · Cited by 2 · see citing works
Khalid, S. M., Wölker, T., Molano, L.-A. G., Graf, S., & Keller, A. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag089 · Cited by 1 · see citing works
AI coupled to pharmacometric modelling to tailor malaria and tuberculosis treatment in AfricaPreprint
Turón, G., Mulubwa, M., Montaner, A., Njoroge, M., Chibale, K., & Duran‐Frigola, M. (2024) · medRxiv · doi:10.1101/2024.11.07.24316884
Hunt, D., & Miceli, M. D. (2026) · American Journal of STEM Education · doi:10.32674/tfptev94
Githinji, P. B., Melliou, A., Liang, Z., Zhang, L., & Qin, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39379-1_35
- Authors
- Feng, H., Ronzano, F., LaFleur, J., Garber, M., de Oliveira, R., Rough, K., Roth, K., Nanavati, J., El Abidine, K. Z., & Mack, C.
- Affiliations
- IQVIA (United Kingdom) (GB)
- Corresponding author
- Feng, Hui
- ORCID
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-05-17
- OpenAlex
- W4397047623
- Cited by
- 9
- References
- 20
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, biomedical natural language processing, prompting strategies, named entity recognition, relation extraction, sentence similarity, document classification, question answering, domain-specific LLMs, GPT-4, LLM evaluation framework, clinical text understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
