The DOI matches its Crossref record; 93 of the 103 references it cites were found in a registry or an index.
Benchmarking large language models for biomedical natural language processing applications and recommendations
Open accessAbstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“For the tasks where the gold standard is fi xed, e.g., a classi fi cation type or multiple-choice option, we conducted qualitative evaluations on collectively hundreds of thousands of raw outputs of the LLMs (the raw outputs from three LLMs under zero-and one-shot conditions acrossthree benchmarks)…”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 4×
“Since the inception of transformers, biomedical language models have adopted their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family includi…”
InfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 1×
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“This is also supported by the existing studies on medical question answering 38,46 .”
Research works printed but not linked: 7
- Jiang, A. Q. et al. Mixtral of experts arXiv preprint arXiv:2401.04088, 2024. · arXiv
- Lee, P, Goldberg, C. & Kohane, I. The AI revolution in medicine: GPT-4 and beyond (Pearson, 2023).
- Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. & Daneshjou, R. Large language models in medicine: the potentials and pitfalls: a narrative review. Ann. Intern. Med. 177, 210–220 (2024). · DOI
- Krallinger, M. et al. Overview of the BioCreative VI chemical-protein interaction Track. In Proc. of the sixth BioCreative challenge evaluation workshop Vol. 1, 141–146 (2017).
- Segura-Bedmar, I., Martínez, P. & Herrero-Zazo, M. Semeval-2013 task 9: extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proc. Seventh International Workshop on Semantic Evaluation (SemEval 2013) 341–350 (Association for Computational Linguistics, 2013).
- Mingliang, D., Jijun, T. & Fei, G. Document-level DDI relation extraction with document-entity embedding. pp. 392–397. · DOI
- Wang, L. L, DeYoung, J. & Wallace, B. Overview of MSLR2022: A shared task on multidocument summarization for literature reviews. In Proc. Third Workshop on Scholarly Document Processing 175–180 (Association for Computational Linguistics, 2022).
Cited by
Sorted by citations
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414
Self-citationOpportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
“Chen et al. [105] performed a pilot study to establish the baselines of using GPT-3.”
BackgroundSelf-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312
“Neverthe-less, current systematic evaluations in biomedicine typically focus on the vanilla LLMs without RAG (Chen et al., 2023a; Nori et al., 2023a).”
BackgroundSelf-citationA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Self-citationSivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2024) · JMIR Medical Informatics · doi:10.2196/55318 · Cited by 293
“For instance, Chen et al [9] used a fixed template as the prompt to measure the performance of LLMs on biomedical NLP tasks but did not investigate different kinds of prompting methods.”
BackgroundMe-LLaMA: Foundation Large Language Models for Medical ApplicationsPreprint
Xie, Q., Chen, Q., Chen, A., Cheng, P., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V. K., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4240043/v1 · Cited by 74
“29,30 When compared with other NLP tasks with higher performance, we noticed that one of the main reasons for low performance is that LLMs’ responses often lacked the conciseness and precision expected, with instances of missing outputs noted.”
BackgroundSelf-citationTaiyi: a bilingual fine-tuned large language model for diverse biomedical tasks
Luo, L., Ning, J., Zhao, Y., Wang, Z., Ding, Z., Chen, P., Fu, W., Han, Q., Xu, G., Qiu, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae037 · Cited by 100
Self-citationAdvancing entity recognition in biomedicine via instruction tuning of large language models
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae163 · Cited by 104
“Previously, in a pilot study (Chen et al. 2023) we evaluated the performance of GPT-3.”
BackgroundMethodInfluentialSelf-citationMedical foundation large language models for comprehensive text analysis and beyond
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01533-1 · Cited by 74
“Despitethesestrengths,weobservedcertainchallengesinspeci fi ctasks, such as NER and RE 14,15 , whereeven advanced modelslike GPT-4 exhibited low performance.”
Self-citationDu, X., Novoa-Laurentiev, J., Plasek, J. M., Chuang, Y.-W., Wang, L., Marshall, G. A., Mueller, S. K., Chang, F., Datta, S., Paek, H., et al. (2024) · eBioMedicine · doi:10.1016/j.ebiom.2024.105401 · Cited by 69
Self-citationUnveiling LLM Evaluation Focused on Metrics: Challenges and SolutionsPreprint
Hu, T., & Zhou, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.09135 · Cited by 98
“Wang et al. (2023) highlighted the promising applications of LLMs in addressing biomedical questions, while Chen et al. (2023) focused on evaluating biomedical LLMs with respect to benchmarks and summarization capabilities.”
BackgroundIdentifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models
Shyr, C., Hu, Y., Bastarache, L. A., Cheng, A., Hamid, R., Harris, P. A., & Xu, H. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00155-0 · Cited by 77
Self-citationZhang, X., Talukdar, N., Vemulapalli, S., Ahn, S., Wang, J., Han, M., Murtaza, S. M. B., Leshchiner, D. R., Dave, A. A., Joseph, D. F., et al. (2024) · medRxiv · doi:10.1101/2024.02.07.24302444 · Cited by 42
“5 and GPT-4 showed promise in biomedical semantic similarity and reasoning tasks but are less effective in information extraction and classification 10 .”
BackgroundLarge language models for drug discovery and development
Zheng, Y., Koh, H. Y., Ju, J., Yang, M., May, L. T., Webb, G. I., Li, L., Pan, S., & Church, G. M. (2025) · Patterns · doi:10.1016/j.patter.2025.101346 · Cited by 121
Sushil, M., Zack, T. I., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., Lituiev, D., & Butte, A. J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae146 · Cited by 64
“We found that even in zero-shot setups, the GPT-4 model performs as well as or 35,36 Similar small datasets are common in medical informatics studies since domain expertise is frequently required for reliably annotating clinical notes, making the process time-consuming and difficult to scale.”
BackgroundPark, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42
An evaluation of GPT models for phenotype concept recognition
Groza, T., Caufield, H., Gration, D., Baynam, G. S., Haendel, M. A., Robinson, P. N., Mungall, C., & Reese, J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02439-w · Cited by 46
“The boundary detection step proved to be challenging to evaluate accurately with the results produced by the OpenAI GPT models - aspect documented also by Chen at [19].”
MethodTang, X.-R., Shao, Y., Sohn, J., Chen, J.-P., Zhang, J., Xiang, J.-Y., Wu, F., Zhao, Y., Wu, C.-L., Shi, W.-Q., et al. (2025) · Patterns · doi:10.1016/j.patter.2026.101610 · Cited by 44
Information extraction from clinical notes: are we ready to switch to large language models?
Hu, Y., Zuo, X., Zhou, Y., Peng, X., Huang, J., Keloth, V. K., Zhang, V., Weng, R.-L., Shyr, C., Chen, Q., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf213 · Cited by 55
“23,24 However, their effectiveness in traditional IE remains mixed.”
Self-citationSivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2309.08008 · Cited by 42
“Chen et al.(8) used a fixed template as the prompt to measure the performance of LLMs on biomedical NLP tasks, but did not investigate different kinds of prompting methods.”
BackgroundMansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 46
“While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines Chen et al., [93].”
Du, X.-S., Zhou, Z., Wang, Y.-F., Chuang, Y.-W., Li, Y.-M., Yang, R., Zhang, W., Wang, X., Chen, X.-Y., Guan, H., et al. (2024) · medRxiv · doi:10.1101/2024.08.11.24311828 · Cited by 36
“27,65 This may be due to the fact that prompting strategies can increase the length of a prompt, and a longer prompt might negatively impact the LLM’s performance.”
BackgroundPeng, X., Qian, L., Wang, Y., Xiang, R., He, Y., Ren, Y., Jiang, M., Zhang, V. J., Guo, Y., Zhao, J., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.770 · Cited by 37
“LLMs have demonstrated remarkable prowess in various tasks (Zhao et al., 2025; Chen et al., 2025b), yet their applications to finance (Liu et al., 2023b; Xie et al., 2023; Wu et al., 2023) require the continual push of the limits of model capabilities (Xiong et al., 2025).”
Self-citationBRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice TextPreprint
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y.-X., Carducci, V., Wyss, R., Desai, R., Alsentzer, E., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.19467 · Cited by 25
“20,21 These standardized question sets offer a basic and rapid assessment, such as the GPT-4 22 and Med-PaLM 2 23 , achieving expert-level scores on the USMLE.”
Keloth, V. K., Selek, S., Chen, Q., Gilman, C., Fu, S., Dang, Y., Chen, X., Hu, X., Zhou, Y., He, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01645-8 · Cited by 24
“Our extensive prior work comparing zero/few-shot and fi ne-tuned LLaMA models have demonstrated the superior performance of fi ne-tuned models 51 – 54 .”
Self-citationde Oliveira, R., Garber, M., Gwinnutt, J. M., Rashidi, E., Hwang, J.-H. (., Gilmour, W., Nanavati, J., Abidine, K. Z. E., & Mack, C. D. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf058 · Cited by 22
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 24
Self-citationLarge Language Models Vote: Prompting for Rare Disease IdentificationPreprint
Oniani, D., Hilsman, J., Dong, H., Gao, F., Verma, S., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.12890 · Cited by 14
“Recently, studies have shown that using formats, such as XML, can work well for model evaluation (Chen et al., 2023).”
BackgroundLiu, Y., Duran, E., Bruce, A., Yildiz, B., Severiano, B. M., Ibrahim, I. A., Rispler, J., Martell, C., & Rougieux, F. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126636 · Cited by 18
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“In the biomedical domain, fine-tuned BioGPT 23 has shown superior performance to BERT-based models in document classification, while the performance of the GPT models in zero-or one-shot settings has been found to trail that of fine-tuned BERT models 53 .”
Method(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18
“At the same time, our results with GPT-4 are consistent with other comparisons of GPT-4 in-context learning with fine-tuned models for text classification 53 .”
MethodResultLiu, S., McCoy, A. B., Chen, Q., & Wright, A. T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106104 · Cited by 17
Self-citationZhu, Y., Gao, J., Wang, Z., Liao, W., Zheng, X., Liang, L., Bernabéu, M. O., Wang, Y., Yu, L., Pan, C., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02539-z · Cited by 17
Self-citationMultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation
Peng, X.-Q., Qian, L.-F., Wang, Y., Xiang, R.-Y., He, Y.-R., Ren, Y., Jiang, M.-Y., Zhao, J., He, H., Han, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.14028 · Cited by 16
“LLMs have demonstrated remarkable prowess in various tasks (Zhao et al., 2025; Chen et al., 2025b), yet their applications to finance (Liu et al., 2023b; Xie et al., 2023; Wu et al., 2023) require the continual push of the limits of model capabilities (Xiong et al., 2025).”
Self-citationKim, H., Sohn, J., Gilson, A., Cochran-Caggiano, N., Applebaum, S. S., Jin, H., Park, S., Park, Y., Park, J., Choi, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.06738 · Cited by 21
Self-citationPeng, C., Zhang, K., Lyu, M., Liu, H., Sun, L., & Wu, Y. (2025) · Journal of Biomedical Informatics · doi:10.48550/arxiv.2505.17436 · Cited by 15
RAG-Enhanced Open SLMs for Hypertension Management Chatbots
Aguzzi, G., Magnini, M., Farahmand, A., Ferretti, S., Pengo, M. F., & Montagna, S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02297-7 · Cited by 16
“For instance, [ 16 ] presents a SOTA fine-tuning approach that, for the open-source LLMs evalu - ated, outperforms zero-and few-shot approaches in the QA task.”
LLM-based UAV Path Planning for Autonomous and Adaptive Industry Systems
Xiao, W., Shi, C., Chen, M., Vasilakos, A. V., Chen, M., & Farouk, A. (2025) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3774655 · Cited by 13
“However, surprisingly the fine-tuned Mistral-7B performs very poorly, which on the one hand highlights the complexity of the UAV swarm path planning task, and on the other hand confirms the existence of a performance gap between different LLM models [7].”
BioContextAI is a community hub for agentic biomedical systems
Kuehl, M. B., Schaub, D. P., Carli, F., Heumos, L., Hellmig, M., Fernández‐Zapata, C., Kaiser, N., Schaul, J., Kulaga, A. Y., Usanov, N., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02900-9 · Cited by 11
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 28
Multi-agent systems for clinical decision support: A systematic review
Silveira, A. L., Righi, R. D. R., & da Costa, C. A. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114447 · Cited by 11
Language models for drug–drug interactions: current applications, pitfalls, and future directions
Meslamani, A. Z. A., & Hajal, A. A. (2025) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2025.2551724 · Cited by 13
“For LLMs to be fully accepted in clinical pharmacology processes, they must improve openness and dependability [119,120].”
Cherif, B., Bisztray, T., Dubniczky, R. A., Aldahmani, A., Alshehhi, S., & Tihanyi, N. (2025) · International Conference on Neural Information Processing · doi:10.48550/arxiv.2505.19973 · Cited by 12
“LLMs hold immense potential for various fields, including cybersecurity [31], software engineering [22], biomedicine [4] or law [6], which has sparked calls for privacy-aware, reliability-oriented, and domain-tailored benchmarks [35].”
DiMB-RE: mining the scientific literature for diet-microbiome associations
Hong, G., Hindle, V., Veasley, N. M., Holscher, H. D., & Halil Kilicoglu, B. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf054 · Cited by 12
“For RE, in both cases, relation types must match.”
BioMamba: Domain-Adaptive Biomedical Language Models
Yue, L., Zhu, M., Xing, S., Cao, Y., Wang, Y., Chenthamarakshan, V., Pan, S., Das, P., & Fu, T. (2026) · Health Data Science · doi:10.34133/hds.0454 · Cited by 16
Elbiach, O., Grissette, H., & Nfaoui, E. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107362 · Cited by 11
van Velzen, M., van der Willigen, R. F., de Beer, V. J., de Graaf-Waar, H. I., Janssen, E., van Leeuwen, S., van der Willigen, M. F., van der Willigen, M. J., Renardus, G., Maaroufi, R. E., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1644084 · Cited by 8
Large language models for spreading dynamics in complex systems
Jiang, S., Ren, H., Gao, Y., Zhang, Y., Li, Q., Xiao, D., Fan, J., Tang, R., & Wang, W. (2026) · Physics Reports · doi:10.1016/j.physrep.2026.01.006 · Cited by 10
Large Language Models in Bio-Ontology Research: A Review
Manda, P. (2025) · Bioengineering · doi:10.3390/bioengineering12111260 · Cited by 9
“Finally, systematic benchmarking efforts [77,78] provide critical evaluations of LLM performance on biomedical tasks, underscoring both the promise and variability of current approaches.”
InfluentialYang, E.-W., Waldrup, B., & Velazquez-Villarreal, E. (2025) · medRxiv · doi:10.1101/2025.05.20.25327967 · Cited by 10
“The emergence of artificial intelligence (AI), especially advancements in large language models (LLMs), has paved the way for conversational tools that convert natural language input into executable data analysis pipelines [23–24].”
Enhancing biomedical relation extraction with directionality
Lai, P., Wei, C.-H., Tian, S., Leaman, R., & Lu, Z. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf226 · Cited by 12
“Recent studies (Chen et al. 2025) indicate that Llama 3 (AI@Meta 2024) exhibits performance comparable to GPT-3.”
Self-citationSushil, M., Zack, T., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., & Butte, A. (2024) · Research Square · doi:10.21203/rs.3.rs-3914899/v1 · Cited by 10
“Finally, we often find that skin is one of the specimens examined.”
BackgroundATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue
Cao, R.-K., Bai, S., Yao, F.-G., Dong, L., Xu, J., & Xiao, L.-Q. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2603.02216 · Cited by 9
“…[6], Qwen3 Yang et al. [41], and DeepSeek-R1 Guo et al. [11] have demonstrated exceptional capabilities across a range of natural language processing tasks, including open-domain question answering, dialogue generation, and code generation, continuously pushing the boundaries of AI performance [4].”
Causal graph neural networks for healthcare
Mesinovic, M., Buhlan, M., & Zhu, T. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01742-3 · Cited by 8
“…biomedical natural language processing tasks demonstrates that while LLMs excel at reasoning-intensive question answering, they under-perform fine-tuned domain-specific models in information extraction, with qualitative analyses revealing prevalent missing information and inconsistent outputs [84].”
Syahputri, I. W., Budiardjo, E. K., & Putra, P. H. H. (2025) · Applied Informatics · doi:10.3390/ai6090206 · Cited by 9
Entry-level guide to the use of large language models for medical researchPreprint
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z.-Z., Yang, Y.-F., Wang, Z., Xiong, G.-Z., Lai, P.-T., Zhu, Q.-Q., et al. (2024) · PubMed · doi:10.48550/arxiv.2410.18856 · Cited by 9
“For example, LLMs are better for tasks that require medical knowledge and clinical reasoning, but do not often outperform fine-tuned BERT models 63 in simpler tasks such as structurization 64 .”
Self-citationEnhancing Large Language Models with Domain-specific Retrieval Augment Generation
Gilson, A., Ai, X.-G., Arunachalam, T., Chen, Z., Cheong, K. X., Dave, A. D., Duic, C., Kibe, M., Kaminaka, A., Prasad, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2409.13902 · Cited by 9
Self-citationDong, Y., Zhang, Z., Zhi, Y., Li, X., Guo, T., He, L., Zhao, S., Yang, X., Tang, J. T., Zhong, W., et al. (2025) · JHEP Reports · doi:10.1016/j.jhepr.2025.101579 · Cited by 12
Multi-Modal Topology-Aware Graph Neural Network for Robust Chemical–Protein Interaction Prediction
Wang, J. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26178666 · Cited by 11
Shao, C., Snyder, D., Li, C., Gu, B., Ngan, K., Yang, C.-T., Wu, J., Wyss, R., Lin, K., & Yang, J. (2025) · Journal of Clinical Epidemiology · doi:10.48550/arxiv.2506.11137 · Cited by 8
“- Macro Precision - Macro Recall - Macro F1”
OpenAI ChatGPT: A Potential Medical Application
Chaddad, A., Jiang, Y., & He, C. (2023) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom56612.2023.10472387 · Cited by 9
“…American College of Gastroenterology Self-Assessment Tests [18] 65.1% 62.4% PSITE 2021 Exam [19] 51.2% 75.3% PubMedQA [20] 51.6% 80.4% NCBI-disease [16] 24.05% 56.73% MedMCQA [13] 50.08% 73.42% SANS ABNS Self-Assessment Exam [21] 73.4% 83.4% BC5CDR: BioCreative V CDR; SCE: Specialty Certificate…”
InfluentialEmerging trends of recommender system for e-commerce: a comprehensive review
Rajpoot, C. S., Tiwari, V., & Vishwakarma, S. K. (2026) · Discover Computing · doi:10.1007/s10791-026-09923-z · Cited by 7
“Text summarization techniques–both extractive and abstractive address this issue by condensing large volumes of textual data into concise, meaningful summaries that retain essential semantic information [119].”
Ghosh, A., Jain, R., Jhangra, A., Saha, S., & Jatowt, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70045 · Cited by 5
“…RoBERTa for entailment and CUI linking was reported, which propagates into higher downstream ROUGE when these representations seed the summarizer (Chen et al. 2025 (Sundrelingam et al. 2025) processes each note in about 0.5 s and posts an F1 of 0.90 on both Canadian and n2c2 discharge-summary…”
InfluentialSholehrasa, H., Ghanaatian, A., Caragea, D., Tell, L. A., Riviere, J. E., & Jaberi-Douraki, M. (2025) · IEEE International Conference on Tools with Artificial Intelligence · doi:10.1109/ictai66417.2025.00051 · Cited by 7
“1, Step 8) using a 5-shot prompting [33] (Fig.”
Benchmarking GPT-5 for biomedical natural language processingPreprint
Hou, Y., Zhan, Z.-F., Zeng, M., Wu, Y.-F., Zhou, S., & Zhang, R. (2025) · arXiv.org · doi:10.48550/arxiv.2509.04462 · Cited by 7
“5, and LLaMA-2-13B across five text-based task families originally benchmarked by Chen et al. 16 : named entity recognition (NER), relation extraction (RE), multi-label document classification, text summarization, and text simplification.”
InfluentialSelf-citationChen, H., Alfred, M. C., & Cohen, E. (2025) · JMIR Medical Informatics · doi:10.2196/68955 · Cited by 13
“While our findings highlight the promise of open-source LLMs in data-scarce settings, the exclusion of proprietary models limits the generalizability of our findings.”
Adam, T. J., Abosabie, S. A., Dittmer, M., Wolf, E., Abosabie, S. A., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C. U., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10093 · Cited by 8
Huang, X., Xiao, M., Qin, C., Long, Q., Chen, J., Zhou, Y., & Zhu, H. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817489 · Cited by 3
Latency-Aware Benchmarking of Large Language Models for Natural-Language Robot Navigation in ROS 2
Das, M., Hussain, Z., & Nawaz, M. (2026) · Sensors · doi:10.3390/s26020608 · Cited by 5
“This metric is relevant for large-scale or cost-sensitive deployments [21]. https://doi.org/10.3390/s26020608 2.7.”
InfluentialApplication and Prospects of Large Language Models in Small-Molecule Drug Discovery
Ma, J., Liu, J., Xu, D., Song, X., & Zhang, Z. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.5c04083 · Cited by 9
Zhan, Z.-F., Zhou, S., Zeng, M., Yu, K., Song, M.-J., Chen, X., Wang, J., Hou, Y., & Zhang, R. (2025) · arXiv.org · doi:10.48550/arxiv.2509.04534 · Cited by 6
Self-citationLi, X., Zhao, D., Meng, J., & Lin, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128342 · Cited by 5
LLM-Based Agents for Competitive Landscape Mapping in Drug Asset Due Diligence
Vinogradova, A., Vinogradov, V., Radkevich, D., Yasny, I., Kobyzev, D., Izmailov, I., Yanchanka, K., Doronin, R., & Doronichev, A. (2025) · arXiv.org · doi:10.48550/arxiv.2508.16571 · Cited by 6
“Moreover, recent studies report that general-purpose, frontier LLMs can be competitive with—or even outperform—task-specific fine-tuned models on certain pharma-adjacent evaluations (Chen et al. 2025; McDuff et al. 2025).”
Yu, J., Tang, Y., Feng, K., Liang, L., Zhang, Q., Ding, K., & Chen, H. (2026) · Scientific Data · doi:10.1038/s41597-026-06594-9 · Cited by 6
“Existing LLM benchmarks in scientific domains (Chen et al., 2025; Sun et al., 2024; Feng et al., 2024; Saikh et al., 2022; Pedersen et al., 2020; Rubungo et al., 2024; Jiang et al., 2025) primarily focus on direct question-answering tasks, offering limited insight into how well LLMs perform in…”
Sarol, M. J., Hong, G., Guerra, E., & Halil Kilicoglu, B. (2024) · Database · doi:10.1093/database/baae079 · Cited by 6
Liu, Z., Liu, S., Yan, W., Song, P., Song, Y., & Gao, R. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792577 · Cited by 2
Yu, S., Cho, E.-J., Kim, S., Park, K., Kim, M.-S., Oh, Y., & Ryu, H. (2026) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2026.106270 · Cited by 5
Zeng, H., Liu, G., Liu, W., Zou, S.-Y., Chen, B., Yang, R., Lin, F., Peng, J., Gao, Y., & Hao, L. (2025) · Public Health · doi:10.1016/j.puhe.2025.106042 · Cited by 5
Self-citationDual-LLM Adversarial Framework for Information Extraction from Research Literature
Li, Z., Yu, Y., Gu, W.-H., Zhu, T.-T., Song, H., Guo, W.-B., Yang, X., & Zhu, Z.-X. (2025) · bioRxiv · doi:10.1101/2025.09.11.675507 · Cited by 5
Li, W., Tang, F., Zhao, M., Omachi, M., & Kato, N. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3644352 · Cited by 3
“Inrecentyears,largelanguagemodels(LLMs)havedemon- stratedextraordinarycapabilitiesinnaturallanguageprocess-ing(NLP),includingconversation,reasoning,andtextgener-ation[28],[29],[30].”
Yin, L., Xue, X., Pan, S., Liu, X., Zhou, Y., Nahar, L., Chen, Y., Zhao, J., Zeng, S., & Xu, T. (2025) · Acta Materia Medica · doi:10.15212/amm-2025-0045 · Cited by 6
“These practices establish a robust foundation for applying LLMs in complex, individualized TCM contexts [159].”
Freeman, J., Millikin, R. J., Xu, L., Sharma, I., Moore, B., Lock, C., Shine George, K., Bal, A., Mohanty, C., & Stewart, R. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06350-7 · Cited by 4
“General-purpose LLMs [2, 9, 24] achieve state-of-the-art scores on biomedical reasoning and question-answering benchmarks [7, 17, 28].”
SynNER: syntax-infused named entity recognition in the biomedical domain
Imran, M., Zamaraeva, O., & Gómez‐Rodríguez, C. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf149 · Cited by 2
“In this study, we use five biomedical benchmark datasets described in Chen et al. 24 (Table 1).”
Chen, Z., Gao, Y., Gu, S., Yong, Z., Li, L., Wang, R., Lei, Q., & Zeng, S. (2026) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00140-3 · Cited by 4
“However, since its training data spans a wide array of general domains rather than medical-specific datasets, its performance in specialized tasks such as medical reasoning and anesthesia plan generation may encounter limitations [19].”
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Adib, S. E. S., Sani, A. A., Esham, E. A., Abrar, A., & Chowdhury, T. M. (2025) · 2025 28th International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491589 · Cited by 4
“We employed standardized automatic evaluation metrics commonly used in medical NLP research to assess model performance across multiple dimensions [2], [15]: • BLEU Scores: Measures n-gram overlap between generated and reference answers, providing insights into lexical similarity and content…”
InfluentialHySim-LLM: Embedding-Weighted Fine-Tuning Bounds and Manifold Denoising for Domain-Adapted LLMs
Jaberi-Douraki, M., Sholehrasa, H., Xu, X., & Ramachandran, R. (2025) · arXiv.org · doi:10.48550/arxiv.2510.07796 · Cited by 4
“Recent advances in large language models (LLMs) have revolutionized natural language processing, enabling state-of-the-art performance in text summarization, retrieval, and reasoning [4].”
The application of large language models in qualitative nursing research: A scoping review
Zhou, T., Xing, M., Hu, Y., Liang, J., Ren, Q., Yang, Y., & Ye, L. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102544 · Cited by 6
Self-citationA review on recent advancements of ChatGPT and datafication in healthcare applications
Jagatheesaperumal, S. K., Pandiyarajan, A., Prabadevi, B., Deepa, N., Barreto, A. G., & de Albuquerque, V. H. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110885 · Cited by 6
Community-based biomedical context to unlock agentic systems
Kuehl, M., Schaub, D. P., Carli, F., Heumos, L., Fernández-Zapata, C., Kaiser, N., Schaul, J., Panzer, U., Bonn, S., Lobentanzer, S., et al. (2025) · bioRxiv · doi:10.1101/2025.07.21.665729 · Cited by 4
“Despite their potential applications, LLMs have significant limitations in biomedicine as they often generate hallucinations [4], cannot access specialized databases, and lack the necessary domain knowledge and functional capabilities [5], which are crucial for providing reliable scientific outputs.”
An In-Depth Evaluation of Federated Learning on Biomedical Natural Language ProcessingPreprint
Peng, L., Luo, G., Zhou, S.-C., Chen, J., Zhang, R., Xu, Z.-C., & Sun, J. (2023) · medRxiv · doi:10.1101/2023.11.24.23299002 · Cited by 4
“We followed the experimental protocol outlined in a recent study 32 and evaluated all the models on two NER datasets (2018 n2c2 and NCBI-disease) and two RE datasets (2018 n2c2, and GAD).”
MethodSelf-citationBenchmarking zero-shot biomedical relation triplet extraction across language model architectures
Gade, F. S., Lund, O., & Mendoza, M. L. (2025) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2025.bionlp-1.9
“Multiple papers have benchmarked LLMs for IE tasks on biomedical texts (Dai et al., 2024; Jahan et al., 2024; Chen et al., 2025), and there are multiple established combined benchmark datasets (e.g. BLURB (Gu et al., 2021)) and LLM instruction datasets (e.g. SciRIFF (Wadden et al., 2024)), but two…”
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 15
Self-citationMedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical ContextPreprint
Zhou, H., Zou, X., Wu, J., Wu, S., Yu, J., Segal, B., Niebuhr, T. E., Amro, S., Petrus, M., Momin, S., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.25.727671 · Cited by 2
Wang, R.-D., Ba, W., Li, Y., Zhou, Y., Liu, B.-W., Wang, B.-Z., Wang, Y.-H., Yang, Z.-D., Zhang, K., Yan, R., et al. (2026) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.48550/arxiv.2603.01647 · Cited by 3
“Crucially, we introduce a quality control module that utilizes advanced LLMs [6,27,1] to simulate a pathologist’s auditing behavior.”
Fodeh, S., Wang, Y., Ma, L.-H., Talakokkul, S., Alpert, J. M., & Schellhorn, S. (2026) · Artif. Intell. Medicine · doi:10.48550/arxiv.2603.00028 · Cited by 3
“Similarly, BLURB [19] and recent large-scale evaluations [20] target traditional biomedical tasks.”
Zhu, J., Li, J., Zhao, S., Deng, Y., Miao, Y., & Xu, J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-025-08182-x · Cited by 2
Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores
Amirian, B., Dale, A. S., Kalinin, S. V., & Hattrick-Simpers, J. (2025) · arXiv.org · doi:10.48550/arxiv.2512.01080 · Cited by 3
“Errors should be analyzed to ensure the model meets reliability standards [32].”
Kumari, M., Chauhan, R., Jain, R., & Garg, P. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103902 · Cited by 2
Chen, D., Wei, Y., He, Z., Kuang, G., Ye, C., An, M., Peng, H. Y., Hu, Y. S., Tao, H., & Cheung, K. M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.00588 · Cited by 1
“On the other hand, currently reported evaluation metrics for LLM in health-care were based on question banks to benchmark LLM performance against medical knowledge mastery within specific disease domains or clinical subspecialties[25, 28, 23, 30, 19].”
InfluentialSelf-citation
Metadata
- Authors
- Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., Keloth, V. K., Raja, K., Huang, J., He, H., Lin, F., Du, J., Zhang, R., Zheng, W. J., Adelman, R. A., Lu, Z., & Xu, H.
- Affiliations
- National Institutes of Health (US)
- The University of Texas Health Science Center at Houston (US)
- Yale University (US)
- University of Minnesota (US)
- Corresponding author
- Lu, Zhiyong
- Xu, Hua
- ORCID
- Venue
- Nature Communications
- Published
- 2025-04-06
- Volume
- 16
- Issue
- 1
- Pages
- 3280-3280
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11972378
- OpenAlex
- W4409190866
- Cited by
- 219
- References
- 103
- Citations by year
- 2026: 116 · 2025: 76
- FWCI
- 57.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (1K99LM01402)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, biomedical natural language processing, benchmarking, GPT-4, llamas, fine-tuning, zero-shot learning, few-shot learning, hallucinations, medical question answering, BERT, BART, cost analysis
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
