Benchmarking large language models for biomedical natural language processing applications and recommendations
Open accessAbstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models.
“Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1× · see citing works
“For the tasks where the gold standard is fi xed, e.g., a classi fi cation type or multiple-choice option, we conducted qualitative evaluations on collectively hundreds of thousands of raw outputs of the LLMs (the raw outputs from three LLMs under zero-and one-shot conditions acrossthree benchmarks)…”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 4× · see citing works
“Since the inception of transformers, biomedical language models have adopted their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family includi…”
InfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1× · see citing works
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 1× · see citing works
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1× · see citing works
“This is also supported by the existing studies on medical question answering 38,46 .”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · cited 1× · see citing works
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2× · see citing works
“Nori et al. further found that general-domain LLMs with advanced prompt engineering can achieve the highest accuracy in medical question answering without fi ne-tuning 27 .”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · cited 1× · see citing works
“1.”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 3× · see citing works
“The remarkable advancements have sparked extensive discussions among society, with excitement and concerns alike.”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 1× · see citing works
“Pioneering studies have conducted early experiments on LLMs in the biomedical domain and reported encouraging results.”
Self-citationBioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · cited 1× · see citing works
“It has demonstrated strong capabilities in the tasks that its predecessors fail to do; for instance, GPT-4 passed over 20 academic and professional exams, including the Uniform Bar Exam, SATEvidence-Based Reading& Writing, andMedicalKnowledge Self-Assessment Program 21 .”
BioCreative V CDR task corpus: a resource for chemical disease relation extraction
Li, J., Sun, Y.-P., Johnson, R. J., Sciaky, D., Wei, C.-H., Leaman, R., Davis, A., Mattingly, C., Wiegers, T. C., & Lu, Z.-Y. (2016) · Database J. Biol. Databases Curation · doi:10.1093/database/baw068 · Cited by 1,115 · cited 1× · see citing works
“BC5CDR-chemical 59 andNCBI-disease 60 aremanuallyannotatednamed entity recognition datasets for chemicals and diseases mentioned in biomedical literature, respectively.”
Self-citationNCBI Disease Corpus: A Resource for Disease Name Recognition and Concept Normalization
Dogan, R., Leaman, R., & Lu, Z.-Y. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.12.006 · Cited by 994 · cited 1× · see citing works
“1.”
Self-citationPeng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · cited 1× · see citing works
“The remarkable advancements have sparked extensive discussions among society, with excitement and concerns alike.”
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1× · see citing works
“1.”
Self-citationA Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658 · cited 1× · see citing works
“1.”
Text and Code Embeddings by Contrastive Pre-TrainingPreprint
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q.-M., Tezak, N., Kim, J. W., Hallacy, C., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.10005 · Cited by 657 · cited 2× · see citing works
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · cited 2× · see citing works
“5 zero-shot and GPT-4 zero-shot, respectively (0.4988 and 0.7156 in our study, respectively) 38 .”
InfluentialBioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455 · cited 1× · see citing works
“Moreover, the enhancement of LLMs is achieved through reinforcement learning with human feedback, thereby aligning text generation with human preferences 19 .”
Self-citationDatabase resources of the National Center for Biotechnology Information in 2023
Sayers, E. W., Bolton, E., Brister, J. R., Canese, K., Chan, J., Comeau, D. C., Farrell, C. M., Feldgarden, M., Fine, A. M., Funk, K., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac1032 · Cited by 436 · cited 1× · see citing works
“…their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family including BioBERT and PubMedBERT 9 – 11 , (2)…”
Self-citationOpportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · cited 1× · see citing works
“For instance, few studies evaluated more advanced closed-source LLMs such as GPT-4, LLM representatives from the general domain such as LLaMA 22 , and bio-medical domain-speci fi c LLMs such as PMC-LLaMA 34 .”
Self-citationLarge language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · cited 1× · see citing works
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · cited 1× · see citing works
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Large Language Models: A Survey
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., & Gao, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06196 · Cited by 325 · cited 1× · see citing works
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · cited 1× · see citing works
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · cited 2× · see citing works
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Self-citationPMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · cited 5× · see citing works
“An accuracyof 0.7790 on PubMedQA wasalsoreported for the fi ne-tuned PMC LLaMA 13B (combined multiple question answering datasets for fi ne-tuning) 34 ; our study also reported a similar accuracy of 0.7680 using the PubMedQA training set only.”
InfluentialDeep Bidirectional Language-Knowledge Graph PretrainingPreprint
Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., & Leskovec, J. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.09338 · Cited by 300 · see citing works
BioSentVec: creating sentence embeddings for biomedical texts
Chen, Q.-Y., Peng, Y., & Lu, Z.-Y. (2018) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi.2019.8904728 · Cited by 241 · cited 1× · see citing works
“networks such as BioWordVec and BioSentVec 4,7,8 .”
Self-citationThinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
Gutierrez, B. J., McNeal, N., Washington, C., Chen, Y., Li, L., Sun, H., & Su, Y. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.08410 · Cited by 206 · cited 1×
“In contrast, LLMs may have the advantage when minimal manually labeled instances are available, and they do not require fi ne-tuning or retraining for every new task through zero/few-shot learning.”
Fine-tuning large neural language models for biomedical natural language processing
Tinn, R., Cheng, H., Gu, Y., Usuyama, N., Liu, X.-D., Naumann, T., Gao, J.-F., & Poon, H. (2021) · Patterns · doi:10.1016/j.patter.2023.100729 · Cited by 205 · cited 3× · see citing works
“In the biomedical domain, a primary challenge is the limited availability of labeled datasets, which have a signi fi cantly lower scale than those in the general domain (e.g., a biomedical sentence similarity dataset only has 100 labeled instances in total 31 ) 32,33 .”
InfluentialSciFive: a text-to-text transformer model for biomedical literaturePreprint
Phan, L., Anibal, J., Tran, H., Chanana, S., Bahadroglu, E., Peltekian, A., & Altan-Bonnet, G. (2021) · arXiv.org · doi:10.48550/arxiv.2106.03598 · Cited by 203 · cited 1× · see citing works
“It has demonstrated strong capabilities in the tasks that its predecessors fail to do; for instance, GPT-4 passed over 20 academic and professional exams, including the Uniform Bar Exam, SATEvidence-Based Reading& Writing, andMedicalKnowledge Self-Assessment Program 21 .”
BioBART: Pretraining and Evaluation of A Biomedical Generative Language ModelPreprint
Yuan, H., Yuan, Z., Gan, R., Zhang, J., Xie, Y., & Yu, S. (2022) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2204.03905 · Cited by 200 · cited 1× · see citing works
“…, (2) decoder-based, generative language models using the decoder from the transformer architecture such as the generative pre-trained transformer (GPT) family including BioGPT and BioMedLM 12,13 , and (3) encoder-decoder-based, using both encoders and decoders such as BioBART and Sci fi ve 14,15 .”
Wei, C.-H., Peng, Y., Leaman, R., Davis, A. P., Mattingly, C. J., Li, J., Wiegers, T. C., & Lu, Z. (2016) · Database · doi:10.1093/database/baw032 · Cited by 198 · see citing works
Self-citationCan Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · cited 3× · see citing works
“5, which may not provide a full spectrum of representative LLMs from different categories.”
Automatic semantic classification of scientific literature according to the hallmarks of cancer
Baker, S., Silins, I., Guo, Y., Ali, I., Högberg, J., Stenius, U., & Korhonen, A. (2016) · Bioinform. · doi:10.1093/bioinformatics/btv585 · Cited by 166 · cited 1× · see citing works
“1.”
BIOSSES: a semantic sentence similarity estimation system for the biomedical domain
Soğancıoğlu, G., Öztürk, H., & Özgür, A. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx238 · Cited by 159 · cited 1× · see citing works
“Arguably, existing transformer models have demonstrated satisfactory performance in extractive tasks, while generative tasks remain a challenge in terms of achieving similar levels of pro fi ciency.”
ML-Net: multi-label classification of biomedical texts with deep neural networks
Du, J., Chen, Q.-Y., Peng, Y., Xiang, Y., Tao, C., & Lu, Z.-Y. (2018) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocz085 · Cited by 148 · cited 1× · see citing works
“1.”
Self-citationParagraph-level Simplification of Medical Texts
Devaraj, A., Marshall, I., Wallace, B. C., & Li, J. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.05767 · Cited by 144
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2022) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 128 · cited 2× · see citing works
“1.”
Self-citationAn Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics
Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2022) · ACM Computing Surveys · doi:10.1145/3545176 · Cited by 124 · see citing works
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 1× · see citing works
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding TasksPreprint
Chen, X., Ye, J., Zu, C., Xu, N., Zheng, R., Peng, M.-L., Zhou, J., Gui, T., Zhang, Q., & Huang, X. (2023) · arXiv.org · doi:10.48550/arxiv.2303.00293 · Cited by 116 · cited 3× · see citing works
“The MedQA dataset is collected from questions in the United States Medical License Examination (USMLE), whereeach instancecontains aquestion (usually a patient description) and fi ve answer choices (e.g., fi ve potential diagnoses) 66 .”
Advancing entity recognition in biomedicine via instruction tuning of large language models
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae163 · Cited by 104 · cited 2× · see citing works
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Self-citationBiocuration: Distilling data into knowledge
International Society for Biocuration (2018) · PLoS Biology · doi:10.1371/journal.pbio.2002846 · Cited by 104 · cited 1× · see citing works
“1.”
MSˆ2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.594 · Cited by 98 · cited 2× · see citing works
“1.”
Blake, C. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.11.001 · Cited by 91 · cited 1× · see citing works
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Labrak, Y., Rouvier, M., & Dufour, R. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2307.12114 · Cited by 76 · cited 2×
“Recall that we categorized inconsistencies, missing information, and hallucinations on the tasks where the gold standard is a fi xed classi fi cation type or a multiple-choice option.”
InfluentialDoğan, R. I., Kim, S., Chatr-aryamontri, A., Wei, C.-H., Comeau, D. C., Antunes, R., Matos, S., Chen, Q.-Y., Elangovan, A., Panyam, N. C., et al. (2019) · Database J. Biol. Databases Curation · doi:10.1093/database/bay147 · Cited by 72 · see citing works
Self-citationOptimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning
Zhang, S., Cheng, H., Gao, J.-F., & Poon, H. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2208.14565 · Cited by 67
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65 · cited 4× · see citing works
“Other studies reported similar fi ndings regarding the low correlation between automatic and manual evaluations 35,47 .”
InfluentialDo We Still Need Clinical Language Models?
Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., & Alsentzer, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08091 · Cited by 62 · see citing works
Chen, Q., Lee, K., Yan, S., Kim, S., Wei, C.-H., & Lu, Z. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007617 · Cited by 62 · cited 1× · see citing works
“networks such as BioWordVec and BioSentVec 4,7,8 .”
Self-citationOverview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles
Goldsack, T., Scarton, C., Shardlow, M., & Lin, C. (2024) · Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/2024.bionlp-1.10 · Cited by 61 · cited 1×
“1.”
A survey of automated methods for biomedical text simplification
Ondov, B. D., Attal, K., & Demner-Fushman, D. (2022) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocac149 · Cited by 60 · see citing works
LitSense: making sense of biomedical literature at sentence level
Allot, A., Chen, Q., Kim, S., Alvarez, R. V., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz289 · Cited by 57 · cited 1× · see citing works
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Self-citationExploring the Boundaries of GPT-4 in Radiology
Liu, Q.-C., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M., Tinn, R., Sharma, H., Pérez-García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.14573 · Cited by 55 · cited 1×
“There are several primary limitations.”
Readability Controllable Biomedical Document SummarizationPreprint
Luo, Z.-H., Xie, Q., & Ananiadou, S. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.04705 · Cited by 51 · see citing works
Self-citationChen, Q.-Y., Allot, A., Leaman, R., Islamaj, R., Du, J.-C., Fang, L., Kai, W., Xu, S., Zhang, Y., Bagherzadeh, P., et al. (2022) · Database J. Biol. Databases Curation · doi:10.48550/arxiv.2204.09781 · Cited by 48 · cited 1×
“1.”
Self-citationRehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y., Özgür, A., & Hur, J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17728 · Cited by 47 · cited 1× · see citing works
“In contrast, LLMs may have the advantage when minimal manually labeled instances are available, and they do not require fi ne-tuning or retraining for every new task through zero/few-shot learning.”
Chen, Q., Du, J., Allot, A., & Lu, Z. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3173562 · Cited by 43 · see citing works
Self-citationStricker, J., Chasiotis, A., Kerwer, M., & Günther, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231160 · Cited by 37 · see citing works
GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization
Xie, Q., Huang, J.-M., Saha, T., & Ananiadou, S. (2022) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2208.09982 · Cited by 35 · cited 1×
“BC5CDR-chemical 59 andNCBI-disease 60 aremanuallyannotatednamed entity recognition datasets for chemicals and diseases mentioned in biomedical literature, respectively.”
Self-citationLitCovid in 2022: an information resource for the COVID-19 literature
Chen, Q.-Y., Allot, A., Leaman, R., Wei, C.-H., Aghaarabi, E., Guerrerio, J. J., Xu, L., & Lu, Z.-Y. (2022) · Nucleic Acids Res. · doi:10.48550/arxiv.2209.13428 · Cited by 34 · cited 1×
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Self-citationWang, L., Bi, W., Zhao, S., Ma, Y., Lv, L., Meng, C., Fu, J., & Lv, H. (2024) · JMIR Formative Research · doi:10.2196/53216 · Cited by 33 · cited 1× · see citing works
“To date, prompt design remains an open research problem 73 – 75 .”
Prompt Tuning in Biomedical Relation Extraction
He, J., Li, F., Li, J., Hu, X., Nian, Y., Xiang, Y., Wang, J., Wei, Q., Li, Y., Xu, H., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00162-9 · Cited by 32 · see citing works
Self-citationEvaluating GPT and BERT models for protein–protein interaction identification in biomedical text
Rehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y., Özgür, A., & Hur, J. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae133 · Cited by 32 · see citing works
Li, X., Rousseau, J. F., Ding, Y., Song, M., & Lu, W. (2020) · JMIR Medical Informatics · doi:10.2196/16739 · Cited by 32 · cited 1× · see citing works
“Relation extraction involves identifying the relationships between entities, which is important for drug repurposing and knowledge discovery 61 .”
LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation
Bishop, J. A., Ananiadou, S., & Xie, Q. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2309.12455 · Cited by 30
Self-citationScaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology
Wong, C., Zhang, S., Gu, Y., Moung, C., Abel, J., Usuyama, N., Weerasinghe, R., Piening, B., Naumann, T., Bifulco, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02180 · Cited by 28 · cited 1× · see citing works
“5 and GPT-4 for end-to-end clinical trial matching, handling complex eligibility criteria, and extracting complex matching logic 25 .”
Bioformer: an efficient transformer language model for biomedical text miningPreprint
Li, F., Chen, Q., Wei, C.-H., Lu, Z., & Wang, K. (2023) · PubMed · doi:10.48550/arxiv.2302.01588 · Cited by 28 · cited 2× · see citing works
“5 and GPT-4 — has marked a milestone in generative arti fi cial intelligence.”
Self-citationHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2023) · arXiv · doi:10.48550/arxiv.2310.05694 · Cited by 27 · see citing works
Dr. LLaMA: Improving Small Language Models on PubMedQA via Generative Data Augmentation
Guo, Z., Wang, P.-Q., Wang, Y., & Yu, S.-D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.07804 · Cited by 27
Correction to: Evaluating the ChatGPT family of models for biomedical reasoning and classification
(2024) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocae083 · Cited by 26
A Comprehensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPTPreprint
Chen, Q., Sun, H., Liu, H., Jiang, Y., Ran, T., Jin, X., Xiao, X.-L., Lin, Z.-M., Niu, Z., & Chen, H. (2023) · bioRxiv · doi:10.1101/2023.04.19.537463 · Cited by 24 · see citing works
Self-citationChen, Q., Rankine, A., Peng, Y., Aghaarabi, E., & Lu, Z. (2021) · JMIR Medical Informatics · doi:10.2196/27386 · Cited by 21 · cited 1× · see citing works
“Arguably, existing transformer models have demonstrated satisfactory performance in extractive tasks, while generative tasks remain a challenge in terms of achieving similar levels of pro fi ciency.”
Self-citationFlores, L. J. Y., Huang, H., Shi, K., Chheang, S., & Cohan, A. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.11191 · Cited by 19
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.-Y., Chen, H., & Hu, X. (2024) · arXiv · doi:10.48550/arxiv.2401.01325 · Cited by 16 · cited 1× · see citing works
“Under the fi ne-tuning setting, the results also indicate that PMC LLaMA 13B, as a continuously pretrained biomedical domain-speci fi c LLM, did not achieve an overall higher performance than LLaMA 2 13B. Fine-tuned LLaMA 2 13B had better performance than that of PMC LLaMA 13B in 10 out of the 12…”
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., & Yang, M. (2024) · arXiv · doi:10.48550/arxiv.2402.13753 · Cited by 15 · cited 1× · see citing works
“Under the fi ne-tuning setting, the results also indicate that PMC LLaMA 13B, as a continuously pretrained biomedical domain-speci fi c LLM, did not achieve an overall higher performance than LLaMA 2 13B. Fine-tuned LLaMA 2 13B had better performance than that of PMC LLaMA 13B in 10 out of the 12…”
Su, Y.-S., Wang, M., Wang, P., Zheng, C.-H., Liu, Y.-S., & Zeng, X.-X. (2022) · Briefings Bioinform. · doi:10.1093/bib/bbac342 · Cited by 14 · cited 2× · see citing works
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Biomedical Question Answering: A Survey of Methods and Datasets
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds50568.2020.9268742 · Cited by 12 · cited 1× · see citing works
“BiomedicalQA datasets such as MedQA and PubMedQA have been widely used in the evaluation of language models 65 .”
Comprehensively identifying Long Covid articles with human-in-the-loop machine learning
Leaman, R., Islamaj, R., Allot, A., Chen, Q.-Y., Wilbur, W., & Lu, Z.-Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100659 · Cited by 5 · see citing works
Self-citationChemical-Protein Relation Extraction with Pre-trained Prompt Tuning
He, J.-P., Li, F., Hu, X.-Y., Li, J., Yi, N., Wang, J.-Q., Xiang, Y., Wei, Q., Xu, H., & Tao, C. (2022) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi54592.2022.00120 · Cited by 5 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“5 and GPT-4 for end-to-end clinical trial matching, handling complex eligibility criteria, and extracting complex matching logic 25 .”
Flagged first, then by citations
12 of the 103 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- [23]Jiang, A. Q. et al. Mixtral of experts arXiv preprint arXiv:2401.04088, 2024. · arXiv
- [24]Lee, P, Goldberg, C. & Kohane, I. The AI revolution in medicine: GPT-4 and beyond (Pearson, 2023).
- [30]Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. & Daneshjou, R. Large language models in medicine: the potentials and pitfalls: a narrative review. Ann. Intern. Med. 177, 210–220 (2024). 38285984
- [55]Krallinger, M. et al. Overview of the BioCreative VI chemical-protein interaction Track. In Proc. of the sixth BioCreative challenge evaluation workshop Vol. 1, 141–146 (2017).
- [62]Segura-Bedmar, I., Martínez, P. & Herrero-Zazo, M. Semeval-2013 task 9: extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proc. Seventh International Workshop on Semantic Evaluation (SemEval 2013) 341–350 (Association for Computational Linguistics, 2013).
- [85]Mingliang, D., Jijun, T. & Fei, G. Document-level DDI relation extraction with document-entity embedding. pp. 392–397.
- [94]Wang, L. L, DeYoung, J. & Wallace, B. Overview of MSLR2022: A shared task on multidocument summarization for literature reviews. In Proc. Third Workshop on Scholarly Document Processing 175–180 (Association for Computational Linguistics, 2022).
Cited by
Sorted by citations
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
“Chen et al. [105] performed a pilot study to establish the baselines of using GPT-3.”
BackgroundSelf-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312 · see citing works
“Neverthe-less, current systematic evaluations in biomedicine typically focus on the vanilla LLMs without RAG (Chen et al., 2023a; Nori et al., 2023a).”
BackgroundSelf-citationA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Self-citationSivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2024) · JMIR Medical Informatics · doi:10.2196/55318 · Cited by 253 · see citing works
“For instance, Chen et al [9] used a fixed template as the prompt to measure the performance of LLMs on biomedical NLP tasks but did not investigate different kinds of prompting methods.”
BackgroundMe-LLaMA: Foundation Large Language Models for Medical ApplicationsPreprint
Xie, Q., Chen, Q., Chen, A., Cheng, P., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V. K., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4240043/v1 · Cited by 62 · see citing works
“29,30 When compared with other NLP tasks with higher performance, we noticed that one of the main reasons for low performance is that LLMs’ responses often lacked the conciseness and precision expected, with instances of missing outputs noted.”
BackgroundSelf-citationTaiyi: a bilingual fine-tuned large language model for diverse biomedical tasks
Luo, L., Ning, J., Zhao, Y., Wang, Z., Ding, Z., Chen, P., Fu, W., Han, Q., Xu, G., Qiu, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae037 · Cited by 89 · see citing works
Self-citationAdvancing entity recognition in biomedicine via instruction tuning of large language models
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae163 · Cited by 104 · see citing works
“Previously, in a pilot study (Chen et al. 2023) we evaluated the performance of GPT-3.”
BackgroundMethodInfluentialSelf-citationMedical foundation large language models for comprehensive text analysis and beyond
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01533-1 · Cited by 74 · see citing works
“Despitethesestrengths,weobservedcertainchallengesinspeci fi ctasks, such as NER and RE 14,15 , whereeven advanced modelslike GPT-4 exhibited low performance.”
Self-citationUnveiling LLM Evaluation Focused on Metrics: Challenges and SolutionsPreprint
Hu, T., & Zhou, X.-H. (2024) · arXiv.org · doi:10.48550/arxiv.2404.09135 · Cited by 66 · see citing works
“Wang et al. (2023) highlighted the promising applications of LLMs in addressing biomedical questions, while Chen et al. (2023) focused on evaluating biomedical LLMs with respect to benchmarks and summarization capabilities.”
BackgroundIdentifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models
Shyr, C., Hu, Y., Bastarache, L., Cheng, A. C., Hamid, R., Harris, P. A., & Xu, H. (2023) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00155-0 · Cited by 63 · see citing works
Self-citationZhang, X., Talukdar, N., Vemulapalli, S., Ahn, S., Wang, J., Han, M., Murtaza, S. M. B., Leshchiner, D. R., Dave, A. A., Joseph, D. F., et al. (2024) · medRxiv · doi:10.1101/2024.02.07.24302444 · Cited by 40 · see citing works
“5 and GPT-4 showed promise in biomedical semantic similarity and reasoning tasks but are less effective in information extraction and classification 10 .”
BackgroundLarge language models for drug discovery and development
Zheng, Y., Koh, H. Y., Ju, J., Yang, M., May, L. T., Webb, G. I., Li, L., Pan, S., & Church, G. M. (2025) · Patterns · doi:10.1016/j.patter.2025.101346 · Cited by 89 · see citing works
Sushil, M., Zack, T. I., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., Lituiev, D., & Butte, A. J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae146 · Cited by 62 · see citing works
“We found that even in zero-shot setups, the GPT-4 model performs as well as or 35,36 Similar small datasets are common in medical informatics studies since domain expertise is frequently required for reliably annotating clinical notes, making the process time-consuming and difficult to scale.”
BackgroundPark, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42 · see citing works
An evaluation of GPT models for phenotype concept recognition
Groza, T., Caufield, H., Gration, D., Baynam, G. S., Haendel, M. A., Robinson, P. N., Mungall, C., & Reese, J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02439-w · Cited by 39 · see citing works
“The boundary detection step proved to be challenging to evaluate accurately with the results produced by the OpenAI GPT models - aspect documented also by Chen at [19].”
MethodTang, X.-R., Shao, Y., Sohn, J., Chen, J.-P., Zhang, J., Xiang, J.-Y., Wu, F., Zhao, Y., Wu, C.-L., Shi, W.-Q., et al. (2025) · Patterns · doi:10.1016/j.patter.2026.101610 · Cited by 44
Information extraction from clinical notes: are we ready to switch to large language models?
Hu, Y., Zuo, X., Zhou, Y.-J., Peng, X.-Q., Huang, J.-M., Keloth, V., Zhang, V., Weng, R.-L., Shyr, C., Chen, Q.-Y., et al. (2024) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocaf213 · Cited by 44 · see citing works
“23,24 However, their effectiveness in traditional IE remains mixed.”
Self-citationSivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2309.08008 · Cited by 42
“Chen et al.(8) used a fixed template as the prompt to measure the performance of LLMs on biomedical NLP tasks, but did not investigate different kinds of prompting methods.”
BackgroundMansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 44 · see citing works
“While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines Chen et al., [93].”
Du, X.-S., Zhou, Z., Wang, Y.-F., Chuang, Y.-W., Li, Y.-M., Yang, R., Zhang, W., Wang, X., Chen, X.-Y., Guan, H., et al. (2024) · medRxiv · doi:10.1101/2024.08.11.24311828 · Cited by 36 · see citing works
“27,65 This may be due to the fact that prompting strategies can increase the length of a prompt, and a longer prompt might negatively impact the LLM’s performance.”
BackgroundPeng, X.-Q., Qian, L.-F., Wang, Y., Xiang, R.-Y., He, Y.-R., Ren, Y., Jiang, M.-Y., Zhang, V., Guo, Y., Zhao, J., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2026.acl-long.770 · Cited by 26
“LLMs have demonstrated remarkable prowess in various tasks (Zhao et al., 2025; Chen et al., 2025b), yet their applications to finance (Liu et al., 2023b; Xie et al., 2023; Wu et al., 2023) require the continual push of the limits of model capabilities (Xiong et al., 2025).”
Self-citationBRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice TextPreprint
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y.-X., Carducci, V., Wyss, R., Desai, R., Alsentzer, E., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.19467 · Cited by 25 · see citing works
“20,21 These standardized question sets offer a basic and rapid assessment, such as the GPT-4 22 and Med-PaLM 2 23 , achieving expert-level scores on the USMLE.”
Keloth, V. K., Selek, S., Chen, Q., Gilman, C., Fu, S., Dang, Y., Chen, X., Hu, X., Zhou, Y., He, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01645-8 · Cited by 24 · see citing works
“Our extensive prior work comparing zero/few-shot and fi ne-tuned LLaMA models have demonstrated the superior performance of fi ne-tuned models 51 – 54 .”
Self-citationde Oliveira, R., Garber, M., Gwinnutt, J. M., Rashidi, E., Hwang, J.-H. (., Gilmour, W., Nanavati, J., Abidine, K. Z. E., & Mack, C. D. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf058 · Cited by 17 · see citing works
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 23 · see citing works
Self-citationLarge Language Models Vote: Prompting for Rare Disease Identification
Oniani, D., Hilsman, J., Dong, H., Gao, F., Verma, S., & Wang, Y.-S. (2023) · arXiv.org · doi:10.48550/arxiv.2308.12890 · Cited by 19
“Recently, studies have shown that using formats, such as XML, can work well for model evaluation (Chen et al., 2023).”
BackgroundLiu, Y., Duran, E., Bruce, A., Yildiz, B., Severiano, B. M., Ibrahim, I. A., Rispler, J., Martell, C., & Rougieux, F. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126636 · Cited by 18 · see citing works
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18 · see citing works
“In the biomedical domain, fine-tuned BioGPT 23 has shown superior performance to BERT-based models in document classification, while the performance of the GPT models in zero-or one-shot settings has been found to trail that of fine-tuned BERT models 53 .”
Method(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18 · see citing works
“At the same time, our results with GPT-4 are consistent with other comparisons of GPT-4 in-context learning with fine-tuned models for text classification 53 .”
MethodResultLiu, S.-R., McCoy, A. B., Chen, Q.-Y., & Wright, A. T. (2025) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2025.106104 · Cited by 16 · see citing works
Self-citationZhu, Y., Gao, J.-Y., Wang, Z.-X., Liao, W., Zheng, X.-C., Liang, L., Bernabeu, M., Wang, Y.-S., Yu, L.-Q., Pan, C.-W., et al. (2024) · npj Digit. Medicine · doi:10.1038/s41746-026-02539-z · Cited by 16 · see citing works
Self-citationMultiFinBen: A Multilingual, Multimodal, and Difficulty-Aware Benchmark for Financial LLM Evaluation
Peng, X.-Q., Qian, L.-F., Wang, Y., Xiang, R.-Y., He, Y.-R., Ren, Y., Jiang, M.-Y., Zhao, J., He, H., Han, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.14028 · Cited by 16
“LLMs have demonstrated remarkable prowess in various tasks (Zhao et al., 2025; Chen et al., 2025b), yet their applications to finance (Liu et al., 2023b; Xie et al., 2023; Wu et al., 2023) require the continual push of the limits of model capabilities (Xiong et al., 2025).”
Self-citationKim, H., Sohn, J., Gilson, A., Cochran-Caggiano, N. C., Applebaum, S., Jin, H., Park, S., Park, Y., Park, J., Choi, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.06738 · Cited by 15 · see citing works
Self-citationPeng, C., Zhang, K., Lyu, M., Liu, H., Sun, L., & Wu, Y. (2025) · Journal of Biomedical Informatics · doi:10.48550/arxiv.2505.17436 · Cited by 15
RAG-Enhanced Open SLMs for Hypertension Management Chatbots
Aguzzi, G., Magnini, M., Farahmand, A., Ferretti, S., Pengo, M. F., & Montagna, S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02297-7 · Cited by 16 · see citing works
“For instance, [ 16 ] presents a SOTA fine-tuning approach that, for the open-source LLMs evalu - ated, outperforms zero-and few-shot approaches in the QA task.”
LLM-based UAV Path Planning for Autonomous and Adaptive Industry Systems
Xiao, W.-J., Shi, C., Chen, M.-J., Vasilakos, A. V., Chen, M., & Farouk, A. (2025) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3774655 · Cited by 13
“However, surprisingly the fine-tuned Mistral-7B performs very poorly, which on the one hand highlights the complexity of the UAV swarm path planning task, and on the other hand confirms the existence of a performance gap between different LLM models [7].”
BioContextAI is a community hub for agentic biomedical systems
Kuehl, M., Schaub, D. P., Carli, F., Heumos, L., Hellmig, M., Fernández-Zapata, C., Kaiser, N., Schaul, J., Kulaga, A., Usanov, N., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02900-9 · Cited by 13 · see citing works
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 14 · see citing works
Multi-agent systems for clinical decision support: A systematic review
Silveira, A. L., Righi, R. D. R., & da Costa, C. A. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114447 · Cited by 11 · see citing works
Language models for drug–drug interactions: current applications, pitfalls, and future directions
Meslamani, A. Z. A., & Hajal, A. A. (2025) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2025.2551724 · Cited by 12 · see citing works
“For LLMs to be fully accepted in clinical pharmacology processes, they must improve openness and dependability [119,120].”
Cherif, B., Bisztray, T., Dubniczky, R. A., Aldahmani, A., Alshehhi, S., & Tihanyi, N. (2025) · International Conference on Neural Information Processing · doi:10.48550/arxiv.2505.19973 · Cited by 12
“LLMs hold immense potential for various fields, including cybersecurity [31], software engineering [22], biomedicine [4] or law [6], which has sparked calls for privacy-aware, reliability-oriented, and domain-tailored benchmarks [35].”
DiMB-RE: mining the scientific literature for diet-microbiome associations
Hong, G., Hindle, V. K., Veasley, N. M., Holscher, H., & Kilicoglu, H. (2024) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocaf054 · Cited by 12 · see citing works
“For RE, in both cases, relation types must match.”
BioMamba: Domain-Adaptive Biomedical Language Models
Yue, L., Zhu, M.-Z., Xing, S.-X., Cao, Y.-N., Wang, Y.-B., Shan, S., Liu, J.-F., Chenthamarakshan, V., Pan, S.-W., Das, P., et al. (2024) · Health Data Science · doi:10.34133/hds.0454 · Cited by 12
Elbiach, O., Grissette, H., & Nfaoui, E. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107362 · Cited by 11 · see citing works
van Velzen, M., van der Willigen, R. F., de Beer, V. D., de Graaf-Waar, H. D., Janssen, E. R. C., van Leeuwen, S., van der Willigen, M. F., van der Willigen, M. J., Renardus, G., Maaroufi, R. E., et al. (2025) · Frontiers Artif. Intell. · doi:10.3389/frai.2025.1644084 · Cited by 11 · see citing works
Large language models for spreading dynamics in complex systems
Jiang, S., Ren, H., Gao, Y., Zhang, Y., Li, Q., Xiao, D., Fan, J., Tang, R., & Wang, W. (2026) · Physics Reports · doi:10.1016/j.physrep.2026.01.006 · Cited by 9 · see citing works
Large Language Models in Bio-Ontology Research: A Review
Manda, P. (2025) · Bioengineering · doi:10.3390/bioengineering12111260 · Cited by 7 · see citing works
“Finally, systematic benchmarking efforts [77,78] provide critical evaluations of LLM performance on biomedical tasks, underscoring both the promise and variability of current approaches.”
InfluentialYang, E.-W., Waldrup, B., & Velazquez-Villarreal, E. (2025) · medRxiv · doi:10.1101/2025.05.20.25327967 · Cited by 10
“The emergence of artificial intelligence (AI), especially advancements in large language models (LLMs), has paved the way for conversational tools that convert natural language input into executable data analysis pipelines [23–24].”
Enhancing biomedical relation extraction with directionality
Lai, P., Wei, C.-H., Tian, S., Leaman, R., & Lu, Z. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf226 · Cited by 10 · see citing works
“Recent studies (Chen et al. 2025) indicate that Llama 3 (AI@Meta 2024) exhibits performance comparable to GPT-3.”
Self-citationSushil, M., Zack, T., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., & Butte, A. (2024) · Research Square · doi:10.21203/rs.3.rs-3914899/v1 · Cited by 10 · see citing works
“Finally, we often find that skin is one of the specimens examined.”
BackgroundATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue
Cao, R.-K., Bai, S., Yao, F.-G., Dong, L., Xu, J., & Xiao, L.-Q. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2603.02216 · Cited by 9
“…[6], Qwen3 Yang et al. [41], and DeepSeek-R1 Guo et al. [11] have demonstrated exceptional capabilities across a range of natural language processing tasks, including open-domain question answering, dialogue generation, and code generation, continuously pushing the boundaries of AI performance [4].”
Causal graph neural networks for healthcare
Mesinovic, M., Buhlan, M., & Zhu, T. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01742-3 · Cited by 9
“…biomedical natural language processing tasks demonstrates that while LLMs excel at reasoning-intensive question answering, they under-perform fine-tuned domain-specific models in information extraction, with qualitative analyses revealing prevalent missing information and inconsistent outputs [84].”
Syahputri, I. W., Budiardjo, E. K., & Putra, P. H. H. (2025) · Applied Informatics · doi:10.3390/ai6090206 · Cited by 9 · see citing works
Entry-level guide to the use of large language models for medical researchPreprint
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z.-Z., Yang, Y.-F., Wang, Z., Xiong, G.-Z., Lai, P.-T., Zhu, Q.-Q., et al. (2024) · PubMed · doi:10.48550/arxiv.2410.18856 · Cited by 9 · see citing works
“For example, LLMs are better for tasks that require medical knowledge and clinical reasoning, but do not often outperform fine-tuned BERT models 63 in simpler tasks such as structurization 64 .”
Self-citationEnhancing Large Language Models with Domain-specific Retrieval Augment Generation
Gilson, A., Ai, X.-G., Arunachalam, T., Chen, Z., Cheong, K. X., Dave, A. D., Duic, C., Kibe, M., Kaminaka, A., Prasad, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2409.13902 · Cited by 9
Self-citationDong, Y.-N., Zhang, Z., Zhi, Y., Li, X.-Y., Guo, T., He, L., Zhao, S., Yang, X.-L., Tang, J.-T., Zhong, W., et al. (2025) · JHEP Reports · doi:10.1016/j.jhepr.2025.101579 · Cited by 8 · see citing works
Multi-Modal Topology-Aware Graph Neural Network for Robust Chemical–Protein Interaction Prediction
Wang, J. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26178666 · Cited by 11 · see citing works
Shao, C., Snyder, D., Li, C., Gu, B., Ngan, K., Yang, C.-T., Wu, J., Wyss, R., Lin, K., & Yang, J. (2025) · Journal of Clinical Epidemiology · doi:10.48550/arxiv.2506.11137 · Cited by 8
“- Macro Precision - Macro Recall - Macro F1”
OpenAI ChatGPT: A Potential Medical Application
Chaddad, A., Jiang, Y., & He, C. (2023) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom56612.2023.10472387 · Cited by 8 · see citing works
“…American College of Gastroenterology Self-Assessment Tests [18] 65.1% 62.4% PSITE 2021 Exam [19] 51.2% 75.3% PubMedQA [20] 51.6% 80.4% NCBI-disease [16] 24.05% 56.73% MedMCQA [13] 50.08% 73.42% SANS ABNS Self-Assessment Exam [21] 73.4% 83.4% BC5CDR: BioCreative V CDR; SCE: Specialty Certificate…”
InfluentialEmerging trends of recommender system for e-commerce: a comprehensive review
Rajpoot, C. S., Tiwari, V., & Vishwakarma, S. K. (2026) · Discover Computing · doi:10.1007/s10791-026-09923-z · Cited by 7 · see citing works
“Text summarization techniques–both extractive and abstractive address this issue by condensing large volumes of textual data into concise, meaningful summaries that retain essential semantic information [119].”
Ghosh, A., Jain, R., Jhangra, A., Saha, S., & Jatowt, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70045 · Cited by 5 · see citing works
“…RoBERTa for entailment and CUI linking was reported, which propagates into higher downstream ROUGE when these representations seed the summarizer (Chen et al. 2025 (Sundrelingam et al. 2025) processes each note in about 0.5 s and posts an F1 of 0.90 on both Canadian and n2c2 discharge-summary…”
InfluentialSholehrasa, H., Ghanaatian, A., Caragea, D., Tell, L. A., Riviere, J. E., & Jaberi-Douraki, M. (2025) · IEEE International Conference on Tools with Artificial Intelligence · doi:10.1109/ictai66417.2025.00051 · Cited by 7 · see citing works
“1, Step 8) using a 5-shot prompting [33] (Fig.”
Benchmarking GPT-5 for biomedical natural language processingPreprint
Hou, Y., Zhan, Z.-F., Zeng, M., Wu, Y.-F., Zhou, S., & Zhang, R. (2025) · arXiv.org · doi:10.48550/arxiv.2509.04462 · Cited by 7 · see citing works
“5, and LLaMA-2-13B across five text-based task families originally benchmarked by Chen et al. 16 : named entity recognition (NER), relation extraction (RE), multi-label document classification, text summarization, and text simplification.”
InfluentialSelf-citationChen, H.-B., Alfred, M. C., & Cohen, E. (2024) · JMIR Medical Informatics · doi:10.2196/68955 · Cited by 7 · see citing works
“While our findings highlight the promise of open-source LLMs in data-scarce settings, the exclusion of proprietary models limits the generalizability of our findings.”
Adam, T., Abosabie, S. A. S., Dittmer, M., Wolf, E., Abosabie, S., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10093 · Cited by 6 · see citing works
Huang, X.-H., Xiao, M., Qin, C., Long, Q.-Q., Chen, J.-M., Zhou, Y.-C., & Zhu, H.-S. (2026) · Knowledge Discovery and Data Mining · doi:10.1145/3770855.3817489 · Cited by 6 · see citing works
Latency-Aware Benchmarking of Large Language Models for Natural-Language Robot Navigation in ROS 2
Das, M., Hussain, Z., & Nawaz, M. (2026) · Sensors · doi:10.3390/s26020608 · Cited by 5 · see citing works
“This metric is relevant for large-scale or cost-sensitive deployments [21]. https://doi.org/10.3390/s26020608 2.7.”
InfluentialApplication and Prospects of Large Language Models in Small-Molecule Drug Discovery
Ma, J., Liu, J., Xu, D., Song, X., & Zhang, Z. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.5c04083 · Cited by 9 · see citing works
Zhan, Z.-F., Zhou, S., Zeng, M., Yu, K., Song, M.-J., Chen, X., Wang, J., Hou, Y., & Zhang, R. (2025) · arXiv.org · doi:10.48550/arxiv.2509.04534 · Cited by 6
Self-citationLi, X., Zhao, D., Meng, J.-N., & Lin, H. (2025) · Expert systems with applications · doi:10.1016/j.eswa.2025.128342 · Cited by 6
LLM-Based Agents for Competitive Landscape Mapping in Drug Asset Due Diligence
Vinogradova, A., Vinogradov, V., Radkevich, D., Yasny, I., Kobyzev, D., Izmailov, I., Yanchanka, K., Doronin, R., & Doronichev, A. (2025) · arXiv.org · doi:10.48550/arxiv.2508.16571 · Cited by 6
“Moreover, recent studies report that general-purpose, frontier LLMs can be competitive with—or even outperform—task-specific fine-tuned models on certain pharma-adjacent evaluations (Chen et al. 2025; McDuff et al. 2025).”
Yu, J., Tang, Y., Feng, K., Liang, L., Zhang, Q., Ding, K., & Chen, H. (2026) · Scientific Data · doi:10.1038/s41597-026-06594-9 · Cited by 2 · see citing works
“Existing LLM benchmarks in scientific domains (Chen et al., 2025; Sun et al., 2024; Feng et al., 2024; Saikh et al., 2022; Pedersen et al., 2020; Rubungo et al., 2024; Jiang et al., 2025) primarily focus on direct question-answering tasks, offering limited insight into how well LLMs perform in…”
Sarol, M. J., Hong, G., Guerra, E., & Halil Kilicoglu, B. (2024) · Database · doi:10.1093/database/baae079 · Cited by 6 · see citing works
Liu, Z., Liu, S., Yan, W., Song, P., Song, Y., & Gao, R. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792577 · Cited by 2 · see citing works
Yu, S., Cho, E.-J., Kim, S., Park, K., Kim, M.-S., Oh, Y., & Ryu, H. (2026) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2026.106270 · Cited by 5
Zeng, H., Liu, G., Liu, W., Zou, S.-Y., Chen, B., Yang, R., Lin, F., Peng, J., Gao, Y., & Hao, L. (2025) · Public Health · doi:10.1016/j.puhe.2025.106042 · Cited by 5 · see citing works
Self-citationDual-LLM Adversarial Framework for Information Extraction from Research Literature
Li, Z., Yu, Y., Gu, W.-H., Zhu, T.-T., Song, H., Guo, W.-B., Yang, X., & Zhu, Z.-X. (2025) · bioRxiv · doi:10.1101/2025.09.11.675507 · Cited by 5
Li, W., Tang, F.-X., Zhao, M., Omachi, M., & Kato, N. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3644352 · Cited by 5 · see citing works
“Inrecentyears,largelanguagemodels(LLMs)havedemon- stratedextraordinarycapabilitiesinnaturallanguageprocess-ing(NLP),includingconversation,reasoning,andtextgener-ation[28],[29],[30].”
Yin, L., Xue, X., Pan, S., Liu, X., Zhou, Y., Nahar, L., Chen, Y., Zhao, J., Zeng, S., & Xu, T. (2025) · Acta Materia Medica · doi:10.15212/amm-2025-0045 · Cited by 6 · see citing works
“These practices establish a robust foundation for applying LLMs in complex, individualized TCM contexts [159].”
Freeman, J., Millikin, R. J., Xu, L., Sharma, I., Moore, B., Lock, C., Shine George, K., Bal, A., Mohanty, C., & Stewart, R. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06350-7 · Cited by 4 · see citing works
“General-purpose LLMs [2, 9, 24] achieve state-of-the-art scores on biomedical reasoning and question-answering benchmarks [7, 17, 28].”
SynNER: syntax-infused named entity recognition in the biomedical domain
Imran, M., Zamaraeva, O., & Gómez-Rodríguez, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooaf149 · Cited by 4
“In this study, we use five biomedical benchmark datasets described in Chen et al. 24 (Table 1).”
Chen, Z., Gao, Y., Gu, S., Yong, Z.-P., Lei, L., Wang, R., Lei, Q., & Zeng, S. (2026) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00140-3 · Cited by 4 · see citing works
“However, since its training data spans a wide array of general domains rather than medical-specific datasets, its performance in specialized tasks such as medical reasoning and anesthesia plan generation may encounter limitations [19].”
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Adib, S. E. S., Sani, A. A., Esham, E. A., Abrar, A., & Chowdhury, T. M. (2025) · 2025 28th International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491589 · Cited by 4 · see citing works
“We employed standardized automatic evaluation metrics commonly used in medical NLP research to assess model performance across multiple dimensions [2], [15]: • BLEU Scores: Measures n-gram overlap between generated and reference answers, providing insights into lexical similarity and content…”
InfluentialHySim-LLM: Embedding-Weighted Fine-Tuning Bounds and Manifold Denoising for Domain-Adapted LLMs
Jaberi-Douraki, M., Sholehrasa, H., Xu, X., & Ramachandran, R. (2025) · arXiv.org · doi:10.48550/arxiv.2510.07796 · Cited by 4
“Recent advances in large language models (LLMs) have revolutionized natural language processing, enabling state-of-the-art performance in text summarization, retrieval, and reasoning [4].”
The application of large language models in qualitative nursing research: A scoping review.
Zhou, T.-T., Xing, M.-Y., Hu, Y., Liang, J., Ren, Q., Yang, Y.-Q., & Ye, L. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102544 · Cited by 4 · see citing works
Self-citationA review on recent advancements of ChatGPT and datafication in healthcare applications
Jagatheesaperumal, S. K., Pandiyarajan, A., Boopathy, P., Deepa, N., Barreto, A. G., & de Albuquerque, V. D. (2025) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2025.110885 · Cited by 4 · see citing works
Community-based biomedical context to unlock agentic systems
Kuehl, M., Schaub, D. P., Carli, F., Heumos, L., Fernández-Zapata, C., Kaiser, N., Schaul, J., Panzer, U., Bonn, S., Lobentanzer, S., et al. (2025) · bioRxiv · doi:10.1101/2025.07.21.665729 · Cited by 4
“Despite their potential applications, LLMs have significant limitations in biomedicine as they often generate hallucinations [4], cannot access specialized databases, and lack the necessary domain knowledge and functional capabilities [5], which are crucial for providing reliable scientific outputs.”
An In-Depth Evaluation of Federated Learning on Biomedical Natural Language ProcessingPreprint
Peng, L., Luo, G., Zhou, S.-C., Chen, J., Zhang, R., Xu, Z.-C., & Sun, J. (2023) · medRxiv · doi:10.1101/2023.11.24.23299002 · Cited by 4 · see citing works
“We followed the experimental protocol outlined in a recent study 32 and evaluated all the models on two NER datasets (2018 n2c2 and NCBI-disease) and two RE datasets (2018 n2c2, and GAD).”
MethodSelf-citationBenchmarking zero-shot biomedical relation triplet extraction across language model architectures
Gade, F. S., Lund, O., & Mendoza, M. L. (2025) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2025.bionlp-1.9
“Multiple papers have benchmarked LLMs for IE tasks on biomedical texts (Dai et al., 2024; Jahan et al., 2024; Chen et al., 2025), and there are multiple established combined benchmark datasets (e.g. BLURB (Gu et al., 2021)) and LLM instruction datasets (e.g. SciRIFF (Wadden et al., 2024)), but two…”
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 9 · see citing works
Self-citationMedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical ContextPreprint
Zhou, H.-J., Zou, X.-Y., Wu, J.-G., Wu, S., Yu, J.-C., Segal, B., Niebuhr, T. E., Amro, S., Petrus, M., Momin, S., et al. (2026) · bioRxiv · doi:10.64898/2026.05.25.727671 · Cited by 3 · see citing works
Wang, R.-D., Ba, W., Li, Y., Zhou, Y., Liu, B.-W., Wang, B.-Z., Wang, Y.-H., Yang, Z.-D., Zhang, K., Yan, R., et al. (2026) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.48550/arxiv.2603.01647 · Cited by 3
“Crucially, we introduce a quality control module that utilizes advanced LLMs [6,27,1] to simulate a pathologist’s auditing behavior.”
Fodeh, S., Wang, Y., Ma, L.-H., Talakokkul, S., Alpert, J. M., & Schellhorn, S. (2026) · Artif. Intell. Medicine · doi:10.48550/arxiv.2603.00028 · Cited by 3
“Similarly, BLURB [19] and recent large-scale evaluations [20] target traditional biomedical tasks.”
Zhu, J., Li, J., Zhao, S., Deng, Y., Miao, Y., & Xu, J. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-025-08182-x · Cited by 2 · see citing works
Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores
Amirian, B., Dale, A. S., Kalinin, S. V., & Hattrick-Simpers, J. (2025) · arXiv.org · doi:10.48550/arxiv.2512.01080 · Cited by 3
“Errors should be analyzed to ensure the model meets reliability standards [32].”
Kumari, M., Chauhan, R., Jain, R., & Garg, P. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103902 · Cited by 2 · see citing works
Chen, D., Wei, Y.-Z., He, Z.-L., Kuang, G.-M., Ye, C., An, M., Peng, H.-L., Hu, Y., Tao, H., & Cheung, K. M. C. (2025) · arXiv.org · doi:10.48550/arxiv.2511.00588 · Cited by 3 · see citing works
“On the other hand, currently reported evaluation metrics for LLM in health-care were based on question banks to benchmark LLM performance against medical knowledge mastery within specific disease domains or clinical subspecialties[25, 28, 23, 30, 19].”
InfluentialSelf-citationGireesh, H., Shukla, L., Shivaprakash, P., Mukherjee, A., Chand, P., & Murthy, P. (2025) · Drug and Alcohol Review · doi:10.1111/dar.70059 · Cited by 3 · see citing works
Chen, M., Zhang, Z., Chen, J., Rebibo, D., Wu, H., Fung, S.-H., Cheng, A. S., Tsui, S. K., Sinha, S., Cao, Q., et al. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.042 · Cited by 2 · see citing works
“Our benchmark complements other valuable benchmarks [32, 33], including those assessing symbolic or combinatorial reasoning, forming a varied and essential toolkit for advancing LLMs from general-purpose tools into reliable, domain-specific solutions.”
- Authors
- Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., Keloth, V. K., Raja, K., Huang, J., He, H., Lin, F., Du, J., Zhang, R., Zheng, W. J., Adelman, R. A., Lu, Z., & Xu, H.
- Affiliations
- National Institutes of Health (US)
- The University of Texas Health Science Center at Houston (US)
- Yale University (US)
- University of Minnesota (US)
- Corresponding author
- Lu, Zhiyong
- Xu, Hua
- ORCID
- Venue
- Nature Communications
- Published
- 2025-04-06
- Volume
- 16
- Issue
- 1
- Pages
- 3280-3280
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11972378
- OpenAlex
- W4409190866
- Cited by
- 219
- References
- 103
- Citations by year
- 2026: 116 · 2025: 76 · 2024: 19 · 2023: 8
- FWCI
- 57.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (1K99LM01402)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, biomedical natural language processing, benchmarking, GPT-4, llamas, fine-tuning, zero-shot learning, few-shot learning, hallucinations, medical question answering, BERT, BART, cost analysis
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
