The DOI matches its Crossref record; 93 of the 103 references it cites were found in a registry or an index.
Benchmarking large language models for biomedical natural language processing applications and recommendations
Open accessAbstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“For the tasks where the gold standard is fi xed, e.g., a classi fi cation type or multiple-choice option, we conducted qualitative evaluations on collectively hundreds of thousands of raw outputs of the LLMs (the raw outputs from three LLMs under zero-and one-shot conditions acrossthree benchmarks)…”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 4×
“Since the inception of transformers, biomedical language models have adopted their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family includi…”
InfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 1×
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“This is also supported by the existing studies on medical question answering 38,46 .”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“Nori et al. further found that general-domain LLMs with advanced prompt engineering can achieve the highest accuracy in medical question answering without fi ne-tuning 27 .”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339 · cited 1×
“1.”
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 3×
“The remarkable advancements have sparked extensive discussions among society, with excitement and concerns alike.”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 1×
“Pioneering studies have conducted early experiments on LLMs in the biomedical domain and reported encouraging results.”
Self-citationBioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · cited 1×
“It has demonstrated strong capabilities in the tasks that its predecessors fail to do; for instance, GPT-4 passed over 20 academic and professional exams, including the Uniform Bar Exam, SATEvidence-Based Reading& Writing, andMedicalKnowledge Self-Assessment Program 21 .”
BioCreative V CDR task corpus: a resource for chemical disease relation extraction
Li, J., Sun, Y., Johnson, R. J., Sciaky, D., Wei, C.-H., Leaman, R., Davis, A. P., Mattingly, C. J., Wiegers, T. C., & Lu, Z. (2016) · Database · doi:10.1093/database/baw068 · Cited by 910 · cited 1×
“BC5CDR-chemical 59 andNCBI-disease 60 aremanuallyannotatednamed entity recognition datasets for chemicals and diseases mentioned in biomedical literature, respectively.”
Self-citationPeng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · cited 1×
“The remarkable advancements have sparked extensive discussions among society, with excitement and concerns alike.”
Self-citationNCBI disease corpus: A resource for disease name recognition and concept normalization
Islamaj, R., Leaman, R., & Lu, Z. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.12.006 · Cited by 857 · cited 1×
“1.”
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1×
“1.”
Self-citationText and Code Embeddings by Contrastive Pre-TrainingPreprint
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q.-M., Tezak, N., Kim, J. W., Hallacy, C., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.10005 · Cited by 661 · cited 2×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658 · cited 1×
“1.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · cited 2×
“5 zero-shot and GPT-4 zero-shot, respectively (0.4988 and 0.7156 in our study, respectively) 38 .”
InfluentialBioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455 · cited 1×
“Moreover, the enhancement of LLMs is achieved through reinforcement learning with human feedback, thereby aligning text generation with human preferences 19 .”
Self-citationDatabase resources of the National Center for Biotechnology Information in 2023
Sayers, E. W., Bolton, E., Brister, J. R., Canese, K., Chan, J., Comeau, D. C., Farrell, C. M., Feldgarden, M., Fine, A. M., Funk, K., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac1032 · Cited by 452 · cited 1×
“…their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family including BioBERT and PubMedBERT 9 – 11 , (2)…”
Self-citationOpportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · cited 1×
“For instance, few studies evaluated more advanced closed-source LLMs such as GPT-4, LLM representatives from the general domain such as LLaMA 22 , and bio-medical domain-speci fi c LLMs such as PMC-LLaMA 34 .”
Self-citationLarge language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · cited 1×
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Large Language Models: A Survey
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., & Gao, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06196 · Cited by 325 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · cited 2×
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Self-citationPMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · cited 5×
“An accuracyof 0.7790 on PubMedQA wasalsoreported for the fi ne-tuned PMC LLaMA 13B (combined multiple question answering datasets for fi ne-tuning) 34 ; our study also reported a similar accuracy of 0.7680 using the PubMedQA training set only.”
InfluentialDeep Bidirectional Language-Knowledge Graph PretrainingPreprint
Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., & Leskovec, J. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.09338 · Cited by 300
BioSentVec: creating sentence embeddings for biomedical texts
Chen, Q.-Y., Peng, Y., & Lu, Z.-Y. (2018) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi.2019.8904728 · Cited by 241 · cited 1×
“networks such as BioWordVec and BioSentVec 4,7,8 .”
Self-citationFine-tuning large neural language models for biomedical natural language processing
Tinn, R., Cheng, H., 池谷, 裕., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2023) · Patterns · doi:10.1016/j.patter.2023.100729 · Cited by 210 · cited 3×
“In the biomedical domain, a primary challenge is the limited availability of labeled datasets, which have a signi fi cantly lower scale than those in the general domain (e.g., a biomedical sentence similarity dataset only has 100 labeled instances in total 31 ) 32,33 .”
InfluentialThinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
Gutierrez, B. J., McNeal, N., Washington, C., Chen, Y., Li, L., Sun, H., & Su, Y. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.08410 · Cited by 207 · cited 1×
“In contrast, LLMs may have the advantage when minimal manually labeled instances are available, and they do not require fi ne-tuning or retraining for every new task through zero/few-shot learning.”
SciFive: a text-to-text transformer model for biomedical literaturePreprint
Phan, L., Anibal, J., Tran, H., Chanana, S., Bahadroglu, E., Peltekian, A., & Altan-Bonnet, G. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03598 · Cited by 204 · cited 1×
“It has demonstrated strong capabilities in the tasks that its predecessors fail to do; for instance, GPT-4 passed over 20 academic and professional exams, including the Uniform Bar Exam, SATEvidence-Based Reading& Writing, andMedicalKnowledge Self-Assessment Program 21 .”
BioBART: Pretraining and Evaluation of A Biomedical Generative Language ModelPreprint
Yuan, H., Yuan, Z., Gan, R., Zhang, J., Xie, Y., & Yu, S. (2022) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2204.03905 · Cited by 200 · cited 1×
“…, (2) decoder-based, generative language models using the decoder from the transformer architecture such as the generative pre-trained transformer (GPT) family including BioGPT and BioMedLM 12,13 , and (3) encoder-decoder-based, using both encoders and decoders such as BioBART and Sci fi ve 14,15 .”
Wei, C.-H., Peng, Y., Leaman, R., Davis, A. P., Mattingly, C. J., Li, J., Wiegers, T. C., & Lu, Z. (2016) · Database · doi:10.1093/database/baw032 · Cited by 198
Self-citationCan Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · cited 3×
“5, which may not provide a full spectrum of representative LLMs from different categories.”
BIOSSES: a semantic sentence similarity estimation system for the biomedical domain
Soğancıoğlu, G., Öztürk, H., & Özgür, A. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx238 · Cited by 159 · cited 1×
“Arguably, existing transformer models have demonstrated satisfactory performance in extractive tasks, while generative tasks remain a challenge in terms of achieving similar levels of pro fi ciency.”
An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics
Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2022) · ACM Computing Surveys · doi:10.1145/3545176 · Cited by 152
ML-Net: multi-label classification of biomedical texts with deep neural networks
Du, J., Chen, Q.-Y., Peng, Y., Xiang, Y., Tao, C., & Lu, Z.-Y. (2018) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocz085 · Cited by 148 · cited 1×
“1.”
Self-citationBiomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2022) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 145 · cited 2×
“1.”
Self-citationParagraph-level Simplification of Medical Texts
Devaraj, A., Marshall, I., Wallace, B. C., & Li, J. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.05767 · Cited by 144
Automatic semantic classification of scientific literature according to the hallmarks of cancer
Baker, S., Silins, I., Guo, Y., Ali, I., Högberg, J., Stenius, U., & Korhonen, A. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv585 · Cited by 128 · cited 1×
“1.”
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Biocuration: Distilling data into knowledge
International Society for Biocuration (2018) · PLoS Biology · doi:10.1371/journal.pbio.2002846 · Cited by 121 · cited 1×
“1.”
MSˆ2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.594 · Cited by 120 · cited 2×
“1.”
How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding TasksPreprint
Chen, X., Ye, J., Zu, C., Xu, N., Zheng, R., Peng, M.-L., Zhou, J., Gui, T., Zhang, Q., & Huang, X. (2023) · arXiv.org · doi:10.48550/arxiv.2303.00293 · Cited by 116 · cited 3×
“The MedQA dataset is collected from questions in the United States Medical License Examination (USMLE), whereeach instancecontains aquestion (usually a patient description) and fi ve answer choices (e.g., fi ve potential diagnoses) 66 .”
Advancing entity recognition in biomedicine via instruction tuning of large language models
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae163 · Cited by 104 · cited 2×
“This approach aligns with previous studies in the biomedical domain, which have demonstrated that incorporating task guidance into the prompt leads to improved performance 74,76 and was also employed and evaluated in our previous study, speci fi cally focusing on named entity recognition 77 .”
Self-citationBlake, C. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.11.001 · Cited by 91 · cited 1×
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Labrak, Y., Rouvier, M., & Dufour, R. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2307.12114 · Cited by 76 · cited 2×
“Recall that we categorized inconsistencies, missing information, and hallucinations on the tasks where the gold standard is a fi xed classi fi cation type or a multiple-choice option.”
InfluentialIslamaj, R., Kim, S., Chatr‐aryamontri, A., Wei, C.-H., Comeau, D. C., Antunes, R., Matos, S., Chen, Q., Elangovan, A., Panyam, N. C., et al. (2018) · Database · doi:10.1093/database/bay147 · Cited by 76
Self-citationEvaluating GPT and BERT models for protein–protein interaction identification in biomedical text
Rehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y. O., Özgür, A., & Hur, J. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae133 · Cited by 73
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65 · cited 4×
“Other studies reported similar fi ndings regarding the low correlation between automatic and manual evaluations 35,47 .”
InfluentialDo We Still Need Clinical Language Models?
Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., & Alsentzer, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08091 · Cited by 62
Chen, Q., Lee, K., Yan, S., Kim, S., Wei, C.-H., & Lu, Z. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007617 · Cited by 62 · cited 1×
“networks such as BioWordVec and BioSentVec 4,7,8 .”
Self-citationOverview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles
Goldsack, T., Scarton, C., Shardlow, M., & Lin, C. (2024) · Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/2024.bionlp-1.10 · Cited by 61 · cited 1×
“1.”
LitSense: making sense of biomedical literature at sentence level
Allot, A., Chen, Q., Kim, S., Alvarez, R. V., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2019) · Nucleic Acids Research · doi:10.1093/nar/gkz289 · Cited by 55 · cited 1×
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Self-citationExploring the Boundaries of GPT-4 in Radiology
Liu, Q.-C., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M., Tinn, R., Sharma, H., Pérez-García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.14573 · Cited by 55 · cited 1×
“There are several primary limitations.”
Readability Controllable Biomedical Document SummarizationPreprint
Luo, Z.-H., Xie, Q., & Ananiadou, S. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.04705 · Cited by 51
Self-citationChen, Q.-Y., Allot, A., Leaman, R., Islamaj, R., Du, J.-C., Fang, L., Kai, W., Xu, S., Zhang, Y., Bagherzadeh, P., et al. (2022) · Database J. Biol. Databases Curation · doi:10.48550/arxiv.2204.09781 · Cited by 48 · cited 1×
“1.”
Self-citationRehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y., Özgür, A., & Hur, J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17728 · Cited by 47 · cited 1×
“In contrast, LLMs may have the advantage when minimal manually labeled instances are available, and they do not require fi ne-tuning or retraining for every new task through zero/few-shot learning.”
Optimizing Bi-Encoder for Named Entity Recognition via Contrastive LearningPreprint
Zhang, S., Cheng, H., Gao, J., & Poon, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.14565 · Cited by 46
Chen, Q., Du, J., Allot, A., & Lu, Z. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3173562 · Cited by 44
Self-citationStricker, J., Chasiotis, A., Kerwer, M., & Günther, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0231160 · Cited by 44
A survey of automated methods for biomedical text simplification
Ondov, B., Attal, K., & Demner‐Fushman, D. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac149 · Cited by 42
Bioformer: an efficient transformer language model for biomedical text miningPreprint
Li, F., Chen, Q., Wei, C.-H., Lu, Z., & Wang, K. (2023) · PubMed · doi:10.48550/arxiv.2302.01588 · Cited by 38 · cited 2×
“5 and GPT-4 — has marked a milestone in generative arti fi cial intelligence.”
Self-citationPrompt Tuning in Biomedical Relation Extraction
He, J., Li, F., Li, J., Hu, X., Nian, Y., Xiang, Y., Wang, J., Wei, Q., Li, Y., Xu, H., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00162-9 · Cited by 36
Self-citationGRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization
Xie, Q., Huang, J.-M., Saha, T., & Ananiadou, S. (2022) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2208.09982 · Cited by 35 · cited 1×
“BC5CDR-chemical 59 andNCBI-disease 60 aremanuallyannotatednamed entity recognition datasets for chemicals and diseases mentioned in biomedical literature, respectively.”
Self-citationWang, L., Bi, W., Zhao, S., Ma, Y., Lv, L., Meng, C., Fu, J., & Lv, H. (2024) · JMIR Formative Research · doi:10.2196/53216 · Cited by 35 · cited 1×
“To date, prompt design remains an open research problem 73 – 75 .”
LitCovid in 2022: an information resource for the COVID-19 literature
Chen, Q.-Y., Allot, A., Leaman, R., Wei, C.-H., Aghaarabi, E., Guerrerio, J. J., Xu, L., & Lu, Z.-Y. (2022) · Nucleic Acids Res. · doi:10.48550/arxiv.2209.13428 · Cited by 34 · cited 1×
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Self-citationImproving Small Language Models on PubMedQA via Generative Data AugmentationPreprint
Guo, Z., Wang, P., Wang, Y., & Yu, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07804 · Cited by 34
Li, X., Rousseau, J. F., Ding, Y., Song, M., & Lu, W. P. (2020) · JMIR Medical Informatics · doi:10.2196/16739 · Cited by 31 · cited 1×
“Relation extraction involves identifying the relationships between entities, which is important for drug repurposing and knowledge discovery 61 .”
LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation
Bishop, J. A., Ananiadou, S., & Xie, Q. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2309.12455 · Cited by 30
Self-citationScaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology
Wong, C., Zhang, S., Gu, Y., Moung, C., Abel, J., Usuyama, N., Weerasinghe, R., Piening, B., Naumann, T., Bifulco, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02180 · Cited by 28 · cited 1×
“5 and GPT-4 for end-to-end clinical trial matching, handling complex eligibility criteria, and extracting complex matching logic 25 .”
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2023) · arXiv · doi:10.48550/arxiv.2310.05694 · Cited by 27
Correction to: Evaluating the ChatGPT family of models for biomedical reasoning and classification
(2024) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocae083 · Cited by 26
A Comprehensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPTPreprint
Chen, Q., Sun, H., Liu, H., Jiang, Y., Ran, T., Jin, X., Xiao, X.-L., Lin, Z.-M., Niu, Z., & Chen, H. (2023) · bioRxiv · doi:10.1101/2023.04.19.537463 · Cited by 24
Self-citationChen, Q., Rankine, A., Peng, Y., Aghaarabi, E., & Lu, Z. (2021) · JMIR Medical Informatics · doi:10.2196/27386 · Cited by 21 · cited 1×
“Arguably, existing transformer models have demonstrated satisfactory performance in extractive tasks, while generative tasks remain a challenge in terms of achieving similar levels of pro fi ciency.”
Self-citationFlores, L. J. Y., Huang, H., Shi, K., Chheang, S., & Cohan, A. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.11191 · Cited by 19
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.-Y., Chen, H., & Hu, X. (2024) · arXiv · doi:10.48550/arxiv.2401.01325 · Cited by 16 · cited 1×
“Under the fi ne-tuning setting, the results also indicate that PMC LLaMA 13B, as a continuously pretrained biomedical domain-speci fi c LLM, did not achieve an overall higher performance than LLaMA 2 13B. Fine-tuned LLaMA 2 13B had better performance than that of PMC LLaMA 13B in 10 out of the 12…”
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., & Yang, M. (2024) · arXiv · doi:10.48550/arxiv.2402.13753 · Cited by 15 · cited 1×
“Under the fi ne-tuning setting, the results also indicate that PMC LLaMA 13B, as a continuously pretrained biomedical domain-speci fi c LLM, did not achieve an overall higher performance than LLaMA 2 13B. Fine-tuned LLaMA 2 13B had better performance than that of PMC LLaMA 13B in 10 out of the 12…”
Su, Y.-S., Wang, M., Wang, P., Zheng, C.-H., Liu, Y.-S., & Zeng, X.-X. (2022) · Briefings Bioinform. · doi:10.1093/bib/bbac342 · Cited by 14 · cited 2×
“…studies fi ne-tuned those language models and demonstrated that they achieved the SOTA performance in various BioNLP applications 10,16 , and those models have been successfully employed in PubMed-scale downstream applications such as biomedical sentence search 17 and COVID-19 literature mining 2 .”
Biomedical Question Answering: A Survey of Methods and Datasets
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds50568.2020.9268742 · Cited by 12 · cited 1×
“BiomedicalQA datasets such as MedQA and PubMedQA have been widely used in the evaluation of language models 65 .”
Comprehensively identifying Long Covid articles with human-in-the-loop machine learning
Leaman, R., Islamaj, R., Allot, A., Chen, Q.-Y., Wilbur, W., & Lu, Z.-Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100659 · Cited by 5
Self-citationChemical-Protein Relation Extraction with Pre-trained Prompt Tuning
He, J.-P., Li, F., Hu, X.-Y., Li, J., Yi, N., Wang, J.-Q., Xiang, Y., Wei, Q., Xu, H., & Tao, C. (2022) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi54592.2022.00120 · Cited by 5
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“5 and GPT-4 for end-to-end clinical trial matching, handling complex eligibility criteria, and extracting complex matching logic 25 .”
Research works printed but not linked: 7
- Jiang, A. Q. et al. Mixtral of experts arXiv preprint arXiv:2401.04088, 2024. · arXiv
- Lee, P, Goldberg, C. & Kohane, I. The AI revolution in medicine: GPT-4 and beyond (Pearson, 2023).
- Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. & Daneshjou, R. Large language models in medicine: the potentials and pitfalls: a narrative review. Ann. Intern. Med. 177, 210–220 (2024). · DOI
- Krallinger, M. et al. Overview of the BioCreative VI chemical-protein interaction Track. In Proc. of the sixth BioCreative challenge evaluation workshop Vol. 1, 141–146 (2017).
- Segura-Bedmar, I., Martínez, P. & Herrero-Zazo, M. Semeval-2013 task 9: extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proc. Seventh International Workshop on Semantic Evaluation (SemEval 2013) 341–350 (Association for Computational Linguistics, 2013).
- Mingliang, D., Jijun, T. & Fei, G. Document-level DDI relation extraction with document-entity embedding. pp. 392–397. · DOI
- Wang, L. L, DeYoung, J. & Wallace, B. Overview of MSLR2022: A shared task on multidocument summarization for literature reviews. In Proc. Third Workshop on Scholarly Document Processing 175–180 (Association for Computational Linguistics, 2022).
Cited by
Sorted by citations
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414
Self-citationOpportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
“Chen et al. [105] performed a pilot study to establish the baselines of using GPT-3.”
BackgroundSelf-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312
“Neverthe-less, current systematic evaluations in biomedicine typically focus on the vanilla LLMs without RAG (Chen et al., 2023a; Nori et al., 2023a).”
BackgroundSelf-citationA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
Self-citation
Metadata
- Authors
- Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., Keloth, V. K., Raja, K., Huang, J., He, H., Lin, F., Du, J., Zhang, R., Zheng, W. J., Adelman, R. A., Lu, Z., & Xu, H.
- Affiliations
- National Institutes of Health (US)
- The University of Texas Health Science Center at Houston (US)
- Yale University (US)
- University of Minnesota (US)
- Corresponding author
- Lu, Zhiyong
- Xu, Hua
- ORCID
- Venue
- Nature Communications
- Published
- 2025-04-06
- Volume
- 16
- Issue
- 1
- Pages
- 3280-3280
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11972378
- OpenAlex
- W4409190866
- Cited by
- 219
- References
- 103
- Citations by year
- 2026: 116 · 2025: 76
- FWCI
- 57.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (1K99LM01402)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, biomedical natural language processing, benchmarking, GPT-4, llamas, fine-tuning, zero-shot learning, few-shot learning, hallucinations, medical question answering, BERT, BART, cost analysis
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
