The DOI matches its Crossref record; 93 of the 103 references it cites were found in a registry or an index.
Benchmarking large language models for biomedical natural language processing applications and recommendations
Open accessAbstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs—GPT and LLaMA representatives—on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“For the tasks where the gold standard is fi xed, e.g., a classi fi cation type or multiple-choice option, we conducted qualitative evaluations on collectively hundreds of thousands of raw outputs of the LLMs (the raw outputs from three LLMs under zero-and one-shot conditions acrossthree benchmarks)…”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 4×
“Since the inception of transformers, biomedical language models have adopted their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family includi…”
InfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 1×
“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“This is also supported by the existing studies on medical question answering 38,46 .”
Research works printed but not linked: 7
- Jiang, A. Q. et al. Mixtral of experts arXiv preprint arXiv:2401.04088, 2024. · arXiv
- Lee, P, Goldberg, C. & Kohane, I. The AI revolution in medicine: GPT-4 and beyond (Pearson, 2023).
- Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. & Daneshjou, R. Large language models in medicine: the potentials and pitfalls: a narrative review. Ann. Intern. Med. 177, 210–220 (2024). · DOI
- Krallinger, M. et al. Overview of the BioCreative VI chemical-protein interaction Track. In Proc. of the sixth BioCreative challenge evaluation workshop Vol. 1, 141–146 (2017).
- Segura-Bedmar, I., Martínez, P. & Herrero-Zazo, M. Semeval-2013 task 9: extraction of drug-drug interactions from biomedical texts (ddiextraction 2013). In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proc. Seventh International Workshop on Semantic Evaluation (SemEval 2013) 341–350 (Association for Computational Linguistics, 2013).
- Mingliang, D., Jijun, T. & Fei, G. Document-level DDI relation extraction with document-entity embedding. pp. 392–397. · DOI
- Wang, L. L, DeYoung, J. & Wallace, B. Overview of MSLR2022: A shared task on multidocument summarization for literature reviews. In Proc. Third Workshop on Scholarly Document Processing 175–180 (Association for Computational Linguistics, 2022).
Cited by
Sorted by citations
Items 101–200 of 281
Gireesh, H., Shukla, L., Shivaprakash, P., Mukherjee, A., Chand, P., & Murthy, P. (2025) · Drug and Alcohol Review · doi:10.1111/dar.70059 · Cited by 3
Chen, M., Zhang, Z., Chen, J., Rebibo, D., Wu, H., Fung, S.-H., Cheng, A. S., Tsui, S. K., Sinha, S., Cao, Q., et al. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.042 · Cited by 2
“Our benchmark complements other valuable benchmarks [32, 33], including those assessing symbolic or combinatorial reasoning, forming a varied and essential toolkit for advancing LLMs from general-purpose tools into reliable, domain-specific solutions.”
ExpNote: Black-box Large Language Models are Better Task Solvers with Experience Notebook
Sun, W., Yu, X.-Q., He, S.-Z., Zhao, J., & Liu, K. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2311.07032 · Cited by 3
“As a result, LLMs may perform mediocre on some specific tasks, such as inductive reasoning (Bang et al., 2023) and entity recognition (Chen et al., 2023).”
BackgroundHexAI-TJAtxt: A textual dataset to advance open scientific research in total joint arthroplasty
Amirian, S., Ghazaleh, H., Carlson, L. A., Gong, M. F., Finger, L. E., Plate, J. F., & Tafti, A. P. (2023) · Data in Brief · doi:10.1016/j.dib.2023.109738 · Cited by 3
“…powerful LLMs, such as GPT-2, GPT-3, GPT-4, PaLM2, LlaMA, and Bard have demonstrated impressive performance in various domains, such as healthcare, offering the potential to enhance healthcare accessibility, promote informed decision-making, and contribute to improved patient outcomes [ 7–10 ].”
BackgroundConversational LLM-Based Decision Support for Defect Classification in AFM Images
Biswas, A., Rade, J., Masud, N., Hasib, M. H. H., Balu, A., Zhang, J., Sarkar, S., Krishnamurthy, A., Ren, J., & Sarkar, A. (2025) · IEEE Open Journal of Instrumentation and Measurement · doi:10.1109/ojim.2025.3592284 · Cited by 4
“…process vast amounts of information, enabling a broad spectrum of tasks [20], including question-answering, document summarization, language translation, data analysis, and code generation across diverse domains, such as biomedical and health applications [21], [22], agriculture [23], [24], and 3-”
Bannour, N., Andrew, J. J., & Vincent, M. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.39 · Cited by 2
“With the rise of LLMs, recent research efforts attempt to apply these models to the TRE task, but results are still debatable (Han et al., 2023; Chen et al., 2023; Li et al., 2023a).”
MethodA roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 3
“…such as AlphaFold [2], which primarily garnered attention within specific scientific communities, ChatGPT demonstrated exceptional capabilities in natural language understanding (NLU) and generation (NLG), capturing widespread interest across academia, industry, and the general public [3].”
Self-citationGenerative Artificial Intelligence and Large Language Models in Clinical Oncology
Yu, Y., Zhao, Z., Wang, Z., Lin, R., Tan, Y., Chen, Y., Li, T., Baptista‐Hon, D. T., Zhang, X., Wu, C., et al. (2026) · MedComm · doi:10.1002/mco2.70833 · Cited by 2
Self-citationEvidence-based AI: from trailblazer to trustblazer?
Luechtefeld, T., & Hartung, T. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1818128 · Cited by 4
Ang, M. Y., Chen, L., Song, L., Lipovich, L., & Choo, S. W. (2026) · Life · doi:10.3390/life16061032 · Cited by 2
“…are unlikely to be sufficient because they often do not test microbial-genomics-specific reasoning, such as gene-to-phenotype interpretation, database-aware AMR explanation, strain-level virulence reasoning, compositional microbiome interpretation, workflow validation, or citation fidelity [4,8].”
Influentialcadmus: a robust pipeline for scalable retrieval of full-text biomedical literaturePreprint
Campbell, J., Lain, A. D., & Simpson, T. (2026) · bioRxiv · doi:10.64898/2026.05.16.725623 · Cited by 2
“Creating large, domain-specific full-text corpora, therefore remains a challenge for Biomedical Natural Language (BioNLP) applications, including knowledge graph construction (3, 4), and reliable biomedical question answering systems (5–7).”
Zouaghi, M., Kacem, F., & Fathallah, H. (2026) · IEEE Middle East and North Africa Communications Conference (MENACOMM) · doi:10.1109/menacomm69507.2026.11532824 · Cited by 2
“Recent benchmarking studies [11] evaluate a narrow set of tasks without covering the full spectrum of architectural families or pretraining strategies.”
Cheng, F., Li, Q., He, L., Li, H., Brooks, B. W., Yu, Z., & You, J. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.6c01342 · Cited by 3
Efficient and Verified Research Data Extraction with LLM
Serdiukov, A., Dravgelis, V., Smutin, D., Taldaev, A., Ivanov, A., Adonin, L., & Muravyov, S. (2026) · Algorithms · doi:10.3390/a19030214 · Cited by 2
“While these methods are much more efficient than SOTA fine-tuning approaches, they tend to be less precise in most cases [12].”
A data-centric approach to detecting and mitigating demographic bias in pediatric mental health text
Ive, J., Bondaronek, P., Yadav, V., Santel, D., Glauser, T. A., Strawn, J. R., Agasthya, G. A., Tschida, J., Choo, S., Chandrashekar, M., et al. (2026) · Communications Medicine · doi:10.1038/s43856-026-01480-2 · Cited by 3
“…38,39 ; (ii) such models have been shown to perform on par with, or in some cases out-perform, muchlargergenerativemodelsonin-domainclassi fi cationtasks 40 ; and (iii) its low computational requirements allowed us to run systematic experiments in the limited computational environment.”
Pan, Y., Duan, C., Du, J., Zhang, J., Du, K., Zhang, C., Liu, Z., Zhang, W., Wang, B., Ren, Y., et al. (2026) · Annals of Surgical Oncology · doi:10.1245/s10434-026-19176-1 · Cited by 2
“34 Therefore, the high robustness of DeepSeek only reflects the stability of its output and does not directly demonstrate its clinical decision-making abilities.”
Self-citationKim, J., & Shin, H.-W. (2026) · Syst. · doi:10.3390/systems14020153 · Cited by 2
Cimiano, P., Brazda, N., Härtung, M., Starke-Knäusel, C., Schmidt, A. L., Vuono, M. C. D., Tyagi, A. S., Gottowik, J., Rodriguez‐Esteban, R., Collins, B., et al. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2025.1703923 · Cited by 2
Zambare, N., Aghakasiri, K., Lin, C., Ye, C., Mitchell, J., & Abdalla, M. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.findings-eacl.222 · Cited by 2
“Chen et al. (2025) and Naguib et al. (2024) evaluated pretrained models on biomedical NLP tasks (performance and cost) and NER (performance and carbon emissions), respectively.”
FracLogGPT: A Multimodal Large Language Model for Fracture Interpretation in Imaging Logging
Shen, H., Li, A., Zhang, L.-Y., & Liu, X.-X. (2025) · Electronics · doi:10.3390/electronics15010127 · Cited by 2
Srinivasan, S., Ai, X., Lo, T. W. S., Gilson, A., Zou, M., Zou, K., Kim, H., Yang, M., Pushpanathan, K., Yew, S. M. E., et al. (2025) · Ophthalmology Science · doi:10.1016/j.xops.2025.101050 · Cited by 4
Self-citationLiu, Y., Bian, J., Zhai, W., Zheng, J., Zhang, Y., & Zhu, S. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00400-3 · Cited by 2
“Some works directly prompted LLMs such as GPT-4 for Biomedical NER and found that even with well-crafted prompts, the results are still not comparable with supervised BERT-based models [7, 8].”
Element2Vec: Build Chemical Element Representation from Text for Property Prediction
Li, Y.-H., Lai, K., Wang, T., Liu, Q., Ma, J., & Hu, Y. (2025) · arXiv.org · doi:10.48550/arxiv.2510.13916 · Cited by 2
“Still, the gap in constructing element-level embeddings from text still exists, which motivate our Element2Vec approach that learns broadly applicable element embeddings from text and evaluates them in material property prediction.”
Self-citationBIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025
Li, C.-Y., Zheng, X., & Liu, S. (2025) · Conference and Labs of the Evaluation Forum · doi:10.48550/arxiv.2509.09725 · Cited by 2
Sezgin, A., Tanık, V. O., Akdoğan, M., Şahin, Y. B., Akbuğa, K., Hekimsoy, V., Tunca, Ç., Saraçoğlu, E., & Özlek, B. (2025) · Türk Kardiyoloji Derneği arşivi : Türk Kardiyoloji Derneğinin yayın organıdır · doi:10.5543/tkda.2025.54968 · Cited by 2
“15 Future research should focus on developing adaptive validation frameworks that account for version changes and enable continuous performance monitoring over time.”
Alkarmouty, M., Ooka, J., & Yamashita, F. (2025) · International journal of pharmaceutics · doi:10.1016/j.ijpharm.2025.126111 · Cited by 2
He, J., Lu, Y.-Y., Lin, Z., Chen, K., & Zhao, Y. (2025) · USENIX Security Symposium · doi:10.48550/arxiv.2506.19563 · Cited by 2
“In practice, domain adaptation typically prioritizes downstream utility over benchmark preservation [8,51].”
Zhou, X., Huang, L., Chen, P., Yin, W., Zhang, R., Hao, W., & Cheng, L. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402087 · Cited by 3
“For example, the best performance of LLM shown in [4] for presents 0.3391 accuracy generated by GPT4.”
InfluentialSelf-citationRapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Engine
Rohanian, O., Nouriborji, M., Seminog, O., Furst, R., Mendy, T., Levanita, S., Kadri-Alabi, Z., Jabin, N., Toale, D., Humphreys, G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2407.10086 · Cited by 2
“Chen et al. (2023) and Tian et al. (2024) have studied the use of LLMs in a number of biomedical text processing tasks, including document classification, although the focus is mostly on closed-source frozen models like GPT-4.”
BackgroundRa, D., Lee, D., & Lee, S. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.118157 · Cited by 1
Utama, P. K. L., Tashu, T. M., & Dibangoye, J. (2026) · Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026) · doi:10.18653/v1/2026.loreslm-1.21 · Cited by 2
MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
Lu, J., Liu, J., Zheng, X., Yang, M., Wang, J., Wang, P., & Zhang, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i45.41243 · Cited by 1
“Large Language Models (LLMs) have emerged as a technology with the potential to fundamentally reshape healthcare, offering powerful capabilities for augmenting clinical work-flows, from summarizing electronic health records to providing decision support (Al-Hakami et al. 2024; Chen et al. 2025).”
Advancing AI Research Assistants with Expert-Involved LearningPreprint
Liu, T., Han, S., Wang, H., Luo, X., Lu, P., Zhu, B., Wang, Y., Li, K., Chen, J., Qu, R., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04638 · Cited by 6
“Compared with existing benchmarks (32–34), ARIEL is both both domain-specific and more rigorous, reflecting the complexity of scientific language.”
Self-citationAdding layers of information to scRNA-seq data using pre-trained language modelsPreprint
Krissmer, S. M., Menger, J., Rollin, J., Vogel, T., Binder, H., & Hackenberg, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.23.671699 · Cited by 1
“For example, joint embedding spaces shared between transcriptomic models such as Geneformer [3] or scGPT [2] and literature-trained models like BioBERT [20] integrate raw expression data with textual knowledge by allowing user queries to be matched directly to cells via semantic similarity [4, 21,…”
Tang, D., & Miškov-Živanov, N. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.06.668950
“This observation is consistent with broader biomedical natural language processing evaluations showing that task-specific fine-tuning remains important for reliable LLM performance 42 .”
Hu, Q., Zhu, Q.-Q., Zhang, H., Liu, T., & Liu, S. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag641 · Cited by 1
“An LLM may hallucinate invalid command-line options or rely on machine-specific dependencies, leading to workflows that function on one machine but fail on another (Ji et al. 2023, Chen et al. 2025).”
A machine-assisted framework for systematic error analysis in clinical concept extraction
Fu, S., Lu, Q., Ahn, J., Chen, F., Yin, H., Wen, J., Yue, Z., Harrison, T., Jun, J., Ruan, X., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-77067-1 · Cited by 2
“These results support the feasibility of LLM-assisted error analysis while con fi rming that fully automated classi fi cationremainsinsuf fi cientforhigh-stakesclinicaldeployment 10 .”
Huynh, D.-L., Asp, E., Ballante, F., Puigvert, J. C., DeGrave, A. N., Karki, R., Nader, K. M., Östling, P., Pokharel, B. K., Rietdijk, J., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.20.719538 · Cited by 2
Liu, J., Luo, L., Pan, D., Kang, Y., Sun, Y., Lin, H., Yu, L., & Zhuang, H. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00473-8 · Cited by 1
Kim, H., Kim, D., Xiao, P., Applebaum, S., Chung, Y., Ai, X.-G., Yin, Y., Jiang, R., Du, Y., Wei, Y.-W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.07673 · Cited by 1
“Given the rapid development of medical foundation models, it is not feasible for one study to evaluate every emerging architecture or clinical use case 19,74,75 .”
Self-citationTranslating control: Benchmarking language AI and sociotechnical governance
Yang, D. (2026) · Human Organization · doi:10.1080/00187259.2026.2688981 · Cited by 1
Masthoff, M., Zipser, A., Praktiknjo, M., Trebicka, J., Morgul, H. M., Pascher, A., Pöhler, G. H., Köhler, M., & Schindler, P. (2026) · Scientific Reports · doi:10.1038/s41598-026-56992-7 · Cited by 1
“These findings are congruent with other studies that emphasize the need for detailed input when using LLM to summarize and analyze complex medical data, with closed-source LLM however performing mostly better in unseen data tasks 9,30,31 .”
Synergistic Prompting for Complementarity and Consistency in Incomplete Multi-View Clustering
Hao, X.-S., Zhang, Z.-H., Tang, Y., Zhang, L.-F., Hao, P., Diao, Y., Jin, G.-Y., & Liu, Y. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3700907 · Cited by 1
“To address these challenges, we draw inspiration from the emerging paradigm of prompt learning , which has gained traction in natural language processing [4], [5], [18], [59] and is increasingly applied to vision-language modeling [32], [50], [53].”
Liu, W., Zhou, G., Gao, X., Li, B., & Zhao, Y. (2026) · Journal of Nanobiotechnology · doi:10.1186/s12951-026-04474-3 · Cited by 1
“Compared to traditional approaches, LLMs such as ChatGPT possess multimodal understanding, database integration, and reasoning capabilities, enabling efficient identification of potential targets and construction of disease-drug association networks [31, 32].”
Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiries
Zhou, L.-Q., & Li, J.-F. (2026) · arXiv.org · doi:10.48550/arxiv.2605.15680 · Cited by 1
Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in DermatologyPreprint
Jiang, R., Kim, H., Qin, Z., Lee, M., MacGibeny, M. A., Hanly, A. M., Sadlowski, A., Chowdhury, S., Ai, X., Gehlhausen, J. R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.04098 · Cited by 2
Self-citationAgarwal, R., Wenzel, M., Baur, S., Zimmer, J., Harvey, G., & Ma, J. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.15203 · Cited by 1
GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding
Long, W.-C., Hou, Y., Feng, J., Su, H., Yang, S., Xie, D., & Zhang, Y. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.05774 · Cited by 1
“In contrast, exam-style benchmarks (Chen et al., 2025; Queen et al., 2025; Yin et al., 2025) evaluate biological knowledge using text-only questions without requiring sequence-level analysis.”
EMLC: An extensible multi-level correction framework for text-to-SQL
Lei, J., Tan, Y., & Wang, Y. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2025.104560 · Cited by 1
BioShield: A Context-Aware Firewall for Securing Bio-LLMs
Das, P., Chakraborty, S., Narula, S., Potter, L., Palmer, X., Rana, P., Takabi, D., & GhashemiGol, M. (2026) · Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics · doi:10.1145/3807502.3827569 · Cited by 1
Toward Reliable, Safe, and Secure LLMs for Scientific ApplicationsPreprint
Chaturvedi, S. S., Bergerson, J., & Mallick, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.18235 · Cited by 1
“Furthermore, in research papers, benchmarks such as RoBBR 62 have redefined “bias” as methodological bias , while UniTox 47 redefines “toxicity” as drug-induced toxicity , highlighting the critical need for highly specialized scientific evaluations 63–65 .”
Fang, Y., Zhang, G., Tejada, S. S., Chen, F., Shortliffe, E. H., Patel, V., Mor Peleg, B., & Weng, C. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105013 · Cited by 3
Natural language querying of biological databases with large language models
Makarov, V. A., Stroganov, O. V., Furlong, L. I., Evarts, B., van den Biggelaar, L., Goulas, A., Stolte, E., Marren, D., & Greiffenberg, L. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104654 · Cited by 1
Cai, J., Li, G. Z., Yang, Y., Xu, K., Zhao, S., Hospedales, T. M., Zhao, L., Yang, J., Wu, Z., Zhang, T. J., et al. (2026) · Journal of genetics and genomics/Journal of Genetics and Genomics · doi:10.1016/j.jgg.2026.02.009 · Cited by 3
Balancing Binary Biomedical QA: Parameter-Efficient Instruction Tuning on BioASQ and PubMedQA
Sarker, T. K., Islam, K. J., & Sarker, U. (2026) · International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE) · doi:10.1109/icecte69292.2026.11429214 · Cited by 1
“Recent advancements in large language models (LLMs) have demonstrated remarkable progress in biomedical NLP [3].”
Rogić, S., Mancarci, B. O., Xu, B., Xiao, A., Yan, C., & Pavlidis, P. (2026) · Database · doi:10.1093/database/baag041 · Cited by 4
Davila-Garcia, D. M., Schuelke, M. J., & Wilcox, A. B. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf182 · Cited by 2
Kim, S., Lee, D. S. U., Kim, Y., Yoon, H., & Lee, H. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.104982 · Cited by 1
Truong, V. Q., Yang, S., Shen, L., & Ritchie, M. D. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356640
“Nonetheless, their application remains limited by high hallucination rates, outdated pretraining data, and lack of direct access to authoritative sources [11]–[15].”
A Knowledge-Guided Large Language Model Framework for Microbiome-Based Disease Diagnosis
Liu, C., Han, H.-Y., Qi, Y.-R., & Ling, W.-D. (2025) · IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm66473.2025.11356512 · Cited by 1
“These characteristics of LLMs make them not only powerful classifiers, but also knowledge-guided engines that help bridge the gap between empirical data and biological understanding [4], [5].”
Error-Aware Text-to-SQL Generation for Clinical Trial Eligibility Criteria Querying in EHR Databases
Kottam, S., Annem, S., & Sun, Y.-C. (2025) · BigData Congress [Services Society] · doi:10.1109/bigdata66926.2025.11402102 · Cited by 1
“Inspired by VerifiNER [22], our domain-specific verification system combines UML-S/OHDSI knowledge enrichment with LLM-based contextual reasoning to systematically correct entity mistakes that could otherwise exclude 30–40% of appropriate patients from clinical trials.”
Liu, J., Li, D. M., Ju, Q. R., & Zhang, X. (2025) · Asian Journal of Criminology · doi:10.1007/s11417-025-09475-z · Cited by 11
“Recent advances have shown that LLMs can accurately identify and classify domain-specific facts and entities in specialized texts, and even produce structured, organized outputs suitable for analysis (Chen et al., 2025; Dagdelen et al., 2024).”
Singh, S., & Singh, A. K. (2025) · International Conference on Signal Processing, Computation, Electronics, Power and Telecommunication (IConSCEPT) · doi:10.1109/iconscept66142.2025.11437317
Semi-Automatic Extraction and Analysis of Health Equity Covariates in Registered Research Projects
Nananukul, N., & Kejriwal, M. (2025) · Applied Sciences · doi:10.3390/app152211853 · Cited by 1
Githinji, P., Melliou, A., Liang, Z., Zhang, L., & Qin, P. (2025) · doi:10.48550/arxiv.2511.05080 · Cited by 1
“As for general-purpose LLMs, large architectural models such as GPT-4 and Llama3 70B exhibit superior performance or minimal numerical differences from domain-adapted alternatives, while small models underperform considerably (Feng et al., 2024; Dorfner et al., 2024; Chen et al., 2025).”
Li, L., Duan, Z., Li, X., Xu, X., Xue, Z., Ma, S., & Jin, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.00818 · Cited by 1
“…literature and knowledge mining (e.g., information extraction from electronic health records [Yang et al., 2022]), drug discovery and genomics (e.g., deciphering protein structures [Valentini et al., 2023]), and clinical diagnoses and decision support [Zhou et al., 2025, Chen et al., 2025].”
Talukdar, N., Zhang, X., Paithankar, S., Wang, H., & Chen, B. (2025) · Drug Discovery Today · doi:10.1016/j.drudis.2025.104552 · Cited by 2
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
Lu, Y., Zhao, X.-K., Tamo, J. B., Nnamdi, M. C., Peng, R., Zeng, S., Hu, X., Wang, J.-Z., & Wang, M. D. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2510.14944 · Cited by 1
“Current biomedical evaluation frameworks cannot answer these questions (Chen et al., 2025; Krithara et al., 2023; Jin et al., 2019).”
Leveraging large language models for rare disease named entity recognition
Xi, N. M., Deng, Y., & WANG, L. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001242 · Cited by 3
“These gaps are consistent with recent evaluations showing that general purpose LLMs can underperform fine-tuned domain models on token-level clinical NER under strict boundary matching, even with prompt engineering or few-shot prompting [18,19,42,43].”
González-Márquez, R., Berens, P., & Kobak, D. (2025) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2508.03453 · Cited by 1
“The BGE-base embedding model (Xiao et al., 2024) and its further modifications are currently best performing on MTEB among models of BERT-base size.”
Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions
Cook, R., Kahan, A. A., Scharfenberger, T., Tasoulas, J., Hawks-Ladds, N., Chouake, R. J., Jariwala, S. P., & Arora, S. (2026) · Applied Clinical Informatics · doi:10.1055/a-2835-4634
Shahriar, A., Shahriyar, R., & Rahman, M. S. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2505.20496 · Cited by 1
“Despite the emergence of LLMs, fine-tuned encoder models remain state-of-the-art in non-generative tasks like classification and NER (Edwards and Camacho-Collados, 2024; Sun et al., 2023; Chen et al., 2025).”
Large language models in aquatic risk assessment: research status and future perspectives
Li, Q., Cheng, F., & You, J. (2026) · Environmental and Biogeochemical Processes · doi:10.48130/ebp-0026-0002 · Cited by 1
Ji, H., Wang, C.-X., Li, Z.-H., Chen, R., Wu, X.-P., Huang, C.-Y., & Meng, Z.-L. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3693704 · Cited by 1
“In recent years, large language models (LLMs) have shown remarkable potential in natural language processing, including conversation, reasoning, and context generation [10], [11].”
VaxScope: Document-Level Structured Evidence Extraction from Immunization Systematic Reviews
Ilgen, B., Awotoro, E., & Hattab, G. (2026) · BioNLP@ACL · doi:10.18653/v1/2026.bionlp-1.69 · Cited by 1
“More recently, large language models have been applied to PICO extraction in zero-shot and few-shot settings (Ghosh et al., 2024; Chen et al., 2025), achieving competitive performance without requiring labeled training data.”
Zero-Shot Cross-Sentential Scientific Relation Extraction via Entity-Guided Summarization
Kanjirangat, V., & Rinaldi, F. (2025) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2025.wasp-main.4 · Cited by 1
“These limitations are especially pronounced in complex tasks like relation identification and causality detection(Armengol-Estapé et al., 2021; Khondaker et al., 2023; Lai et al., 2023; Yang et al., 2023; Bi et al., 2025; Chen et al., 2025).”
Reducing confirmation bias on controversial issues through LLM-generated multi-persona debates
Shi, L., Liu, H., Wong, Y., Mujumdar, U., Zhang, D., Jacek Gwizdka, A., & Lease, M. (2026) · Information Processing & Management · doi:10.48550/arxiv.2412.04629 · Cited by 27
Jeong, E., Jeong, W., Oh, K. H., & Jun, J. K. (2026) · Journal of Medical Internet Research · doi:10.2196/91101
Ay, B. K., & Yıldır, C. (2026) · Journal of Back and Musculoskeletal Rehabilitation · doi:10.1177/10538127261492778
Yu, S., Gu, X., Gao, Y., Zhao, F., & Zhang, J. (2026) · Integrated Diagnostics · doi:10.1016/j.intdia.2026.100006
Harrison, T. B., Hu, D., Jia, H., Tan, S. H., Chen, F., Zhang, Z., Lu, Q., Wang, J., Wang, L., Huang, M., et al. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001768
Lu, Y.-P., Huang, X.-X., Zhang, J.-F., & Xing, Y. (2026) · Journal of Hazardous Materials Advances · doi:10.1016/j.hazadv.2026.101666
Evaluating Large‐Language Models in Bioinformatics Applications
Yin, H., Cui, Z., Li, D., Yang, Y., Chang, Y., Zhu, X., Zhang, J., Hu, P., Zhou, X., & Hu, L. (2026) · Advanced Computing · doi:10.1002/adco.70007
Waterschoot, C., Tintarev, N., & Barile, F. (2026) · ACM Transactions on Recommender Systems · doi:10.1145/3848640
“To select which LLMs to include, we followed a standard protocol [11].”
Scaling Network Medicine with LLMs for Combinatorial Drug Repurposing in ER+ Breast Cancer
Hamed, A., Fandy, T. E., & Rocha, L. M. (2026) · bioRxiv · doi:10.64898/2026.09.14.728753
“…NLP benchmarks have characterized the strengths and limitations of large language models for entity recognition and relationship extraction tasks 36 , while workshop contributions and comprehensive reviews have examined LLMs and biomedical text-to-graph methods as means for knowledge graph…”
Fu, S., Kwak, M. J., Ahn, J., Lu, Q., Yue, Z., Wang, N., Liu, H., Giles, G., Cumming, A., Thomas, K. M., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00114-y
“…GPT-4o involves an iterative process and in consultation with domain experts in geriatrics and clinical NLP. Based on the annotation guidelines developed from the prior step, we developed a structured in-context prompt that incorporated best practices in few-shot learning from prior studies 39,45 .”
Self-citationCLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine
Wang, S., Zhao, Y.-Z., & Chen, Q.-Y. (2026) · doi:10.48550/arxiv.2609.16301
“MedBullets contains 308 USMLE Step 2 and Step 3-style multiple-choice questions with expert-written explanations, collected from the MedBullets platform Chen et al. [2025].”
InfluentialSelf-citationThe average-farmer illusion in language-model simulations of agricultural decisions
Zhu, Z.-L., Li, Z.-W., Liu, Y.-C., Zhu, L.-J., Wu, R.-Q., Shen, T.-Q., Jin, J.-L., & Zhang, J.-Y. (2026) · doi:10.48550/arxiv.2609.15038
“Prompt designs should therefore be evaluated for a specified decision, validation target and population rather than ranked on a single global leaderboard [46, 47].”
Zhang, W., Feng, Z., Liu, Y., Zhong, R., Chen, S., Li, Z., 李冰燕, Liu, D., Gao, L., & Hu, K. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1915669
Alharbi, N. M. (2026) · Frontiers in Microbiology · doi:10.3389/fmicb.2026.1873220
Razzaghi, H., Nguyen, N., Pargi, M. K., Wieand, K., Bunnell, H. T., & Bailey, L. C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag171
“22,23 LLMs are already increasingly deployed within health systems to leverage their capacity through discharge summary generation and ambient documentation, which reduces burden on clinicians.”
Architectural good practices for reproducible benchmarking in protein machine learning
Vinuesa, J. A. G. , Villegas, D. F., Soto-García, M., Cadet, X. F., Davari, M. D., Cadet, F., Asenjo, J. A., Uribe-Paredes, R., & Medina-Ortiz, D. (2026) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2026.1925740
“…stack, helping protein machine learning move toward workflows whose results can be independently reconstructed, verified, and interpreted in their biological context (Wilkinson et al., 2016; Chen et al., 2025; Boeckhout et al., 2018). archived in Zenodo at https://zenodo.org/records/21810167.”
Liu, Z.-J., Li, H.-X., Tan, Z., Duan, J., Huang, B.-X., Liu, Z., Shu, K., & Chen, T.-L. (2026) · doi:10.48550/arxiv.2609.06779
“In parallel, large language model (LLM)-based methods leverage pretrained knowledge to generate natural-language mechanistic rationales, such as causal chains, pathway-level explanations and alternative biological hypotheses (Singhal et al., 2023a; Chen et al., 2025a; Liu et al., 2025a).”
Patient-Centered Summarization Framework for AI Clinical Summarization: Mixed Methods Study
Jimenez, M. L., Claros, A., Green, K., Toro-Tobón, D., Larios, F., Asthana, S., Wenczenovicz, C., Maldonado, K. G., Vilatuna-Andrango, L., Proano, A. C., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/87061
“For example, recent studies generate structured clinical notes from patient-clinician conversations using LLMs and prioritize diagnoses, medications, and care plans for clinician use, while earlier approaches focused on compressing clinical records for professional documentation [40-42].”
Baumgärtel, F., Bono, E., Fillinger, L., Galou, L., Kęska, K., Walter, S. M., Andorfer, P., Kratochwill, K., Perco, P., & Ley, M. (2026) · iScience · doi:10.1016/j.isci.2026.117438
“Nonetheless, since automatic metrics and human ratings have not fully aligned in other settings, 5,6 we cannot formally guarantee that our metrics capture the full space of valid summaries, which remains an important direction for future work.”
Özden, Y., Yüzügülen, Ö., & Büyükberber, N. (2026) · BMJ Open · doi:10.1136/bmjopen-2026-118459
“10 Nevertheless, evidence supporting their use for endoscopy-related AE surveillance remains limited, particularly in non-English clinical settings, and few studies have evaluated performance against a clinician-adjudicated reference standard.”
Biomedical word sense disambiguation based on chi-square attention and dynamic hard sample mining
Zhang, C.-X., Zhang, H.-B., Gao, X.-Y., & Xue, Y.-Z. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116559
Visual LLM-guided consensus spatial domain detection with L-STAR
Zhao, C.-Y., & Ji, Z.-C. (2026) · bioRxiv · doi:10.64898/2026.08.25.747158
“Large language models (LLMs) have been shown to be broadly applicable in biomedical research 10–12 .”
Exploring Cognitive Load Markers in Construction Fatality Reports: An LLM-Based Approach
Adjei, E., Jin, Z., & Gambatese, J. (2026) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784486993.067
Oh, J., & Ashuri, B. (2026) · Construction Research Congress 2026 · doi:10.1061/9780784486986.041
Metadata
- Authors
- Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., Keloth, V. K., Raja, K., Huang, J., He, H., Lin, F., Du, J., Zhang, R., Zheng, W. J., Adelman, R. A., Lu, Z., & Xu, H.
- Affiliations
- National Institutes of Health (US)
- The University of Texas Health Science Center at Houston (US)
- Yale University (US)
- University of Minnesota (US)
- Corresponding author
- Lu, Zhiyong
- Xu, Hua
- ORCID
- Venue
- Nature Communications
- Published
- 2025-04-06
- Volume
- 16
- Issue
- 1
- Pages
- 3280-3280
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11972378
- OpenAlex
- W4409190866
- Cited by
- 219
- References
- 103
- Citations by year
- 2026: 116 · 2025: 76
- FWCI
- 57.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (1K99LM01402)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, biomedical natural language processing, benchmarking, GPT-4, llamas, fine-tuning, zero-shot learning, few-shot learning, hallucinations, medical question answering, BERT, BART, cost analysis
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
