Benchmarking large language models for biomedical natural language processing applications and recommendations

“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”

“For the tasks where the gold standard is fi xed, e.g., a classi fi cation type or multiple-choice option, we conducted qualitative evaluations on collectively hundreds of thousands of raw outputs of the LLMs (the raw outputs from three LLMs under zero-and one-shot conditions acrossthree benchmarks)…”

“Since the inception of transformers, biomedical language models have adopted their architecture, and can be categorized into (1) encoder-based, masked language models using the encoder from the transformer architecture such as the biomedical bidirectional encoder representations from transformers (BERT) family includi…”

“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”

“Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.”

“…instances withless than 50words forthe trainingandvalidationsets andkeptthetesting setuntouched. b The gold standardofthetesting setofMS^2 isnot publicly available;weused thevalidation setinstead. by the fi ne-tuning approaches to quantify the bene fi ts and challenges of LLMs in BioNLP applications.”

“For instance, Bubeck et al. studied the ability of GPT-4 in a wide spectrum, such as coding, mathematics, and interactions with humans.”

“This is also supported by the existing studies on medical question answering 38,46 .”

“These limitations reinforce the need for curated annotated resources that support supervised training and domain-specific evaluation rather than relying on generic genAI prompting alone (Chen et al. 2025, Miao et al. 2025).”

“…of biomedical applications, including literature summarization and clinical decision support, due to their ability to synthesize complex information and respond flexibly to natural-language questions (Hager et al. 2024, Li et al. 2024, Williams et al. 2024, Chen et al. 2025, Wang et al. 2025).”

“Recent comparative studies indicate that such adaptation does not consistently outperform general-purpose language models on previously unseen biomedical reasoning tasks [23,24].”

“Performance on English benchmarks does not establish whether a model can interpret medical concepts or communicate them reliably in other languages, especially those underrepresented in training data and existing evaluations (Chen et al., 2025b; Singhal et al., 2025).”

“Finetuning produced the largest improvements for smaller models, suggesting that task-specific training is particularly beneficial when base model performance is limited (Chen et al. 2025).”

“To improve performance for certain tasks in specialised fields, fine-tuning BERT [10] or BART [11] models and, more recently, open-weights generative AI models, with domain-specific data has become popular practice [12].”

“Other previous work with LLMs showed promising results for a range of data annotation and information extraction tasks (Gue et al. 2024; Mahmoudi et al. 2024; Chen et al. 2025; Jensen et al. 2025; Kainer 2025; Poretsky et al. 2025; Riquelme-García et al. 2025; Turner et al. 2025).”

“This result further suggests that zero-shot reasoning alone may be insufficient to fully capture the value-related structure underlying users’ attitude responses, while LLMs that are further adapted to the task may provide more competitive performance under a more comparable setting [34].”

“BERT and its biomedical variants including BioBERT and ClinicalBERT, excel in understanding ophthalmic narratives (30–32).”

“The present evaluation used one institution’s notes, which limits external validity but also avoids the test-set contamination that affects public corpora likely present in model pretraining [13, 19].”

“The combination of learned knowledge and prompt-conditioned generation enables LLMs to perform a wide range of tasks without task-specific training [3].”

“We require computational frameworks that are both safe and domain-adapted [8,9].”

“Recent benchmarking studies reveal that traditional fine-tuning approaches generally outperform zero-shot or few-shot LLM applications for most biomedical NLP tasks, though closed-source models like GPT-4 excel in reasoning-related tasks such as medical question answering [11].”

“LLMs have emerged as powerful tools for biomedical natural language processing, supporting tasks such as text summarization, question answering, and relation extraction [19], [20], [21].”

“However, recent studies suggest that generative LLMs do not consistently outperform strong fine-tuned encoder-based biomedical models on information-extraction tasks, although they can be competitive on reasoning-oriented tasks such as question answering [22,38,39].”

“As a core direction in the field of artificial intelligence, Natural Language Processing (NLP) [1–6] aims to enable computers to understand and generate human language.”

“As these linguistic mechanisms recur across domains, they may support a more transferable basis for modeling causality than approaches tied primarily to domain-specific ontologies or knowledge bases (Feder et al., 2022; Singhal et al., 2023; Chen et al., 2025; Lai et al., 2023).”

“Internet penetration in Indonesia reached 79.5% in 2024 with 221 million users, and is predicted to increase to 80.66% or 229.4 million by 2026, indicating that the majority of the population relies more on digital platforms to seek information than conventional methods (Chen et al., 2025).”

“1-mini scores generated outputs on a 1–5 Likert scale across three dimensions: factual correctness, clinical relevance, and reasoning quality [63].”

“However, their ca-* pabilities in biomedical domains remain limited, which hinders their deployment in high-stakes, real-world biomedical applications (Chen et al., 2025; Li et al., 2025a).”

“This finding aligns with previous reports that RAG frameworks maintain stable performance across model scales while markedly improving computational efficiency [9, 25].”

“In biomedical NLP, Chen et al. [9] showed that zero-shot GPT and LLaMA models consistently underperformed compared to domain-trained models like BioBERT across most benchmarks, particularly in information extraction.”

“Critically, these evaluations have identified persistent challenges including information inconsistencies, missing content, and hallucinations in LLM outputs, underscoring the necessity of careful validation even when these models are used for literature screening (Chen et al., 2025).”

“Phenotype research can be classified as the following categories based on applications scenarios, such as public health phenotyping and personalized health phenotyping Zhang et al. (2025) (2024, 2026); Chen et al. (2025).”

“However, such finetuning approaches typically require substantial computational resources for deploying and training, making them costly and even infeasible when handling some closed-source models whose parameters are not accessible [14, 24, 37, 65].”

“Moreover, investigating ADE detection from clinical free-text documents in non-English settings is crucial, given the significant bias toward English-language data [10].”

“For instance, Chen et al. demonstrated that while GPT-4 approaches state-of-the-art performance in question answering, it significantly lags behind task-specific fine-tuned models in precision-demanding tasks such as information extraction 16 .”

“As per the (RAG) framework, the system can have the access to real time accessibility which is trusted, on accessing the outside of the medical related documents [11][12].”

“Recent evaluations have benchmarked LLMs including GPT-4 and domain-specific models on biomedical tasks such as named entity recognition and relation extraction [ 7 , 8 ].”

“As bioinformatics workflows grow more complex, domain-trained large language models will become essential tools for accelerating discovery while maintaining scientific rigor [25].”

“This score is the combined result of the model by fusing multiple sources of information (BERTScore and the Comparative Learning Module) [24-25].”

“Natural Language Processing emerges as an utmost crucial part with each software program, particularly biomedical researches in the present era (Chen et al. 2025 ; El-Hassan how crucial it is to have tools that are designed to scale, developed specifically for a specialized domain, as seen with…”

“High demands for fine-tuning, the enormous computational cost of training specialized models, and significant resource constraints have collectively limited the development of LLMs in bioinformatics [82].”

“This was due to the input context exceeding the allowed input tokens for the model, which was 4096 tokens [7].”

“…reach human-level extraction performance from few examples [28] and that local open-source deployment mitigates the privacy risks of sending notes to external services [29], while benchmark studies find that fine-tuning still tends to outperform zero or few-shot prompting for extraction tasks [30].”

“While previous study (Chen et al., 2025) does not directly investigate the use of LLM in extracting gene-disease-phenotype relationship, it applies relation extraction on datasets e.g. ChemProt and DDI2013, which involves identifying relationships between protein-protein and drug-drug interactions…”

“In the realm of science and biomedical research, recent studies such as DARE[29], PubMedBERT[32], SciBERT[14], BioNLP[3], BioBERT[22], ScholarBERT[12], and BioGPT[26] have demonstrated the efficacy of using well-curated scientific and biomedical datasets for language modeling, both for classification and generation ta…”

“Some studies in this field employ pre-trained models to recognize specific information within texts, achieving accuracies ranging from 68% to 95% [3], [4], [5], [6].”

“Biomedical LLMs improve medical question answering but show mixed gains on information extraction tasks compared to general-domain counterparts (Chen et al., 2025; Garcia-Gasulla et al., 2025).”

“Furthermore, recent benchmarking studies have evaluated the performance of LLMs across a wide range of biomedical NLP tasks, demonstrating their effectiveness in tasks such as biomedical question answering, document classification, and clinical summarization [3], [32], [33].”

“In English, comprehensive benchmarks such as BLURB (Gu et al., 2021) and recent LLM evaluations (Chen et al., 2025) have established baselines for NER, relation extraction, classification, and answer questions.”

“More recently, Chen et al. [65] compared GPT-3.”

“Consequently, lightweight adaptation strategies , such as parameter-efficient fine-tuning (e.g., LoRA) , and the use of smaller language models (sLMs) have gains prominence as practical alternatives [46], [49], [71], [72].”

“In contrast, tasks requiring complex reasoning, such as CCA, MR, and NER remain challenging for all models, consistent with findings from prior work (Chen et al., 2025; Liu et al., 2024; Boyle et al., 2023).”

“Chen et al. [5] compared the natural language processing capabilities of large language models (LLMs) in biomedicine, but did not involve knowledge graph construction in rehabilitation science.”