Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 601–700 of 1,030
A generalizable deep learning system for cardiac MRI
Shad, R., Zakka, C., Kaur, D., Mathur, M., Fong, R., Cho, J., Filice, R. W., Mongan, J., Kallianos, K., Khandwala, N., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01637-3 · Cited by 22
“We tested a combination of different pre-trained weights including those from the original publication, weights fine tuned on the MIMIC dataset, and weights from a model trained on biomedical abstracts from PubMed with a custom vocabulary of 30,522 tokens 79,80 (Supplementary Fig.”
The Development Landscape of Large Language Models for Biomedical Applications
Cao, Z., Keloth, V. K., Xie, Q., Qian, L., Liu, Y., Wang, Y., Shi, R., Zhou, W., Yang, G. F., Zhang, J., et al. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102224-074736 · Cited by 21
Khan, M. Z., Ge, Y., Mollel, M. S., McCann, J. A., Abbasi, Q. H., & Imran, M. A. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3558069 · Cited by 21
“The system architecture implements optimized signal processing techniques for RF data analysis [40], while leveraging modern developments in multi-modal retrieval systems [47].”
Extracting epilepsy‐related information from unstructured clinic letters using large language models
Fang, S., Holgate, B., Shek, A., Winston, J. S., McWilliam, M., Viana, P. F., Teo, J., & Richardson, M. P. (2025) · Epilepsia · doi:10.1111/epi.18475 · Cited by 19
MTMG: A multi-task model with multi-granularity information for drug-drug interaction extraction
Deng, H., Li, Q., Liu, Y., & Zhu, J. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16819 · Cited by 17
EMSAssist: An End-to-End Mobile Voice Assistant at the Edge for Emergency Medical Services
Jin, L., Liu, T., Haroon, A., Stoleru, R., Middleton, M., Zhu, Z., & Chaspari, T. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596853 · Cited by 24
Supplementing domain knowledge to BERT with semi-structured information of documents
Chen, J., Wei, Z., Wang, J., Wang, R., Gong, C., Zhang, H., & Miao, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121054 · Cited by 18
Zhou, H., Austin, R. R., Lu, S.-C., Silverman, G., Zhou, Y., Kilicoglu, H., Xu, H., & Zhang, R. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad216 · Cited by 19
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Zhang, Y., Cheng, H., Shen, Z., Liu, X., Wang, Y., & Gao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.820 · Cited by 28
Self-citationExplainable Fake News Detection Based on BERT and SHAP Applied to COVID-19
Men, X., & Mariano, V. Y. (2024) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2024.01.02 · Cited by 21
Semantic-Oriented Visual Prompt Learning for Diabetic Retinopathy Grading on Fundus Images
Zhang, Y., Ma, X., Huang, K., Li, M., & Heng, P. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3383827 · Cited by 21
“Besides, BERT-derived and GPT-derived models have been extensively studied in medical language processing and gained improved performance over prior methods [40], [42], [43].”
MethodPre-trained language models for keyphrase prediction: A review
Umair, M., Sultana, T., & Lee, Y.-K. (2024) · ICT Express · doi:10.1016/j.icte.2024.05.015 · Cited by 17
“The second subcategory is academic, A number of existing works adopted PLMs [138, 139, 140, 141, 142, 143].”
BackgroundUsing large language model (LLM) to identify high-burden informal caregivers in long-term care
Chien, S.-C., Yen, C.-M., Chang, Y.-H., Chen, Y., Liu, C.-C., Hsiao, Y.-P., Yang, P.-Y., Lin, H.-M., Yang, T.-E., Lu, X., et al. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108329 · Cited by 17
Steurer, B., Vanhaelen, Q., & Zhavoronkov, A. A. (2024) · The Journals of Gerontology Series A · doi:10.1093/gerona/glae006 · Cited by 18
“LLMs trained on scientific literature, biomedical, and clinical records have been released with BioMegatron (43), SciBERT (44), PubMedBERT (45), BioBERT (46), BioNLP (47), and GatorTron (48), to name a few (Table 1).”
MethodHamane, Z., Ben-Bouazza, F.-E., & Fennan, A. (2024) · International Conference on Intelligent Systems and Computer Vision (ISCV) · doi:10.1109/iscv60512.2024.10620139 · Cited by 22
BioMamba: Domain-Adaptive Biomedical Language Models
Yue, L., Zhu, M., Xing, S., Cao, Y., Wang, Y., Chenthamarakshan, V., Pan, S., Das, P., & Fu, T. (2026) · Health Data Science · doi:10.34133/hds.0454 · Cited by 16
Natural Language-Assisted Multi-modal Medication Recommendation
Tan, J., Rong, Y., Zhao, K., Bian, T., Xu, T., Huang, J., Cheng, H., & Meng, H. M. L. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679529 · Cited by 24
Self-citationXiao, L., Zhang, Y. Q., Wang, D., Zhong, H., Li, R., & Wang, Q. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681201 · Cited by 26
LCD benchmark: long clinical document benchmark on mortality prediction for language models
Yoon, W., Chen, S., Gao, Y., Zhao, Z., Dligach, D., Bitterman, D. S., Afshar, M., & Miller, T. A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae287 · Cited by 25
Foundation Models: From Current Developments, Challenges, and Risks to Future Opportunities
Hussain, A., Ali, S., Farwa, U. E., Mozumder, M. A. I., & Kim, H.-C. (2025) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact63878.2025.10936649 · Cited by 18
Zhao, D., Mu, W., Jia, X., Liu, S., Chu, Y., Meng, J., & Lin, H. (2025) · BioData Mining · doi:10.1186/s13040-025-00443-y · Cited by 16
Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks
Zhang, K., He, H., Song, S., Zhang, J., & Letaief, K. B. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3581478 · Cited by 26
“The efficacy of LLMs is primarily attributed to the vast model scale with billions of parameters, which enables them to capture complex semantic relationships and contextual nuances, leading to superior performance across diverse tasks.”
Multi-granularity sequential neural network for document-level biomedical relation extraction
Liu, X., Tan, K., & Dong, S. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102718 · Cited by 16
Self-citationPatricoski, J., Johns Hopkins Molecular Tumor Board Investigators, Kreimeyer, K., Balan, A., Hardart, K., Tao, J., Anagnostou, V., & Botsis, T. (2022) · Studies in health technology and informatics · doi:10.3233/shti210848 · Cited by 19
“Each of the BioBERT [2], BlueBERT [3], Clinical BioBERT [4], SciBERT [4], PubMedBERT [5], and CODER [6] models created embeddings for and checked the cosine similarity between pair titles and between pair EC.”
MethodJin, X., Sun, X., Chen, J., & Sutcliffe, R. F. E. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557318 · Cited by 17
“Group Two consists of PubMedBERT plus one feature (with three choices).”
ResultInfluentialWang, X.-Q., Xu, X., Tong, W., Liu, Q., & Liu, Z.-C. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.999289 · Cited by 14
“Some reports have demonstrated that domain-specific pre-training is a solid foundation for a wide range of downstream domain-specialized NLP tasks (Gu et al., 2021).”
BackgroundZhao, S., & Birbilis, N. (2023) · npj Materials Degradation · doi:10.1038/s41529-022-00319-0 · Cited by 16
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 31
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive BenchmarkPreprint
Liu, F., Li, Z., Zhou, H., Yin, Q., Yang, J., Tang, X., Luo, C., Zeng, M. Y., Jiang, H., Gao, Y., et al. (2024) · medRxiv · doi:10.1101/2024.04.24.24306315 · Cited by 17
“As shown in Table 1, BenchHealth encompasses three different evaluation scenarios (i.e., reasoning, generation, and understanding) and includes seven popular downstream tasks and thirteen representative datasets; Previous popular benchmarks, e.g., BLUE (Peng et al., 2019) and BLURB (Gu et al., 2021), only include the…”
MethodChen, Y.-H., Ruan, S.-J., & Chen, P. (2025) · Journal of Medical Internet Research · doi:10.2196/75052 · Cited by 15
“Specifically, we explored: • Embedding models: MedEmbed [39] and PubMedBERT [40] • Chunk sizes: 150, 200, 250, 300, and 350 The goal of these experiments was to systematically identify the most effective parameter combinations for mortality prediction and ASA classification, ensuring a fair…”
ProtGO: universal protein function prediction utilizing multi-modal gene ontology knowledge
Wang, B., Geng, Y.-A., Cheng, X., Chen, B., Bei, Z., Wang, W., Tang, J., & Song, L. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf390 · Cited by 15
“…first merges the textual representation with the protein sequence representation using PLMs. Owing to any biological and biomedical pre-trained LMs (Gu et al. 2022, Yasunaga et al. 2022), informative representations can be extracted from the text descriptions about protein function (Van Remmen et…”
Question Answering models for information extraction from perovskite materials science literature
Sipilä, M., Mehryary, F., Pyysalo, S., Ginter, F., & Todorović, M. (2025) · Communications Materials · doi:10.1038/s43246-025-00979-w · Cited by 15
“Similarly, 12,843 articles from the Core API lacked a DOI and were labelled by the Core id .”
Applying Artificial Intelligence and machine learning in precision nutrition
Massara, P., Kirkland, J., Pagani, I., Huey, S. L., Hirsh, H., McDonald, D. T., Patel, L., Finkelstein, J. L., Gantz, M. G., Wang, F., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-75004-w · Cited by 16
An Experimental Study on Pretraining Transformers from Scratch for IR
Lassance, C., Déjean, H., & Clinchant, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_32 · Cited by 15
Zhu, C., Xia, X., Li, N., Zhong, F., Yang, Z., & Liu, L. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107262 · Cited by 16
Yang, C., Deng, J., Chen, X., & An, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104456 · Cited by 15
Text Classification of Patient Experience Comments in Saudi Dialect Using Deep Learning Techniques
Alhazzani, N. Z., Al-Turaiki, I. M., & Alkhodair, S. A. (2023) · Applied Sciences · doi:10.3390/app131810305 · Cited by 18
Moodaley, W., & Telukdarie, A. (2023) · European Journal of Sustainable Development · doi:10.14207/ejsd.2023.v12n4p319 · Cited by 15
“Other examples include MatBERT, pretrained on the materials science domain; ClimateBert, pre-trained on the climate-related domain, and PubmedBert pre-trained on biomedical domain (Huang et al., 2023) (Gu et al., 2021) (Webersinke et al., 2022).”
Bautista, Y. J. P., Theran, C., Aló, R., & Lima, V. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47454-5_17 · Cited by 18
Using BERT to Identify Causal Structure in Students’ Scientific Explanations
Cochran, K., Cohn, C., Hastings, P., Tomuro, N., & Hughes, S. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00373-y · Cited by 15
BioBBC: a multi-feature model that enhances the detection of biomedical entities
Alamro, H., Gojobori, T., Essack, M., & Gao, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-58334-x · Cited by 17
Du, J., Soysal, E., Wang, D., He, L., Lin, B., Wang, J., Manion, F. J., Li, Y., Wu, E., & Yao, L. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02224-3 · Cited by 15
“This model has achieved state-of-the-art performance on several biomedical NLP tasks on Biomedical Language Understanding and Reasoning Benchmark [14].”
MethodSemantic Textual Similarity Analysis of Clinical Text in the Era of LLM
Feng, Y. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00227 · Cited by 18
Artificial Intelligence in Indonesian Ports: Opportunities and Challenges
Safuan, S., & Syafira, A. (2024) · Transactions on Maritime Science · doi:10.7225/toms.v13.n02.w07 · Cited by 15
Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
Ndimbo, E. V., Luo, Q., Fernando, G., Xu, Y., & Wang, B. (2025) · Applied Sciences · doi:10.3390/app15020524 · Cited by 24
ARCH: Large-scale knowledge graph via aggregated narrative codified health records analysis
Gan, Z., Zhou, D., Rush, E. N., Panickan, V. A., Ho, Y.-L. A., Ostrouchovm, G., Xu, Z., Shen, S., Xiong, X., Greco, K. F., et al. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104761 · Cited by 23
Manzini, E., Vlacho, B., Franch‐Nadal, J., Escudero, J., Génova, A., Reixach, E., Andrés, E., Pizarro, I., Mauricio, D., & Perera-Lluna, A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126876 · Cited by 18
The DRAGON benchmark for clinical NLP
Bosma, J. S., Dercksen, K., Builtjes, L., André, R., Roest, C., Fransen, S. J., Noordman, C. R., Navarro-Padilla, M., Lefkes, J., Alves, N., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01626-x · Cited by 23
Du, J., Wang, D., Lin, B., He, L., Huang, L., Wang, J., Manion, F. J., Li, Y., Cossrow, N., & Yao, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-03979-5 · Cited by 21
“We evaluated four variants of BERT models, including Clinical BERT 21 , BioBERT 23 , BLUEBERT 24 and PubMedBERT 25 , for full-text data element extraction.”
Kollapally, N. M., Geller, J., Keloth, V. K., He, Z., & Xu, J. Z. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104865 · Cited by 15
Zhou, D., Tong, H., Wang, L., Liu, S., Xiong, X., Gan, Z., Griffier, R., Hejblum, B. P., Liu, Y., Hong, C., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-71152-1 · Cited by 21
Benchmarking of Transformer-Based Pre-Trained Models on Social Media Text Classification Datasets
Guo, Y., Dong, X., Al-Garadi, M. A., Sarker, A., Paris, C. L., & Mollá, D. (2020) · Cited by 33
Representing a Heterogeneous Pharmaceutical Knowledge-Graph with Textual Information
Asada, M., Gunasekaran, N., Miwa, M., & Sasaki, Y. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.670206 · Cited by 19
Named Entity Recognition of Medical Text Based on the Deep Neural Network
Yang, T., He, Y., & Yang, N. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/3990563 · Cited by 15
RetractedAn Italian lexicon-based sentiment analysis approach for medical applications
Martinis, M. C., Zucco, C., & Cannataro, M. (2022) · ACM International Conference on Bioinformatics, Computational Biology and Biomedicine · doi:10.1145/3535508.3545594 · Cited by 7
“Several pre-trained language models have been also proposed in the Biomedical field, such as BioBert [13], ClinicalBert [9], MedBert [17], PubMedBert [8].”
MethodFrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain
Labrak, Y., Bazoge, A., Dufour, R., Daille, B., Gourraud, P., Morin, E., & Rouvier, M. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.5 · Cited by 47
Chen, P., He, T.-L., Lin, S.-C., Chu, Y.-C., Kuo, C.-T., Lai, F., Wang, S.-M., Zhu, W.-X., Chen, K., Kuo, L.-C., et al. (2022) · JMIR Medical Informatics · doi:10.2196/41342 · Cited by 16
“We compared the performance of different variants of BERT, including PubMedBERT [14], RoBERTa (Robustly Optimized BERT Pretraining Approach) [15], ClinicalBERT [16], and BioBERT (BERT for Biomedical Text Mining) [17].”
MethodChemical–protein relation extraction with ensembles of carefully tuned pretrained language models
Weber, L., Sänger, M., Garda, S., Barth, F., Alt, C., & Leser, U. (2022) · Database · doi:10.1093/database/baac098 · Cited by 15
“A body of work has addressed the assessment of such language models on biomedical texts and made their models publicly available for further research (10 , 13).”
BackgroundLeaman, R., Islamaj, R., Adams, V., Alliheedi, M., Almeida, J. R., Antunes, R., Bevan, R., Chang, Y., Erdengasileng, A., Hodgskiss, M., et al. (2023) · Database · doi:10.1093/database/baad005 · Cited by 18
Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
Wang, Y., Li, J., Naumann, T., Xiong, C., Cheng, H., Tinn, R., Wong, C., Usuyama, N., Rogahn, R., Shen, Z., et al. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3469053 · Cited by 20
“To further assess the impact from domain-specific pretraining, we also conducted continual pretraining using CORD19 for 100K BERT steps and evaluated it in our biomedical search system (PubMedBERT-COVID).”
BackgroundMethodInfluentialSelf-citationDobbins, N. J., Mullen, T., Uzuner, Ö., & Yetişgen, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01521-0 · Cited by 15
“For BERT-based prediction, we used two pretrained models trained on published medical texts, SciBERT [22] and PubMedBERT [23].”
MethodEnzChemRED, a rich enzyme chemistry relation extraction dataset
Lai, P., Coudert, E., Aimo, L., Axelsen, K. B., Breuza, L., de Castro, E., Feuermann, M., Morgat, A., Pourcel, L., Pedruzzi, I., et al. (2024) · Scientific Data · doi:10.1038/s41597-024-03835-7 · Cited by 17
“We performed relation classification using PubMedBERT and BioREx 61 , which is a PubMedBERT model trained on the BioRED dataset and eight other common biomedical RE benchmark datasets (PubMedBERT is essentially the same model but without this additional training step).”
InfluentialFine-tuning protein language models unlocks the potential of underrepresented viral proteomes
Sawhney, R., Ferrell, B. D., Dejean, T., Schreiber, Z., Harrigan, W., Polson, S. W., Wommack, K. E., & Belcaid, M. (2025) · PeerJ · doi:10.7717/peerj.19919 · Cited by 14
“…studies suggest this approach has been successful in natural language processing and proteomics ( Ding & Steinhardt, 2024 ; Elnaggar et al., 2021 ; Rao et al., 2019 ; Gu et al., 2021 ; Brandes et al., 2022 ; Rives et al., 2021 ; Howard & Ruder, 2018 ; Devlin et al., 2019 ; Houlsby et al., 2019 ).”
Park, P., Choi, Y., Han, N., Park, Y.-L., Hwang, J., Chae, H., Yoo, C. W., Choi, K.-S., & Kim, H.-J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318726 · Cited by 12
Biomedical document relation extraction with prompt learning and KNN
Zhao, D., Yang, Y., Chen, P., Meng, J., Sun, S., Wang, J., & Lin, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104459 · Cited by 14
Sangariyavanich, E., Ponthongmak, W., Tansawet, A., Theera-Ampornpunt, N., Numthavaj, P., McKay, G. J., Attia, J., & Thakkinstian, A. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101326 · Cited by 14
“Applying health domain-specific pre-trained models, such as PubMedBERT [54], ClinicalBERT [56], and BioBERT [55], should help mitigate these concerns.”
BackgroundMethodInfluentialChen, L., Qi, Y., Wu, A., Deng, L., & Jiang, T. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3315143 · Cited by 17
Zhang, Z., Li, Z., Zhu, J., Guo, Z., Shi, B., & Hu, B. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111650 · Cited by 14
MolTailor: Tailoring Chemical Molecular Representation to Specific Tasks via Text Prompts
Guo, H., Zhao, S., Wang, H., Du, Y., & Qin, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29772 · Cited by 22
Yuan, J., Zhang, F., Qiu, Y., Lin, H., & Zhang, Y. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae418 · Cited by 15
“For BioRED dataset, we adhere to the setup of Luo et al. (2022), utilizing BERT-GT (Lai and Lu 2020) and PubMedBERT (Gu et al. 2021) as comparison models.”
Lu, Q., Wen, A., Nguyen, T. H., & Liu, H. (2024) · JMIR AI · doi:10.2196/56932 · Cited by 15
“In the last few years, several domain-specific PLMs have been proposed to mitigate the issue, for example, BioBERT [16], ClinicalBERT [12], ClinicalBERT [4], PubMedBERT [17], ClinicalT5 [18], etc.”
BackgroundBiomedical Flat and Nested Named Entity Recognition: Methods, Challenges, and Advances
Park, Y., Son, G., & Rho, M. (2024) · Applied Sciences · doi:10.3390/app14209302 · Cited by 19
BioKGrapher: Initial evaluation of automated knowledge graph construction from biomedical literature
Schäfer, H., Idrissi-Yaghir, A., Arzideh, K., Damm, H., Pakull, T. M. G., Schmidt, C. S., Bahn, M., Lodde, G. C., Livingstone, E., Schadendorf, D., et al. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.017 · Cited by 19
SNOBERT: A Benchmark for Clinical Notes Entity Linking in the SNOMED CT Clinical Terminology
Kulyabin, M., Sokolov, G., Galaida, A., Maier, A., & Arias‐Vergara, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78119-3_11 · Cited by 14
Robust multi-modal fusion architecture for medical data with knowledge distillation
Wang, M., Fan, S., Li, Y., Gao, B., Xie, Z., & Chen, H. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108568 · Cited by 17
Thetbanthad, P., Sathanarugsawait, B., & Praneetpolgrang, P. (2025) · Journal of Imaging · doi:10.3390/jimaging11010011 · Cited by 20
“While these models provide powerful tools for healthcare professionals, their integration into workflows must complement, not replace, expert judgment [47,48].”
Liang, M. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251314556 · Cited by 19
NLP modeling recommendations for restricted data availability in clinical settings
Villena, F., Bravo-Márquez, F., & Dunstan, J. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02948-2 · Cited by 19
Reviewing clinical knowledge in medical large language models: Training and beyond
Li, Q. Y., Liu, H., Guo, C., Gao, C., Chen, D., Wang, M., Gao, F., van Harmelen, F., & Gu, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114215 · Cited by 19
Pandey, S. R., Tile, J. D., & Oghaz, M. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328848 · Cited by 17
“In this study, we fine-tuned the PubMedBERT transformer model [53] on the dataset’s text and diagnosis descriptions for comparison.”
Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
Sun, S. Y., Qian, Y., Liu, Z., Xiong, C., Zhang, K., Bao, J., Liu, Z., & Bennett, P. N. (2021) · doi:10.18653/v1/2021.acl-long.390 · Cited by 23
Information Retrieval Using Machine Learning for Biomarker Curation in the Exposome-Explorer
Lamúrias, A., Jesus, S., Neveu, V., Salek, R. M., & Couto, F. M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.689264 · Cited by 15
Modular Self-Supervision for Document-Level Relation Extraction
Zhang, S., Wong, C., Usuyama, N., Jain, S., Naumann, T., & Poon, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.429 · Cited by 14
Self-citationLamproudis, A., Henriksson, A., & Dalianis, H. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_090 · Cited by 14
SentiMedQAer: A Transfer Learning-Based Sentiment-Aware Model for Biomedical Question Answering
Zhu, X., Chen, Y., Gu, Y., & Xiao, Z. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.773329 · Cited by 15
“(2021), BioELECTRA achieved SOTA on all of the 13 tasks in the BLURB benchmark (Gu et al., 2020), and PubMedQA was one of the datasets.”
MethodSelf-citationBlanco, A., Remmer, S., Pérez, A., Dalianis, H., & Casillas, A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104050 · Cited by 18
DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling
Zhuang, Y., Yu, Y., Kong, L., Chen, X., & Zhang, C. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599318 · Cited by 22
“We conducted a comprehensive study through over 1,500 experimental trials of fine-tuning various PLMs, including BERT [10], BioBERT [23], PubMedBERT [12], and RoBERTa [30].”
MethodTang, X., Tran, A., Tan, J. T. C., & Gerstein, M. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.58 · Cited by 19
BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning
Kanakarajan, K. R., Kundumani, B., Abraham, A. N., & Sankarasubbu, M. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.10 · Cited by 27
Zhang, Y., Li, X., Liu, Y., Li, A., Yang, X., & Tang, X. (2023) · JMIR Medical Informatics · doi:10.2196/44892 · Cited by 14
“Typical instances are BioBERT pretrained on PubMed citations and PubMed Central (PMC) full text [39] and PubMedBERT obtained with mixed-domain pretraining using PubMed text and clinical notes [40].”
MethodExploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition
Belkadi, S., Han, L., Wu, Y., & Nenadic, G. (2022) · BigData Congress [Services Society] · doi:10.1109/bigdata59044.2023.10386154 · Cited by 10
“However, ongoing investigations try to understand the extent to which fine-tuning PLMs increases performances against language models trained from scratch on domain-specific tasks [11].”
BackgroundMethodLitake, O., Park, B. H., Tully, J. L., & Gabriel, R. A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae081 · Cited by 14
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 25
“The application of language models to the field of medicine has been heavily studied [69, 70, 71, 72, 73, 74].”
BackgroundDeveloping a general-purpose clinical language inference model from a large corpus of clinical notesPreprint
Sushil, M., Ludwig, D., Butte, A. J., & Rudrapatna, V. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06566 · Cited by 19
Aligning Large Language Models for Clinical TasksPreprint
Manathunga, S. S., & Hettigoda, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02884 · Cited by 15
Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical FieldPreprint
Kerner, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.14076 · Cited by 18
“This is only possible if there is enough data left in the domain-specific dataset for the model to get an understanding of natural language from it while not overfitting [1].”
BackgroundInfluentialrbio1 - training scientific reasoning LLMs with biological world models as soft verifiersPreprint
Istrate, A.-M., Milletarì, F., Castrotorres, F., Tomczak, J. M., Torkar, M., Li, D., & Karaletsos, T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.18.670981 · Cited by 27
AInsteinBench: Benchmarking Coding Agents on Scientific RepositoriesPreprint
Duston, T., Xin, S., Sun, Y., Zan, D., Li, A., Xin, S., Shen, K., Chen, Y., Sun, Q., Zhang, G., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.21373 · Cited by 14
“Early efforts emphasized scientific and biomedical language modeling via in-domain pre-training (e.g., SciBERT, BioBERT, PubMedBERT), improving scientific NLP as a substrate for downstream reasoning and knowledge use [28–30].”
Deep learning for automatic ICD coding: Review, opportunities and challenges
Li, X., Zhang, Y., Hou, X., Wang, S., & Lin, H. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103187 · Cited by 17
Empowering biological knowledgebases: advances in human-in-the-loop AI-driven literature curation
Wood, V., Jeffryes, M., Green, A., Blum, M., Orchard, S. E., Panni, S., Quaglia, F., Rodriguez‐Esteban, R., Seager, J., Tosatto, S. C. E., et al. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag028 · Cited by 15
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.