Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 901–1,000 of 1,034
Discovering Better Model Architectures for Medical Query Understanding
Zhu, W., Ni, Y., Wang, X., & Xie, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.29 · Cited by 21
Improving Biomedical Named Entity Recognition with a Unified Multi-Task MRC Framework
Tong, Y., Zhuang, F., Wang, D., Ying, H., & Wang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746482 · Cited by 14
Yang, L., Wang, S., & Altman, R. B. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac226 · Cited by 10
Using BERT models for breast cancer diagnosis from Turkish radiology reports
Hepsağ, P. U., Özel, S. A., Dalcı, K., & Yazıcı, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09669-w · Cited by 9
“…DistilBERT (Sanh et al., 2019), XLNet (Yang et al., 2019) have been recently introduced and used in almost all NLP tasks, including text classification tasks, and show even better performance, especially on the medical domain such as Lee et al. (2020); Gu et al. (2021), and Smit et al. (2020).”
A Neural CRF-based Hierarchical Approach for Linear Text Segmentation
Nair, I., Garimella, A., Srinivasan, B. V., Modani, N., Chhaya, N., Karanam, S., & Shekhar, S. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.65 · Cited by 7
“We believe that using a medical domain specific encoder (Gu et al., 2021) would improve our model’s performance in the Disease domain.”
MethodAPIE: An information extraction module designed based on the pipeline method
Jiang, X., Cheng, Y., Zhang, S., Wang, J., & Ma, B. (2023) · Array · doi:10.1016/j.array.2023.100331 · Cited by 9
MediBioDeBERTa: Biomedical Language Model With Continuous Learning and Intermediate Fine-Tuning
Kim, E., Jeong, Y., & Choi, M.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3341612 · Cited by 9
“Language features vary in terms of language type, style, terminology, etc.; the larger the gap between the pretraining information and the downstream problem, the more difficult it is to take advantage of them [8], [9], [10], [11], [12].”
BackgroundMethodResultInfluentialModelling long medical documents and code associations for explainable automatic ICD coding
Hou, W., Wang, X., Wang, Y., Wang, J., & Xiao, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123519 · Cited by 13
How do others cope? Extracting coping strategies for adverse drug events from social media
Dirkson, A. R., Verberne, S., van Oortmerssen, G., Gelderblom, H. A. J., & Kraaij, W. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104228 · Cited by 14
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
Self-citationALBERTI, a Multilingual Domain Specific Language Model for Poetry AnalysisPreprint
de la Rosa, J., Pozo, Á. P., Ros, S., & González‐Blanco, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.01387 · Cited by 9
Liu, Z., Xu, S., Wu, Z., Murray, B., F. Barreto, E., Li, S., Liu, W., Li, X., Liu, T., & Sikora, A. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03230-1 · Cited by 8
“1 Generating context for LLMs with medication-specific data using a dynamic context and iterative optimization prompt engineering technique in-context learning capability of LLMs [29] by constructing dynamic contexts using domain-specific data was developed [34–38].”
Remy, F., Scaboro, S., & Portelli, B. (2023) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/2023.socialnlp-1.6 · Cited by 8
Guo, B., Meng, J., Zhao, D., Jia, X., Chu, Y., & Lin, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104717 · Cited by 8
Gondara, L., Simkin, J., Arbour, G., Devji, S., & Ng, R. T. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci.24.00110 · Cited by 13
MedINST: Meta Dataset of Biomedical Instructions
Han, W., Fang, M., Zhang, Z., Yin, Y., Song, Z., Chen, L., Pechenizkiy, M., & Chen, Q. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.482 · Cited by 13
Tailored self-supervised pretraining improves brain MRI diagnostic models
Huang, X., Wang, Z. W., Zhou, W., Yang, K., Wen, K., Liu, H., Huang, S., & Lyu, M. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102560 · Cited by 10
Yeganova, L. E., Kim, W. B., Tian, S., Comeau, D. C., Wilbur, W. J., & Lu, Z. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf417 · Cited by 10
Yu, Z., Fang, L., Ding, Y., Yan, S., Xu, L., Cai, Y., & Liu, X. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104844 · Cited by 8
Self-citationCombining Evidence and Reasoning for Biomedical Fact-Checking
Barone, M., Romano, A., Riccio, G., Postiglione, M., & Moscato, V. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729931 · Cited by 13
Kang, H., Li, J., Hou, L., Xu, X., Zheng, S., & Li, Q. (2025) · JMIR Medical Informatics · doi:10.2196/77837 · Cited by 8
“For example, domain-specific language models, such as BioBERT [14], PubMedBERT (trained on biomedical literature) [15], ClinicalBERT (trained on the MIMIC-III dataset) [16], and PharmBERT (trained on prescription drug labeling) [17], have been further developed.”
Large language models in healthcare and biomedical informatics: A comprehensive review
Hornback, A. M., Sathu, H., Kim, K., Wang, Y., Zhu, Y., Isgut, M., Avula, P., Khimani, A., & Wang, M. D. (2026) · Innovation and Emerging Technologies · doi:10.1142/s2737599426300011 · Cited by 9
Multi-model Comparison for Classification of Medical Records using the BioBERT Models
Tung, N.-Y., Hu, H., Chang, T.-W., Hu, Y.-M., & Lin, H.-M. (2022) · IEEE Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS) · doi:10.1109/ecbios54627.2022.9945029 · Cited by 8
“We proposed two models including BioBERT-base-cased and PubMedBERT [5] and one regular Bert-case-based model for comparison.”
MethodErdengasileng, A., Tian, S., Green, S. K. S., Naar, S., & He, Z. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995568 · Cited by 9
Transferring From Textual Entailment to Biomedical Named Entity Recognition
Liang, T., Xia, C., Zhao, Z., Jiang, Y., Yin, Y., & Yu, P. S. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3236477 · Cited by 10
“[25] presents that pre-training language models from scratch results in more benefits compared to continual pretraining of general-domain language models for biomedical domain.”
BackgroundBadenes-Olmedo, C., & Corcho, Ó. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104382 · Cited by 12
Learning to rank query expansion terms for COVID-19 scholarly search
Khader, A., & Ensan, F. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104386 · Cited by 8
Boukhers, Z., Goswami, P., & Jürjens, J. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08581-2 · Cited by 10
“Also, it is used by most of the state-of-the-art approaches [22, 26, 27, 40, 41].”
BackgroundGeneralizing over Long Tail Concepts for Medical Term Normalization
Portelli, B., Scaboro, S., Santus, E., Sedghamiz, H., Chersoni, E., & Serra, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.588 · Cited by 8
Exploring Pair-Aware Triangular Attention for Biomedical Relation Extraction
Chen, L., Su, J., Lam, T., & Luo, R. (2023) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3584371.3612994 · Cited by 8
“Simultaneously, the advent of domain-specific pretrained language models like BioBERT [6] and PubMedBERT [2] specifically customized for the biomedical domain has revolutionized BioRE task and achieved impressive performance gains by fully learning contextualized entity representations and effectively mining relationa…”
BackgroundMethodResultBergman, E., Pasmooij, A. M. G., Mol, P. G. M., & Westman, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294560 · Cited by 8
Semantic knowledge graph as a companion for catalyst recommendation
Zhang, Z., Ma, S., Zheng, S., Nie, Z., Wang, B., Lei, K., Li, S., & Pan, F. (2024) · National Science Open · doi:10.1360/nso/20230040 · Cited by 8
Bhasuran, B., Manoharan, S., Oviya, I. R., Murugesan, G., Prabahar, A., & Raja, K. (2024) · Biomedicines · doi:10.3390/biomedicines12071535 · Cited by 9
“Over the years, NLP approaches including relation extraction have evolved to include machine learning algorithms, deep learning algorithms [10,11], and transformer-based approaches [10,12,13].”
BackgroundMGLEP: Multimodal Graph Learning for Modeling Emerging Pandemics with Big Data
Tran, K.-T., Hy, T. S., Jiang, L., & Vu, X.-S. (2024) · Scientific Reports · doi:10.1038/s41598-024-67146-y · Cited by 10
Padilla-Cuevas, J., Reyes‐Ortiz, J. A., Rasgado, A. D. C., Mora-Gutiérrez, R. A., & Bravo, M. (2024) · Applied Sciences · doi:10.3390/app14167031 · Cited by 9
Multi-Task Multi-Attention Transformer for Generative Named Entity Recognition
Mo, Y., Liu, J., Tang, H., Wang, Q., Xu, Z., Wang, J., Quan, X., Wu, W., & Li, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3458796 · Cited by 9
LmRaC: a functionally extensible tool for LLM interrogation of user experimental results
Craig, D. B., & Drăghici, S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae679 · Cited by 8
“In particular, LLMs and their precursors (Beltagy et al. 2019, Lee et al. 2020, Gu et al. 2021, Luo et al. 2022) have rapidly moved toward human levels of competence in biology.”
One-shot Biomedical Named Entity Recognition via Knowledge-Inspired Large Language Model
Bian, J., Zheng, J., Zhang, Y., Zhou, H. J., & Zhu, S. (2024) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3698587.3701356 · Cited by 10
“For example, a gene mention often refers to both the DNA and gene products such as the RNA and protein[11].”
Liu, Y., Luo, S., Zhong, Z., Wu, T., Zhang, J., Ou, P., Liang, Y., Liu, L., & Pan, H. (2025) · Studies in computational intelligence · doi:10.1007/978-3-032-11999-5_25 · Cited by 9
Discontinuous named entities in clinical text: A systematic literature review
Alhassan, A., Schlegel, V., Aloud, M., Batista-Navarro, R., & Nenadić, G. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104783 · Cited by 11
Elbiach, O., Grissette, H., & Nfaoui, E. H. (2025) · AI · doi:10.3390/ai6020031 · Cited by 11
A shape composition method for named entity recognition
Hu, Y., Chen, Y., & Xu, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107389 · Cited by 8
Zhou, Y. Q., Litake, O., Meineke, M. N., Tully, J. L., Xu, N. Y., Abdou, W., & Gabriel, R. A. (2025) · Journal of the American Geriatrics Society · doi:10.1111/jgs.19545 · Cited by 10
Inference of dependency knowledge graph for Electronic Health Records
Xu, Z., Gan, Z., Zhou, D., Shen, S., Lu, J., & Cai, T. (2025) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkaf061 · Cited by 8
“…from Transformer (BERT)-based algorithms (Devlin et al., 2019) trained on clinical corpora such as Self-aligning Pretrained BERT (SAPBERT) (Liu et al., 2021), BERT for Biomedical Text Mining (BioBERT) (Lee et al., 2020), and BERT pretrained with PubMed (PubmedBERT) (Gu et al., 2021).”
MethodGOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning
Kalifa, D., Singer, U., & Radinsky, K. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761500 · Cited by 8
Sboev, A. G., Selivanov, A., Rylkov, G., & Борисович, Р. Р. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.06.082 · Cited by 8
Li, J., Wei, Q., Ghiasvand, O., Chen, M., Lobanov, V. S., Weng, C., & Xu, H. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00095 · Cited by 9
Ferdowsi, S., Borissov, N., Knafou, J., Amini, P., & Teodoro, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.48 · Cited by 8
Sboev, A. G., Selivanov, A., Александрович, М. И., Борисович, Р. Р., Gryaznov, A. V., Sboeva, S., & Rylkov, G. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6010010 · Cited by 8
Manzini, E., Garrido-Aguirre, J., Fonollosa, J., & Perera-Lluna, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117446 · Cited by 8
Choi, D., Choi, H., & Lee, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.116 · Cited by 11
“For example, in the biomedical domain, several domain-specific PLMs trained with large biomedical texts, such as BioBERT (Lee et al., 2020), PubMedBERT (Gu et al., 2020) and BlueBERT (Peng et al., 2019), have been successfully used as strong baselines for several downstream tasks.”
MethodA Syntax-enhanced model based on category keywords for biomedical relation extraction
Liu, X., Tan, J., Fan, J., Tan, K., Hu, J., & Dong, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104135 · Cited by 9
Self-citationYang, M., Huang, Z.-A., Gu, W., Han, K., Pan, W., Yang, X. J., & Zhu, Z. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac298 · Cited by 8
Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2023) · Sensors · doi:10.3390/s23031057 · Cited by 8
“[53] have proposed transformer-based large-scale models for image and language modeling in the medical domains, respectively, with good performance, demonstrating the effectiveness of this concept.”
MethodPark, G., McCorkle, S. M., Soto, C., Blaby, I. K., & Yoo, S. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021099 · Cited by 12
“We use domain-specific pre-trained BERT models on biomedical literature, including BioBERT [26] and PubMed-BERT [25], which has demonstrated excellent performance in biomedical NLP applications.”
MethodResultInfluentialA Biomedical Entity Extraction Pipeline for Oncology Health Records in Portuguese
Sousa, H., Jorge, A., Pasquali, A., Santos, C., & Amorim‐Lopes, M. (2023) · ACM Symposium on Applied Computing · doi:10.1145/3555776.3578577 · Cited by 13
“Recently, the developments in NLP, in particular with large language models such as BERT [39], BioBERT [21], or PubMedBERT [12], have mitigated this problem.”
BackgroundBiomedical Relation Extraction Using Dependency Graph and Decoder-Enhanced Transformer Model
Kim, S., Yoon, J., & Kwon, O. (2023) · Bioengineering · doi:10.3390/bioengineering10050586 · Cited by 10
“As shown in the table, encoder-only transformers such as BioBERT, SCIBERT, PubMedBERT, BioM-BERT, and BioLinkBERT exhibited lower performance than encoder– decoder transformer models such as T5 and T5 slim_dec .”
MethodInfluentialSMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
Bashlovkina, V., Matthews, R., Kuang, Z., Baumgartner, S., & Bendersky, M. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599907 · Cited by 9
“However, related results from existing works in learning domain-specific language models are mixed [5, 24].”
BackgroundAracena, C., Villena, F., Rojas, M. S., & Dunstan, J. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.22 · Cited by 10
Biomedical Relation Extraction with Entity Type Markers and Relation-specific Question Answering
Yamada, K., Miwa, M., & Sasaki, Y. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.35 · Cited by 8
“All methods used PubMedBERT-base-uncased-abstract-fulltext (Gu et al., 2022), which has been pre-trained using a large biomedical literature, as an encoder.”
MethodInvernet: An Inversion Attack Framework to Infer Fine-Tuning Datasets through Word Embeddings
Hayet, I., Yao, Z., & Luo, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.368 · Cited by 8
“For example, an intuitive way of converting categorical vocabulary to numerical vectors is to one-hot encode the tokens into a bitstring or commonly known as a bag of words representation Harris (1954). However, one-hot encoding can easily result in high-dimensional representations and incapability of accommodating un…”
Influence Functions for Sequence Tagging Models
Jain, S., Manjunatha, V., Wallace, B., & Nenkova, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.58 · Cited by 11
Towards More Generalizable and Accurate Sentence Classification in Medical Abstracts with Less Data
Hu, Y., Chen, Y., & Xu, H. (2023) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00141-6 · Cited by 8
BioPRO: Context-Infused Prompt Learning for Biomedical Entity Linking
Zhu, T., Qin, Y., Feng, M., Chen, Q., Hu, B., & Xiang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331149 · Cited by 12
On the Linguistic Limitations of ChatGPT: An Experimental Case Study
Baziyad, M. S., Kamel, I., & Rabie, T. F. (2023) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc58260.2023.10323661 · Cited by 10
“With the rapid advancement of AI and the increasing prevalence of digital communication, NLP has become a critical area of research and development, offering significant potential for improving the efficiency and effectiveness of human-machine interactions [4]–[6].”
BackgroundConfliBERT-Spanish: A Pre-trained Spanish Language Model for Political Conflict and Violence
Yang, W., Alsarra, S., Abdeljaber, L., Zawad, N., Delaram, Z., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2023) · IEEE Congress on Information Science and Technology (CiSt) · doi:10.1109/cist56084.2023.10409883 · Cited by 8
“For this reason, scholars have been developing domain-specific pre-trained language models for biology and medicine [14], [15], law [16], science [17], and even the Dark web [19].”
BackgroundGO2Sum: generating human-readable functional summary of proteins from GO terms
Giri, S. J., Ibtehaz, N., & Kihara, D. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00358-0 · Cited by 8
“PubMedBERT 32 is a BERT model trained on PubMed abstracts and performs tasks including relation extraction, question answering, and document classi fi cation.”
Explainable epidemiological thematic features for event based disease surveillance
Menya, E. O., Interdonato, R., Owuor, D. O., & Mathieu Roche, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123894 · Cited by 8
Triplet-Trained Vector Space and Sieve-Based Search Improve Biomedical Concept Normalization
Xu, D., & Bethard, S. J. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.2 · Cited by 9
Dataset Debt in Biomedical Language Modeling
Fries, J., Seelam, N., Altay, G., Weber, L., Kang, M., Datta, D., Su, R., Garda, S., Wang, B., Ott, S., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.10 · Cited by 11
Kato, S., Kanegami, K., & Kano, M. (2022) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2022.07.568 · Cited by 9
Scaboro, S., Portelli, B., Chersoni, E., Santus, E., & Serra, G. (2022) · Experimental Biology and Medicine · doi:10.1177/15353702221128577 · Cited by 12
Wei, Q., Zuo, X., Anjum, O., Hu, Y., Denlinger, R., Bernstam, E. V., Citardi, M. J., & Xu, H. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020569 · Cited by 10
“All multi-modal models including LayoutLMv3 achieved better performance than text-based models (BioClinicalBERRT and PubmedBERT), indicating the effectiveness of layout information in classifying scanned clinical documents.”
MethodSymptoms Based Disease Prediction from Bengali Text Using Transformer Network Based Pretrained Model
Hossain, M. M., Mou, M. A., & Oishi, M. N. N. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10055374 · Cited by 9
BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease DiagnosisPreprint
Monajatipoor, M., Rouhsedaghat, M., Li, L. H., Chien, A., Kuo, C. ‐. J., Scalzo, F., & Chang, K.-W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04938 · Cited by 8
“Recently, BlueBERT has been widely used in the bioNLP community for various NLP tasks [15, 13, 38] and a few V&L tasks, e.”
MethodOpen-world Story Generation with Structured Knowledge Enhancement: A Comprehensive SurveyPreprint
Wang, Y., Lin, J., Yu, Z., Hu, W. P., & Karlsson, B. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.04634 · Cited by 8
“In many research domains, such as bio-medicine, engineering, and finance, people began to train domain-specific PLMs on respective resources [145, 146, 147], and these models have brought great advances in each field.”
BackgroundAli, S., Shahab, O., Shabeeb, R. Q. A., Ladak, F., Yang, J. O., Nadkarni, G. N., Echavarria, J., Babar, S., Shaukat, A., Soroush, A., et al. (2023) · medRxiv · doi:10.1101/2023.09.21.23295918 · Cited by 11
“Previous generation LLMs have already been tailored for medical use cases such as PubMedBERT (7), ClinicalBERT(8) with limited success.”
Large Language Models for Biomedical Text Simplification: Promising But Not There YetPreprint
Li, Z., Belkadi, S., Micheletti, N., Han, L., Shardlow, M., & Nenadić, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.03871 · Cited by 10
“Focus on the biomedical domain, [29] carried out fine-tuning stability investigation using the BLURB data set (Biomedical Language Understanding and Reasoning Benchmark) from [40].”
MethodJiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Kilicoglu, H. (2025) · Scientific Data · doi:10.1038/s41597-025-04629-1 · Cited by 11
OmniCellAgent: An AI Scientist for Omic-Driven Scientific DiscoveryPreprint
Huang, D., Li, H., Li, W., Zhang, H., Xu, T., Lu, Y., Fang, K., Xu, Z., Chen, J., Dickson, P., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.31.667797 · Cited by 9
Multimodal transformers for joint diagnosis and retrieval from chest X-rays and radiology reports
Jahanian, M., Karimi, A., Eraghi, N. O., & Zarafshan, F. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101672 · Cited by 7
Islamaj, R., Wei, C.-H., Lai, P., Huston, M., Coss, C., Kochar, P. G., Miliaras, N. B., Mork, J. G., Rodionov, O., Sekiya, K., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae129 · Cited by 7
Agarwal, S., Wood, D. M., Murray, B. A. K., Wei, Y., Busaidi, A. A., Kafiabadi, S., Guilhem, E., Lynch, J., Townend, M., Mazumder, A., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11500-9 · Cited by 7
Automated Data Harmonization in Clinical Research: Natural Language Processing Approach
Mallya, P., Henao, R., Hong, C., WOJDYLA, D. M., Schibler, T., Manchanda, V., Pencina, M. J., Hall, J. L., & Zhao, J. (2025) · JMIR Formative Research · doi:10.2196/75608 · Cited by 9
“We acknowledge this emerging direction in the field; however, the use of large models often requires fine-tuning on domain-specific data and incurs substantial computational costs, which may limit their practical applicability in resource-constrained settings [46].”
MoEPH: an adaptive fusion-based LLM for predicting phage-host interactions in health informatics
Chen, Q., Zhao, Z., Li, M., Song, W., Xiao, M., & Fang, M. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1634705 · Cited by 7
Refining medical large language models: key insights from instruction tuning
Alqahtani, M. Q., Al-Barakati, A. A., & Alotaibi, F. S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3216 · Cited by 7
TopicForest: embedding-driven hierarchical clustering and labeling for biomedical literature
Chang, C., Ondov, B., Choi, B., Peng, X., He, H., & Xu, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104958 · Cited by 7
Zhang, P., Wang, J., Wang, Y., & Zhang, Z. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3638367 · Cited by 7
Lin, Z., Ren, D., Ran, K., Sun, J., Yu, S., Bai, X., Huang, X., He, H., Pan, P., Fang, Y. G., et al. (2026) · Transactions of Materials Research · doi:10.1016/j.tramat.2026.100176 · Cited by 10
RetractedKnowledge-Guided Efficient Representation Learning for Biomedical Domain
Jha, K., Xun, G., Du, N., & Zhang, A. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467118 · Cited by 7
“As such, we compare the quality of continual (or incremental) embeddings produced by our approach for contextualized embedding models [1, 2, 11, 24] with their batch-mode counterparts in terms of both accuracy and computational efficiency.”
BackgroundMethodSaeed, N., & Naveed, H. (2022) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2022.928530 · Cited by 8
Generating Semantic Matches Between Maintenance Work Orders for Diagnostic Decision Support
Naqvi, S. M. R., Ghufran, M., Meraghni, S., Varnier, C., Nicod, J., & Noureddine Zerhouni, A. (2022) · Annual Conference of the PHM Society · doi:10.36001/phmconf.2022.v14i1.3241 · Cited by 7
“such as biomedical (Gu et al., 2021; Cho & Lee, 2019), legal (Chalkidis, Fergadiotis, Malakasiotis, Aletras, & Androutsopoulos, 2020) and maintenance (Naqvi, Varnier, Nicod, Zerhouni, & Ghufran, 2022).”
BackgroundMapping Chinese Medical Entities to the Unified Medical Language System
Chen, L., Qi, Y., Wu, A., Deng, L., & Jiang, T. (2023) · Health Data Science · doi:10.34133/hds.0011 · Cited by 9
“SapBERT was enhanced by infusing synonymous knowledge from UMLS and inherited context information from PLMs focusing on the biomedical domain, such as PubMedBERT [35].”
BackgroundA Review of Drug-related Associations Prediction Based on Artificial Intelligence Methods
Ma, M., Lei, X., & Zhang, Y. (2023) · Current Bioinformatics · doi:10.2174/1574893618666230707123817 · Cited by 7
Overview of the PromptCBLUE Shared Task in CHIP2023
Zhu, W., Wang, X., Chen, M., & Tang, B. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1717-0_1 · Cited by 8
Almudaifer, A. I., Covington, W., Hairston, J., Deitch, Z., Anand, A., Carroll, C., Crisan, E., Bradford, W., Walter, L. A., Eaton, E., et al. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00311-4 · Cited by 8
“We chose BioBERT over other variants like BERT [18], ClinicalBERT [22], and PubMedBERT [23] since it performed better for detecting modifiers at initial experiments.”
MethodHuang, T.-Y., Chong, C., Lin, H.-Y., Chen, T.-Y., Chang, Y.-C., & Lin, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105564 · Cited by 8
Justice, A. C., McMahon, B. H., Madduri, R., Crivelli, S. N., Damrauer, S. M., Cho, K., Ramoni, R. B., & Muralidhar, S. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae126 · Cited by 7
Domain specific semantic categories in biomedical applications
Ayams, J. N., Okunlola, F. O., Chimbekujwo, K. I., Taiwo, D. I., Oyewole, O. A., Adetunji, C. O., Adetuyi, B. O., Chinazaekpele, C. C., Adetuyi, T. G., Okunlola, A. R., et al. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-15452-2.00029-7 · Cited by 8
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.