Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 801–900 of 1,030
Event-Based Clinical Finding Extraction from Radiology Reports with Pre-trained Language Model
Lau, W., Lybarger, K., Gunn, M. L. D., & Yetişgen, M. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00717-5 · Cited by 17
Improving biomedical named entity recognition through transfer learning and asymmetric tri-training
Bhattacharya, M., Bhat, S., Tripathy, S., Bansal, A., & Choudhary, M. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.244 · Cited by 14
FindZebra online search delving into rare disease case reports using natural language processing
Liévin, V., Hansen, J. M., Lund, A. M., Elstein, D., Matthiesen, M. E., Elomaa, K., Zarakowska, K., Himmelhan, I., Botha, J., Borgeskov, H., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000269 · Cited by 11
Extracting Decision Trees from Medical Texts: An Overview of the Text2DT Track in CHIP2022
Zhu, W., Li, W., Wang, X., Ji, W., Wu, Y., Chen, J., Chen, L., & Tang, B. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-4826-0_9 · Cited by 30
SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
Antypas, D., Ushio, A., Barbieri, F., Neves, L., Rezaee, K., Espinosa-Anke, L., Pei, J., & Camacho-Collados, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.838 · Cited by 30
Sequential Brain CT Image Captioning Based on the Pre-Trained Classifiers and a Language Model
Kong, J.-W., Oh, B.-D., Kim, C., & Kim, Y.-S. (2024) · Applied Sciences · doi:10.3390/app14031193 · Cited by 11
“This may be attributed to the fact that the fine-tuned CNN has learned both normal and abnormal brains and has discriminated features beyond ICH.”
BackgroundLarge Language Models in Neurosurgery
Ieva, A. D., Stewart, C., & Molina, E. S. (2024) · Advances in experimental medicine and biology · doi:10.1007/978-3-031-64892-2_11 · Cited by 12
Lit-OTAR framework for extracting biological evidences from literature
Tirunagari, S., Saha, S., Venkatesan, A., Süveges, D., Carmona, M., Buniello, A., Ochoa, D., McEntyre, J., McDonagh, E. M., & Harrison, M. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf113 · Cited by 11
CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Kusa, W., Mendoza, Ó. E., Samwald, M., Knoth, P., & Hanbury, A. (2023) · neural information processing systems · doi:10.52202/075280-1017 · Cited by 21
An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training
Arumae, K., Sun, Q., & Bhatia, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.394 · Cited by 16
Templated Text Synthesis for Expert-Guided Multi-Label Extraction from Radiology Reports
Schrempf, P., Watson, H. I., Park, E., Pajak, M., MacKinnon, H., Muir, K. W., Harris‐Birtill, D., & O’Neil, A. Q. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3020015 · Cited by 14
“[17]) or PubMedBERT [9] use the same model architecture and pre-training procedures as the original BERT, but use different pre-training datasets, allowing the models to learn the context of domain-specific vocabulary.”
MethodInfluentialVelichkov, B., Vassileva, S., Gerginov, S., Kraychev, B., Ivanov, I., Ivanov, P., Koychev, I. K., & Boytcheva, S. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_162 · Cited by 11
An annotated corpus from biomedical articles to construct a drug-food interaction database
Kim, S., Choi, Y., Won, J., Oh, J. M., & Lee, H. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.103985 · Cited by 11
Mahbub, M., Srinivasan, S., Begoli, E., & Peterson, G. D. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac508 · Cited by 11
“We comprehensively demonstrate that without any label information in the target domain during training, the BioADAPT-MRC framework can achieve SOTA performance on these datasets.”
BackgroundMethodInfluentialMedical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning
Ziletti, A., Akbik, A., Berns, C., Herold, T., Legler, M., & Viell, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.21 · Cited by 15
“=3 k=5 k=10 k>=100 PubMedBERT (PMB) (single) 74.90.5 31.01.8 44.50.8 57.11.3 60.31.8 68.51.7 65.61.2 84.00.7 BioBERT (BB) (single) 74.90.1 29.21.9 41.81.5 53.51.5 60.02.6 65.92.6 62.51.1 84.40.4 sciBERT (SB) (single) 74.90.3 27.81.7 42.52.5 58.41.3 57.52.2 66.41.1 60.32.4 84.60.8 TARS (neg=2 cos) 64.91.2 44.01.2 52.31…”
BackgroundMethodInfluentialA dataset for evaluating clinical research claims in large language models
Zhang, B., Bornet, A., Yazdani, A., Khlebnikov, P., Milutinovic, M., Rouhizadeh, H., Amini, P., & Teodoro, D. (2025) · Scientific Data · doi:10.1038/s41597-025-04417-x · Cited by 22
Tsubota, T., Bollegala, D., Zhao, Y., Jin, Y., & Kozu, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104185 · Cited by 11
Miao, Y., Wen, J., Luo, Y., & Li, J. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106136 · Cited by 10
Al-Hussaini, I., Nakajima An, D., Lee, A. J., Bi, S., & Mitchell, C. S. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020807 · Cited by 10
“Due to the highest average F1Score among all methods, PubMedBERT [38] was selected as the model used to make predictions in the back-end of the”
MethodInfluentialNeves, M., Klippert, A., Knöspel, F., Rudeck, J., Stolz, A., Bán, Z., Becker, M., Diederich, K., Grune, B., Kahnau, P., et al. (2023) · Journal of Biomedical Semantics · doi:10.1186/s13326-023-00292-w · Cited by 12
“State-of-the-art methods for NER are mostly based on the transformers architecture and a pre-trained model, e.g., BioBERT [15] or PubMedBERT [16], and a fine-tuning on NER-specific corpora.”
BackgroundMethodA Streaming Approach to Neural Team Formation Training
Fani, H., Barzegar, R., Dashti, A., & Saeedi, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_20 · Cited by 10
Hugging Face's impact on medical applications of artificial intelligence
Riva, M., Parigi, T. L., Ungaro, F., & Massimino, L. (2024) · Computational and Structural Biotechnology Reports · doi:10.1016/j.csbr.2024.100003 · Cited by 12
A New Approach to Landscape Visual Quality Assessment from a Fine-Tuning Perspective
Fan, R., Chen, Y., & Yocom, K. P. (2024) · Land · doi:10.3390/land13050673 · Cited by 10
“Individual users or small-scale research institutions may find it difficult to fine-tune large models due to hardware constraints, limiting the popularity and scope of application of customized models.”
Ma, C., & Wolfinger, R. D. (2024) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.4c00134 · Cited by 10
Learning to explain is a good biomedical few-shot learner
Chen, P., Wang, J., Luo, L., Lin, H., & Yang, Z. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae589 · Cited by 10
“Specifically, we adopt pretrained PubMedBERT-MNLI-MedNLI (Deka et al. 2023), trained on the MultiNLI (Williams et al. 2018), and MedNLI (Romanov and Shivade 2018) datasets, to serve as NLI discriminators by further fine-tuning.”
BackgroundAbdulnazar, A., Roller, R., Schulz, S., & Kreuzthaler, M. (2024) · Digital Health · doi:10.1177/20552076241288681 · Cited by 13
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 11
Analysis of eligibility criteria clusters based on large language models for clinical trial design
Bornet, A., Khlebnikov, P., Meer, F., Haas, Q., Yazdani, A., Zhang, B., Amini, P., & Teodoro, D. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae311 · Cited by 16
“LLMs can be pre-trained with large databases from the general domain and further pre-trained with biomedical literature to generate embeddings that are more suited for clinical tasks, such as PubMedBERT[31] or BioBERT[32].”
ContReviews: A content-based recommendation system for updating Living Evidences in health care
Tenti, P., Thomas, J., Peñaloza, R., & Pasi, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112981 · Cited by 11
Pan, Y., He, Y., Liu, Y.-Q., Shan, Y., Liu, S., Ma, J.-H., Liu, X., Pan, X., Bai, Y., Xu, Z., et al. (2026) · National Science Review · doi:10.1093/nsr/nwag376 · Cited by 11
Self-citationAhn, J., Kang, B. G., Chang, M. Y., & Yoon, S. (2025) · Clinical and Experimental Otorhinolaryngology · doi:10.21053/ceo.2025-00121 · Cited by 10
“While early medical LLMs, such as BioGPT [9], DRAGON [10], LinkBERT [11], BiomedBERT [12], and PubMedGPT [13], were trained on biomedical corpora using relatively small base models, recent evidence suggests that general-purpose LLMs trained on large-• scale datasets outperform these specialized…”
Item Difficulty Modeling Using Fine-tuned Small and Large Language Models
Li, M., Jiao, H., Zhou, T., Zhang, N., Peters, S., & Lissitz, R. W. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644251344973 · Cited by 21
“The models incorporated are mostly encoder-only language models but also some models with encoder-decoder structures, including BERT (De-vlin et al., 2018), RoBERTa (Liu et al., 2019), DistillBERT (Sanh et al., 2020),deBERTa (He et al., 2021),ELECTRA (Clark et al., 2020), Con-vBERT (Jiang et al., 2020), T5 (Raffel et…”
ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answering
Gichamba, A., Idris, T. K., Ebiyau, B., Nyberg, E., & Mitamura, T. (2024) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshp64532.2024.11100297 · Cited by 11
Decipher-MR: a vision-language foundation model for 3D MRI representations
Yang, Z., D’Souza, N., Megyeri, I., Xu, X., Shandiz, A. H., Haddadpour, F., Koós, K., Ruskó, L., Valeriano, E., Swaminathan, B., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02596-4 · Cited by 13
“We compared our models to different subsets of following baseline vision encoders, selected based on the nature of the tasks.”
MIRA: A Novel Framework for Fusing Modalities in Medical RAG
Wang, J., Ashraf, T., Han, Z., Laaksonen, J. T., & Anwer, R. M. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755760 · Cited by 15
Dhinagar, N. J., Thomopoulos, S. I., & Thompson, P. M. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11251809 · Cited by 14
“This included BERT [21], ClinicalBioBERT [22], DeBERTa [23], BiomedBERT [24], and BERT [25].”
Spiegler, P., Koleilat, T., Harirpoush, A., Miller, C., Rivaz, H. J., Kersten‐Oertel, M., & Xiao, Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00104 · Cited by 16
Domain-oriented Language Modeling with Adaptive Hybrid Masking and Optimal Transport Alignment
Zhang, D., Yuan, Z., Liu, Y., Liu, H., Zhuang, F., Xiong, H., & Chen, H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467215 · Cited by 10
“Moreover, several domainoriented PLMs have been proposed to adapt to specific domains [4, 8, 10].”
BackgroundRe-Ranking System with BERT for Biomedical Concept Normalization
Cho, H., Choi, D., & Lee, H. (2021) · IEEE Access · doi:10.1109/access.2021.3108445 · Cited by 11
“Moreover, the performance can be further improved by additional Task-PT with in-task data, and we found that the calibration approach can significantly improve the performance of PubMedBERT-based models.”
MethodInfluentialGraphine: A Dataset for Graph-aware Terminology Definition Generation
Liu, Z., Wang, S., Gu, Y., Zhang, R., Zhang, M., & Wang, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.278 · Cited by 10
Self-citationRojewska, K., Maćkowska, S., Maćkowski, M., Różańska, A., Barańska, K., Dzieciątko, M., & Spińczyk, D. (2022) · Applied Sciences · doi:10.3390/app12094702 · Cited by 10
Xu, H. A., Maccari, B., Guillain, H., Herzen, J., Agri, F., & Raisaro, J. L. (2022) · JMIR Medical Informatics · doi:10.2196/38150 · Cited by 10
Liu, Y., Elsworth, B., & Gaunt, T. R. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad169 · Cited by 11
Knafou, J., Haas, Q., Borissov, N., Counotte, M. J., Low, N. N., Imeri, H., Ipekci, A. M., Buitrago‐García, D., Heron, L., Amini, P., et al. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02247-9 · Cited by 10
GraphPrompt: Graph-Based Prompt Templates for Biomedical Synonym Prediction
Xu, H., Zhang, J., Wang, Z., Zhang, S., Bhalerao, M. M., Liu, Y., Zhu, D., & Wang, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26256 · Cited by 10
“We calculate the number of mentions of each phrase in 29 million PubMed abstracts, which are used as the pre-training corpus for biomedical pre-trained models (Lee et al. 2020; Gu et al. 2020).”
MethodUMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition
Mannion, A., Schwab, D., & Goeuriot, L. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.35 · Cited by 10
ShellGPT: Generative Pre-trained Transformer Model for Shell Language Understanding
Shi, J., Jiang, S., Xu, B., Liang, J., Xiao, Y., & Wang, W. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00082 · Cited by 21
A Systematic Online Living Evidence Summary of experimental Alzheimer’s disease research
Hair, K., Wilson, E., Maksym, O., Macleod, M. R., & Sena, E. S. (2024) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2024.110209 · Cited by 10
“While employing regular expression dictionaries is suitable for cases with limited variations, leveraging natural language models (NMLs) such as PubMedBERT (Gu et al., 2021) holds promise.”
Woerner, S., & Baumgartner, C. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73471-7_4 · Cited by 10
Investigating the impact of pretraining corpora on the performance of Arabic BERT models
Alammary, A. S. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06698-2 · Cited by 10
“In the medical domain, English BERT base model was able to achieve an F1 score of 0.8 [41] compared to a maximum of 0.6 F1 score achieved by the best performing Arabic model (i.e., AraBERT).”
InfluentialThe gene function prediction challenge: Large language models and knowledge graphs to the rescue
Sunil, R. S., Lim, S. C., Itharajula, M., & Mutwil, M. (2024) · Current Opinion in Plant Biology · doi:10.1016/j.pbi.2024.102665 · Cited by 11
Chen, F., Zhang, G., Fang, Y., Peng, Y., & Weng, C. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae326 · Cited by 12
Do LLMs Surpass Encoders for Biomedical NER?
Obeidat, M. S., Nahian, M. S. A., & Kavuluru, R. (2025) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi64645.2025.00048 · Cited by 17
“Considering these factors, in this paper, we report on an elaborate effort comparing three encoder models (BERT, BiomedBERT [26], and DeBERTa) and two LLMs (Mis-tral [27] and Llama [28]) with identical BIO tagging output scheme across five different biomedical NER datasets with varying proportions…”
CONORM: Context-Aware Entity Normalization for Adverse Drug Event Detection
Yazdani, A., Rouhizadeh, H., Bornet, A., & Teodoro, D. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3858333 · Cited by 10
“Social media data is often marked by slang, abbreviations, misspellings, and colloquial expressions [53], whereas clinical narratives rely on specialized medical terminology [12, 26, 47].”
EMBERT: A Pre-trained Language Model for Chinese Medical Text Mining
Cai, Z., Zhang, T., Wang, C., & He, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85896-4_20 · Cited by 14
“Meanwhile, PubMedBERT [10] learns model weights from scratch by large-scale training corpus and argues that the key point of training domain-specific PLMs is learning from scratch, which can obtain an in-domain vocabulary, alleviating the out-of-vocabulary (OOV) problem.”
BackgroundLin, Y.-C., Hoffmann, P., & Rahm, E. (2022) · SN Computer Science · doi:10.1007/s42979-022-01295-7 · Cited by 13
“The models with UMLS injection (SapBERT and CODER models) outperform the models without UMLS injection (PubMedBERT and BioBERT) remarkably.”
BackgroundResultInfluentialDeveloping Pretrained Language Models for Turkish Biomedical Domain
Türkmen, H., Dikenelli, O., Eraslan, C., Çallı, M. C., & Özbek, S. S. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00117 · Cited by 11
“PubMedBERT [3] evaluated an alternative pretraining approach called pretraining from scratch.”
Methodvan der Linden, S., Sevastjanova, R., Funk, M., & El‐Assady, M. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3564275 · Cited by 11
“There are several medical domain-specific adaptations of BERT, such as BioBERT [38] and PubMedBert [23] (both additionally pre-trained models on large-scale biomedical corpora), and ClinicalBERT [29] and clinical-kb-bert [25] (both additionally pre-trained on the MIMIC-III [35] dataset to capture patient-record relate…”
MethodLADER: Log-Augmented DEnse Retrieval for Biomedical Literature Search
Jin, Q., Shin, A., & Lu, Z. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592005 · Cited by 11
A Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion
Lovelace, J., & Rosé, C. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.398 · Cited by 15
“Further performance improvements can often be gained by scaling up the size of the language model Devlin et al. (2019) or from using specialized, domain-specific language models Gu et al. (2020).”
A self-supervised language model selection strategy for biomedical question answering
Arabzadeh, N., & Bagheri, E. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104486 · Cited by 10
An Artificial-Intelligence-Driven Spanish Poetry Classification Framework
Deng, S., Wang, G., Wang, H., & Chang, F. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7040183 · Cited by 10
A Hybrid Approach to Measure Semantic Relatedness in Biomedical ConceptsPreprint
Kalyan, K. S., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10196 · Cited by 12
“Two versions of PubMedBERT [44] are available a) PubMedBERT trained on PubMed abstracts only and b) PubMedBERT trained on PubMed abstracts and PMC full text.”
MethodDrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsPreprint
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.03.535368 · Cited by 30
“In the biomedical domain, previous works (Gu et al., 2021) on PLMs highlighted the importance of matching the data sources used for its training to the targeted downstream tasks.”
BackgroundMethodInfluentialAmiri, M., & Bocklitz, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.17277 · Cited by 11
“The evaluated domain-pretrained models include SciBERT [35], PubMedBERT [40], Biomed-BERT [60], ChemicalBERT [42], ChemBERTa [43], and MatSciBERT [39].”
Feng, H., Ronzano, F., LaFleur, J., Garber, M., de Oliveira, R., Rough, K., Roth, K., Nanavati, J., El Abidine, K. Z., & Mack, C. (2024) · doi:10.1101/2024.05.17.24307411 · Cited by 9
Millikin, R. J., Raja, K., Steill, J., Lock, C., Tu, X., Ross, I., Tsoi, L. C., Kuusisto, F., Ni, Z., Livny, M., et al. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05539-y · Cited by 9
“The PubMedBERT model from Microsoft [14] was fine-tuned on NER (specifically, the “spans” task) and RE (derived from https:// github. com/ explo sion/ proje cts/ tree/ v3/ tutor ials/ rel_ compo nent) tasks using spaCy [21].”
MethodImproving biomedical Named Entity Recognition with additional external contexts
Tho, B. D., Nguyen, M.-T., Le, D. T., Ying, L.-L., Inoue, S., & Nguyen, T.-T. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104674 · Cited by 12
Mayzura, W. S., Sarno, R., Suroto, N. S., Supriyanto, M. I. A., & Sihaj, G. (2025) · IEEE Access · doi:10.1109/access.2025.3560701 · Cited by 9
“This is where pre-trained medical language models become crucial, as models like BioMedBERT and BioClinicalBERT have shown superior performance in processing medical texts within the field of natural language processing (NLP) [8], [9].”
Krzyzanowski, A., Pickett, S. D., & Pogány, P. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00359 · Cited by 12
ProtChatGPT: Towards Understanding Proteins with Hybrid Representation and Large Language Models
Wang, C., Fan, H., Quan, R., Yao, L., & Yang, Y. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730064 · Cited by 11
“Particularly, we replace the original Bert encoder with PubMedBERT [12], a biomedical description-specific encoder pre-trained on large-scale datasets.”
Artificial Intelligence in Pathology: Advancing Large Models for Scalable Applications
Xiao, Z., Feng, B., Yang, J., Sun, G., Shen, Y., Xu, S., Yang, L., Xu, H., Zhang, M., & Wang, S. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095814 · Cited by 11
Do you actually need an LLM? Rethinking language models for customer reviews analysis
Xiao, Y., Li, Y., Chen, S., Barker, H., & Rad, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11308-5 · Cited by 12
“First, the adaptation of our hybrid approach to specialized domains such as healthcare or legal applications warrants investigation, building on domain-specific language models like those developed for bio - medical (Gu et al. 2021) and legal text processing (Chalkidis et al. 2020).”
Liu, H., Chen, Z., Li, P., Liu, Y.-Z., Liu, X., Xu, R. X., & Sun, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104896 · Cited by 11
Self-citationLLM-Enhanced Multimodal Framework for Drug–Drug Interaction Prediction
Im, S., & Ko, Y. (2025) · Biomedicines · doi:10.3390/biomedicines13102355 · Cited by 9
“Fine-tuning BioBERT or other biomedical language models (e.g., PubMedBERT [26]) on DDI-specific corpora—such as curated interaction reports, clinical notes, or pharmacological texts—may yield more informative embeddings that capture finer-grained interaction semantics.”
Zhou, Y., Jiang, M., Shen, K., He, T., Zhou, S., Shao, A., Li, X., Wang, J., Wu, R., Singla, R. K., et al. (2025) · npj Biofilms and Microbiomes · doi:10.1038/s41522-025-00825-4 · Cited by 10
AutoPCR: automated phenotype concept recognition by prompting
Tao, Y., Huang, Y., Wang, Y., Luo, X., & Liu, J. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag304 · Cited by 11
MetaBlock-SE: A Method to Deal With Missing Metadata in Multimodal Skin Cancer Classification
Bouzon, P. H. G., da Rocha, W. F., de Souza, L. A., Jr., & Pacheco, A. G. C. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3612837 · Cited by 11
Zhu, Y., Shu, X., Liu, X., Li, Y., Yi, B., & Ma, D. (2025) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00156-9 · Cited by 9
“Additionally, models, like BiomedBERT [35], PubMedBERT [36], and BioGPT [37], are capable of retrieving data from electronic health records, extracting information from clinical narratives, and acquiring knowledge from medical literature to complete question-answering and information extraction…”
Self-citationLarge language models for porous materials: from text mining to autonomous laboratory
Han, S., Bae, T., Kim, J., Kim, Y.-H., & Kim, J. (2026) · Digital Discovery · doi:10.1039/d5dd00578g · Cited by 9
Recent Trends in Metabolomics by NMR Spectroscopy
Paco, G. D., Meoni, G., Ghini, V., Vignoli, A., Tenori, L., Turano, P., & Luchinat, C. (2026) · Angewandte Chemie International Edition · doi:10.1002/anie.202525689 · Cited by 10
“For the task of classification of the articles we used PubMedBERT [120], a language model based on BERT (Bidirectional Encoder Representations from Transformers) [121] architecture and pre-trained specifically on millions of abstracts available in PubMed.”
Generative AI for healthcare: fundamentals, challenges, and perspectives
Chen, G., Liu, C., Anne, O. G., Tan, M., Xie, Z., Yin, J., Yip, J. W. L., Zhang, W., ZHU, J., & Ooi, B. C. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4927-2 · Cited by 10
“PubMedBERT [22] further enhances domain semantic representation by pretraining directly on biomedical literature from scratch, while GatorTron [23, 24] scales up both model size and training corpus, pushing clinical NLP toward higher precision and broader coverage.”
Chemical identification and indexing in PubMed full-text articles using deep learning and heuristics
de Almeida, T. D., Antunes, R., Silva, J., Almeida, J. R., & Matos, S. (2022) · Database · doi:10.1093/database/baac047 · Cited by 11
“8072 – 5 Bevan and Hodgskiss (100) PubMedBERT (83) 0.”
BackgroundMethodInfluentialLin, S.-J., Yeh, W.-C., Chiu, Y.-W., Chang, Y., Hsu, M., Chen, Y.-S., & Hsu, W. (2022) · Database · doi:10.1093/database/baac056 · Cited by 11
Mahbub, M., Begoli, E., Martins, S. B., Peluso, A., Tamang, S. R., & Peterson, G. D. (2023) · IEEE Access · doi:10.1109/access.2023.3235265 · Cited by 12
“In recent years, transformer-based deep learning models have become the most popular choice among researchers for biomedical MRC tasks because of their unbeatable performance [19], [20], [21], [22], [23], [24].”
BackgroundMethodInfluentialCreating an ignorance-base: Exploring known unknowns in the scientific literature
Boguslav, M. R., Salem, N. M., White, E. K., Sullivan, K. J., Bada, M. A., Hernandez, T. L., Leach, S. M., & Hunter, L. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104405 · Cited by 12
He, F., Liu, K., Yang, Z., Hannink, M., Hammer, R., Popescu, M., & Xu, D. H. (2023) · Medical Review · doi:10.1515/mr-2023-0011 · Cited by 9
Biomedical Named Entity Recognition via Dictionary-based Synonym Generalization
Fu, Z., Su, Y., Meng, Z., & Collier, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.903 · Cited by 16
Identifying Mentions of Pain in Mental Health Records Text: A Natural Language Processing Approach
Chaturvedi, J., Velupillai, S. U., Stewart, R. J., & Roberts, A. (2024) · Studies in health technology and informatics · doi:10.3233/shti231054 · Cited by 9
Lan, M., Cheng, M., Hoang, L., ter Riet, G., & Kilicoglu, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104628 · Cited by 15
Just how transformative will AI/ML be for immuno-oncology?
Bottomly, D., & McWeeney, S. K. (2024) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2023-007841 · Cited by 9
“73 In addition to the general clinical applications reviewed in, 23 of most relevance to IO research, is the ability to retrieve and summarize existing research.”
BackgroundZhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 9
Pisces: A multi-modal data augmentation approach for drug combination synergy prediction
Xu, H., Lin, J., Woicik, A., Liu, Z., Ma, J., Zhang, S., Poon, H., Wang, L., & Wang, S. (2025) · Cell Genomics · doi:10.1016/j.xgen.2025.100892 · Cited by 13
Self-citationBenchmarking transformer embedding models for biomedical terminology standardization
Lahiri, A., Shukla, S., Stear, B. J., Ahooyi, T. M., Beigel, K., Margolskee, E. M., & Taylor, D. M. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100683 · Cited by 12
Prompt mechanisms in medical imaging: A comprehensive survey
Yang, H., Liang, X., Li, Z., Sun, Y., Hu, Z., Xie, X., Dashtbozorg, B., Huang, J., Zhu, S., Han, L., et al. (2026) · The Innovation · doi:10.1016/j.xinn.2026.101271 · Cited by 9
“14 On the contrary, MI-Zero employs static class-name templates (e.g., “an image showing adeno-carcinoma”) within a standard text encoder such as PubMedBERT.”
InfluentialSohrab, M. G., Nguyen, A.-K. D., Miwa, M., & Takamura, H. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.38 · Cited by 10
Discovering Better Model Architectures for Medical Query Understanding
Zhu, W., Ni, Y., Wang, X., & Xie, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.29 · Cited by 21
Improving Biomedical Named Entity Recognition with a Unified Multi-Task MRC Framework
Tong, Y., Zhuang, F., Wang, D., Ying, H., & Wang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746482 · Cited by 14
Yang, L., Wang, S., & Altman, R. B. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac226 · Cited by 10
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.