Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 501–600 of 1,030
Hu, Y., Keloth, V. K., Raja, K., Chen, Y., & Xu, H. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad542 · Cited by 21
“We also experimented with the NER models with other pre-trained models including BERT (Devlin et al. 2018), BioBERT (Lee et al. 2020), BioM-ALBERT (Alrowili and Vijay-Shanker 2021), and BioM-ELECTRA (Gu et al. 2022).”
MethodInfluentialDomain-Specific Large Language Model for Renewable Energy and Hydrogen Deployment Strategies
Gabbar, H. A., & Hemied, O. S. (2024) · Energies · doi:10.3390/en17236063 · Cited by 26
Application of artificial intelligence large language models in drug target discovery
Liu, X., Zhang, J., Wang, X., Teng, M., Wang, G., & Zhou, X. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1597351 · Cited by 29
Self-citationCitationSum: Citation-aware Graph Contrastive Learning for Scientific Paper Summarization
Luo, Z., Xie, Q., & Ananiadou, S. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583505 · Cited by 21
“We investigate both BERT [11] and PubMedBERT [8] as the encoder in our method.”
MethodChen, L., Hung, K.-H., Tseng, Y., Wang, H., Lu, T., Huang, W., & Tsao, Y. (2023) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2023.3307794 · Cited by 25
“Gu et al. [31] argued that models trained on domain-specific vocabulary outperform those trained on general corpora due to differences in word distributions between general and biomedical corpora.”
BackgroundQiu, J., Han, W., Zhu, J., Xu, M., Rosenberg, M., Liu, E., Weber, D. J., & Zhao, D. X. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.33 · Cited by 36
Open-world story generation with structured knowledge enhancement: A comprehensive survey
Wang, Y., Lin, J., Yu, Z., Hu, W. P., & Karlsson, B. F. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126792 · Cited by 40
A hierarchical convolutional model for biomedical relation extraction
Hu, Y., Chen, Y., Huang, R., Qin, Y., & Zheng, Q. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103560 · Cited by 21
Resilience Assessment of Large Language Models under Transient Hardware Faults
Agarwal, U. K., Chan, A., & Pattabiraman, K. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00052 · Cited by 35
“For RQ1 and RQ2, we chose five benchmarks: PubMedBert [39], CodeBert [3], T5 [20], GPT2 [23], and Roberta [22], as shown in Table I.”
MethodAkhondi‐Asl, A., Yang, Y., Luchette, M., Burns, J. P., Mehta, N. M., & Geva, A. (2024) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000003468 · Cited by 22
“Furthermore, because these large models are trained on general-domain text corpora with limited or no domain-specific text, their performance in specialized domains, such as pediatric critical care, may be limited (17–21).”
BackgroundFoundation Models for Histopathology—Fanfare or Flair
Alfasly, S., Nejat, P., Hemati, S., Khan, J., Lahr, I., Alsaafin, A., Shafique, A., Comfere, N. I., Murphree, D. H., Meroueh, C., et al. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.02.003 · Cited by 27
Using large language models for safety-related table summarization in clinical study reports
Landman, R., Healey, S. P., Loprinzo, V., Kochendoerfer, U., Winnier, A. R., Henstock, P. V., Lin, W., Chen, A., Rajendran, A., Penshanwar, S., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae043 · Cited by 22
Carchiolo, V., & Malgeri, M. (2025) · Future Internet · doi:10.3390/fi17020076 · Cited by 26
Programmable design of functional proteins from natural language
Dai, F., You, S., Zhu, Y., Gao, Y., Fu, L., Zhou, X., Su, J., Wang, C., Fan, Y., Tan, Q., et al. (2026) · Nature · doi:10.1038/s41586-026-11205-z · Cited by 21
Wang, K., Stevens, R., Alachram, H., Li, Y., Soldatova, L., King, R. D., Ananiadou, S., Schoene, A. M., Li, M., Christopoulou, F., et al. (2021) · npj Systems Biology and Applications · doi:10.1038/s41540-021-00200-x · Cited by 21
Xu, X., & Sankar, R. (2025) · Information · doi:10.3390/info16100894 · Cited by 20
“Early domain-specific models such as BioBERT [14] and PubMedBERT [15] were pre-trained on biomedical corpora to enhance text understanding tasks, such as entity recognition, relation extraction, and question answering, and demonstrated the value of domain adaptation and structured knowledge…”
Retrieval-Based Diagnostic Decision Support: Mixed Methods Study
Abdullahi, T., Mercurio, L., Singh, R., & Eickhoff, C. (2024) · JMIR Medical Informatics · doi:10.2196/50209 · Cited by 20
“PubMedBERT [40] was specifically designed to capture domain-specific knowledge present in biomedical literature.”
BackgroundTen simple rules for optimal and careful use of generative AI in science
Helmy, M., Jin, L., Alhossary, A., Mansour, T. A., Pellegrina, D. V. D. S., & Selvarajoo, K. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013588 · Cited by 20
Kierszbaum, S., Klein, T., & Lapasset, L. (2022) · Aerospace · doi:10.3390/aerospace9100591 · Cited by 20
“In the work of [5], the authors obtain very good results on various medical NLU tasks, by using PubMed as a corpus for pre-training.”
BackgroundLi, Q., Wang, Y., You, T., & Lu, Y. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.063 · Cited by 24
Chng, S. Y., Tern, P. J. W., Kan, R. X. M., & Cheng, L. T. (2023) · Health care science · doi:10.1002/hcs2.40 · Cited by 22
“Domain ‐ specific BERT, such as PubMedBERT [38], can also outperform the original BERT model with higher area under the receiver operating characteristic curve in automated labelling of radiology reports [36].”
MethodKnowledge-graph-enabled biomedical entity linking: a survey
Shi, J., Yuan, Z., Guo, W., Ma, C., Chen, J., & Zhang, M. (2023) · World Wide Web · doi:10.1007/s11280-023-01144-4 · Cited by 28
“Thus, their proposed model uses ResCNN as the backbone network to encode the embedding representation, which is initialized by Pub-MedBERT [85].”
MethodDRLK: Dynamic Hierarchical Reasoning with Language Model and Knowledge Graph for Question Answering
Zhang, M., Dai, R., Dong, M., & He, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.342 · Cited by 26
“Recently, largescale pre-trained language models (LMs) (Gu et al., 2022; Liu et al., 2021) have become a popular solution in several QA datasets (Mutabazi et al., 2021), achieving excellent performance.”
BackgroundMethodInfluentialReveal the Unknown: Out-of-Knowledge-Base Mention Discovery with Entity Linking
Dong, H., Chen, J., He, Y., Liu, Y.-N., & Horrocks, I. R. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615036 · Cited by 24
“Domain-Specific LMs. In-domain and knowledge-enhanced LMs, SapBERT (and also PubMedBERT [16]), obtained better inand out-of- KB results for datasets in the biomedical domain.”
Zhang, G., Zhou, Y., Hu, Y., Xu, H., Weng, C., & Peng, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae065 · Cited by 26
Huang, M.-S., Han, J.-C., Lin, P.-Y., You, Y.-T., Tsai, R. T., & Hsu, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae132 · Cited by 25
“Besides BioBERT, we tested five other models: SciBERT [108],BlueBERT [95],PubMedBERT [109],BioRoBERTa [110] and CODER [111].”
MethodA Language Model Based Framework for New Concept Placement in Ontologies
Dong, H., Chen, J., He, Y., Gao, Y., & Horrocks, I. R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60626-7_5 · Cited by 20
Marini, N., Marchesin, S., Wodziński, M., Caputo, A., Podareanu, D., Guevara, B. C., Boytcheva, S., Vatrano, S., Fraggetta, F., Ciompi, F., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103303 · Cited by 23
Nassr, R. M., Alkhnbashi, O. S., & Hammoudeh, M. A. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110157 · Cited by 23
Knowledge-Enhanced Visual-Language Pretraining for Computational Pathology
Zhou, X., Zhang, X., Wu, C., Zhang, Y., Xie, W., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72943-0_20 · Cited by 30
Cao, J., Zhang, J., Yu, Q., Ji, J., Li, J., He, S., & Zhu, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae644 · Cited by 21
“Note that other specific language models, such as BioBERT [33] and BLURB [34], are also applicable to this task.”
Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework
Chang, Y., Yin, J., Li, J., Liu, C., Cao, L., & Lin, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02132-5 · Cited by 23
“Therefore, pre-training from scratch with knowledge specific to the medical field is crucial [41] and serves as an important approach to addressing challenges in medical language understanding.”
Al-Kateb, G., CENGİZ, E., & Gök, M. (2025) · Mesopotamian Journal of Artificial Intelligence in Healthcare · doi:10.58496/mjaih/2025/015 · Cited by 22
Park, Y., Lee, J., Moon, H., Choi, Y. S., & Rho, M. (2021) · Scientific Reports · doi:10.1038/s41598-021-83966-8 · Cited by 20
Knowledge-Driven Online Multimodal Automated Phenotyping SystemPreprint
Xiong, X., Sweet, S. M., Liu, M., Hong, C., Bonzel, C.-L., Panickan, V. A., Zhou, D., Wang, L., Costa, L., Ho, Y.-L. A., et al. (2023) · medRxiv · doi:10.1101/2023.09.29.23296239 · Cited by 20
“For pre-trained language models, we includedBioBert (30), PubmedBert (31), and SAPBert (32) represented by the descriptions of the code or CUI.”
MethodMethodology reporting improved over time in 176,469 randomized controlled trials
Kilicoglu, H., Jiang, L., Hoang, L., Mayo-Wilson, E., Vinkers, C. H., & Otte, W. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.004 · Cited by 19
Tan, Y., Dede, M., Mohanty, V., Dou, J., Hill, H., Bernstam, E., & Chen, K. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104648 · Cited by 19
ProtNote: a multimodal method for protein–function annotation
Char, S., Corley, N., Alamdari, S., Yang, K., & Amini, A. P. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf170 · Cited by 20
“We tested general-domain sentence transformers such as E5 (Wang et al. 2024) and biomedical-specific masked language models that are not trained to produce sentence embeddings (Gu et al. 2022, Luo et al. 2022), and observed that general-domain sentence transformers yielded improved performance.”
LLMs4OL 2025 Overview: The 2nd Large Language Models for Ontology Learning Challenge
Giglou, H. B., D’Souza, J., Mihindukulasooriya, N., & Auer, S. (2025) · Open Conference Proceedings · doi:10.52825/ocp.v6i.2913 · Cited by 21
“Furthermore, simple character-level data augmentation (insertion, deletion, swapping, or substitution) significantly improved performance for PubMedBERT [45] on the OBI dataset.”
Auto Response Generation in Online Medical Chat Services
Jahanshahi, H., Kazmi, S., & Çevik, M. (2022) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-022-00118-x · Cited by 22
“• For our BERT model, we apply a BERT processor and encode it using a vocabulary for English extracted from Wikipedia and BooksCorpus and finetuned on PubMed abstracts [52].”
MethodNavaz, A. N., Kassabi, H. T. E., Serhani, M. A., Oulhaj, A., & Khalil, K. M. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12050768 · Cited by 23
“Five potential directions are available for further improvement: (1) establish how PSN can be applied in survival analysis and implement a cardiovascular risk calculator; (2) address scalability issues when similarity matrices increase in size; (3) enhance the model to support values other than classes of nominal outc…”
MethodInfluentialNatural language processing to identify case factors in child protection court proceedings
Coulthard, B., & Taylor, B. J. (2022) · Methodological Innovations · doi:10.1177/20597991221115967 · Cited by 20
“Effective NLP, however, requires not just a mastery of computer techniques, but a solid understanding of the textual material and its domain specific context (Gu et al., 2022).”
BackgroundBERT-based Topic Modeling Approach for Malaria Research Publication
Hidayat, A. A., Nirwantono, R., Budiarto, A., & Pardamean, B. (2022) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis56303.2022.10017743 · Cited by 19
“On the other hand, Gu et al. provided a state-of-art pre-trained model called PubMedBERT [17].”
BackgroundMethodInfluentialGharagozlou, H., Mohammadzadeh, J., Bastanfard, A., & Ghidary, S. S. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3592601 · Cited by 24
Dong, J. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3605210 · Cited by 19
“In NLP, word segmentation is the process of dividing a continuous text sequence into meaningful words [ 13 , 14 ].”
Bastan, M., Surdeanu, M., & Balasubramanian, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.374 · Cited by 30
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 24
Lo, K., Shen, Z., Newman, B., Chang, J., Authur, R., Bransom, E., Candra, S., Chandrasekhar, Y., Huff, R., Kuehl, B., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.45 · Cited by 36
“…(Lo et al., 2020; Wang et al., 2020) forms a critical first step in pretraining language models of science (Beltagy et al., 2019; Lee et al., 2019; Gu et al., 2020; Luo et al., 2022; Taylor et al., 2022; Tre-wartha et al., 2022; Hong et al., 2023), automatic generation of more accessible paper…”
BackgroundLi, Z., Belkadi, S., Micheletti, N., Han, L., Shardlow, M., & Nenadić, G. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00042 · Cited by 27
“Focus on the biomedical domain, [28] carried out fine-tuning stability investigation using the BLURB data set (Biomedical Language Understanding and Reasoning Benchmark) from [39].”
MethodYan, H., Wang, S., Liu, H., Mamitsuka, H., & Zhu, S. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae401 · Cited by 23
“To embed the textual description, we use PubMedBERT (Gu et al. 2021), which is pre-trained by millions of biomedical papers to obtain state-of-the-art performance on specialized tasks.”
MethodA Retrieval-augmented Generation application for Question-Answering in Nutrigenetics Domain
Benfenati, D., Filippis, G. M. D., Rinaldi, A. M., Russo, C., & Tommasino, C. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.467 · Cited by 21
Alsaedi, S. B., Gao, X., & Gojobori, T. (2025) · FEBS Open Bio · doi:10.1002/2211-5463.70003 · Cited by 21
Raval, S., Sedghamiz, H., Santus, E., Alhanai, T., Ghassemi, M. M., & Chersoni, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.300 · Cited by 25
Deep learning models in detection of dietary supplement adverse event signals from Twitter
Wang, Y., Zhao, Y., Schutte, D., Bian, J., & Zhang, R. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab081 · Cited by 21
Naderi, N., Knafou, J., Copara, J., Ruch, P., & Teodoro, D. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.689803 · Cited by 27
Applying BERT Embeddings to Predict Legal Textual Entailment
Wehnert, S., Dureja, S., Kutty, L., Sudhi, V., & Luca, E. W. D. (2022) · The Review of Socionetwork Strategies · doi:10.1007/s12626-022-00101-3 · Cited by 22
Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction
Yeh, H.-S., Lavergne, T., & Zweigenbaum, P. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x38wgwt3eai · Cited by 19
Extracting medication changes in clinical narratives using pre-trained language models
Ramachandran, G. K., Lybarger, K., Liu, Y., Mahajan, D., Liang, J. J., Tsou, C.-H., Yetişgen, M., & Uzuner, Ö. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104302 · Cited by 20
Ferdowsi, S., Knafou, J., Borissov, N., Alvarez, D. V., Mishra, R., Amini, P., & Teodoro, D. (2023) · Patterns · doi:10.1016/j.patter.2023.100689 · Cited by 23
“Unlike the above case and to minimize computational complexities, however, instead of fine-tuning the transformer weights we use pre-trained networks on large medical corpora and with sentence embedding capabilities.(33)”
MethodAn Evaluation Benchmark for Adverse Drug Event Prediction from Clinical Trial Results
Yazdani, A., Bornet, A., Khlebnikov, P., Zhang, B., Rouhizadeh, H., Amini, P., & Teodoro, D. (2025) · Scientific Data · doi:10.1038/s41597-025-04718-1 · Cited by 22
Dhrangadhariya, A., & Müller, H. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooac107 · Cited by 18
“PubMedBERT was trained on PubMed literature and was chosen because of its domain similarity to our training data (PubMed abstracts) and task.(39) It was tuned on fixed parameters listed in the experimental details section in the Supplementary Material.”
MethodJiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
Lafourcade, C., Kérourédan, O., Ballester, B., & Richert, R. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105764 · Cited by 20
Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning
Shen, C., Lian, C., Zhang, W., Wang, F., Zhang, J., Fan, S., Wei, X., Wang, G., Li, K., Mu, H., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62060-x · Cited by 18
Singh, P., Haimovich, J. S., Reeder, C., Khurshid, S. S., Lau, E. S., Cunningham, J. W., Philippakis, A., Anderson, C. D., Ho, J. E., Lubitz, S. A., et al. (2022) · JMIR Medical Informatics · doi:10.2196/38178 · Cited by 19
“BERT was originally pretrained on general domain text such as Wikipedia but has since been adapted for use in clinical applications by pretraining on domain-specific text [11-14].”
InfluentialChambon, P., Cook, T. S., & Langlotz, C. P. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00714-8 · Cited by 27
“In general, according to the BLURB benchmark [8], the best pre-training on biomedical tasks is PubMedBERT, as it is a from-scratch pre-training with a biomedical vocabulary.”
BackgroundDDI-MuG: Multi-aspect graphs for drug-drug interaction extraction
Yang, J., Ding, Y., Long, S., Poon, J., & Han, S. C. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1154133 · Cited by 18
“, 2019) No No DDI-MuG (Ours) PubMedBERT(Gu et al., 2021) Dependency parse PMI”
BackgroundFine-Tuning GPT on Biomedical NLP Tasks: An Empirical Evaluation
Bousselham, H., Nfaoui, E. H., & Mourhir, A. (2024) · International Conference on Computer, Electrical & Communication Engineering (ICCECE) · doi:10.1109/iccece58645.2024.10497313 · Cited by 27
“…Extraction [12] F1 DDI [12], ADE [18] Named Entity Recognition [14] F1 BC5CDR [12], CADEC [14], ADE [18] Relation Classification [17] F1 GAD [19], ADE [14] Prediction [15] ROC-AUC SIDER [15], BACE [15] [12] Accuracy BioAsq [12], PubMedQA [12] [19] F1 HoC [19] Text Similarity[19] Pearson…”
BackgroundMethodInfluentialImproving biomedical entity linking for complex entity mentions with LLM-based text simplification
Borchert, F., Llorca, I., & Schapranow, M.-P. (2024) · Database · doi:10.1093/database/baae067 · Cited by 18
The overview of the BioRED (Biomedical Relation Extraction Dataset) track at BioCreative VIII
Islamaj, R., Lai, P.-T., Wei, C.-H., Luo, L., de Almeida, T. D., Jonker, R. A. A., Conceição, S. I. R., Sousa, D., Phan, C.-P., Chiang, J.-H., et al. (2024) · Database · doi:10.1093/database/baae069 · Cited by 19
“…prefixes like “MESH:” and “tmVar” and standardizing notations, such as changing “CVCL:” to “CVCL_” and “RS#:” to “RS.” This study uses PubMedBERT [50], which is a state-of-the-art pretrained language model with specialization in the medical domain, to fine-tune a text classification model on the…”
MethodInfluentialZeng, Y., Liao, B., Li, Z., Hua, C., & Li, S. (2024) · IEEE Access · doi:10.1109/access.2024.3461756 · Cited by 23
“In [106], the author developed a comprehensive biomedical NLP benchmark and conducted experiments that showed domain-specific pre-training can achieve new and state-of-the-art results across various biomedical NLP tasks.”
BackgroundMøller, A. G., van der Goot, R., & Plank, B. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.44 · Cited by 20
Coreference Resolution for the Biomedical Domain: A Survey
Lu, P., & Poesio, M. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.2 · Cited by 22
Cross-Domain Limitations of Neural Models on Biomedical Relation Classification
Alimova, I., Tutubalina, E. V., & Nikolenko, S. (2021) · IEEE Access · doi:10.1109/access.2021.3135381 · Cited by 18
“Reported results of different neural networks vary substantially on different corpora, with, for example, the F1-measure (a common metric in this problem) ranging at least from 75% to 86% on scientific abstracts [2, 6, 11] and from 81% to 90% on electronic health records [1, 8].”
BackgroundMethodClinical Flair: A Pre-Trained Language Model for Spanish Clinical Natural Language Processing
Rojas, M. S., Dunstan, J., & Villena, F. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.9 · Cited by 18
“In relation to biomedical information retrieval (IR) tasks in English, the most well-known architectures are BioBERT (Lee et al., 2020), Clinical BERT (Alsentzer et al., 2019), SciBERT (Beltagy et al., 2019), Pubmed BERT (Gu et al., 2022), BioELMo (Jin et al., 2019) and Pubmed Flair.”
BackgroundA Pre-Training Technique to Localize Medical BERT and to Enhance Biomedical BERTPreprint
Wada, S., Takeda, T., Manabe, S., Konishi, S., Kamohara, J., & Matsumura, Y. (2020) · doi:10.21203/rs.3.rs-103477/v1 · Cited by 20
Tanaka, Y., Chen, H. Y., Belloni, P., Gisladottir, U., Kefeli, J., Patterson, J., Srinivasan, A., Zietz, M., Sirdeshmukh, G., Berkowitz, J., et al. (2025) · Med · doi:10.1016/j.medj.2025.100642 · Cited by 17
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17
MedSumGraph: enhancing GraphRAG for medical QA with summarization and optimized prompts
Kim, D.-H., Yoo, S., & Jeong, O. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103311 · Cited by 22
LMs go Phishing: Adapting Pre-trained Language Models to Detect Phishing Emails
Misra, K., & Rayz, J. T. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00028 · Cited by 21
“, biomedical text classification [24]) by adapting pre-trained LMs on specialized corpora that approximate the desired domain.”
MethodToward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities
Chen, Z., Varma, M., Wan, X., Langlotz, C. P., & Delbrouck, J.-B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.41 · Cited by 24
NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
Jullien, M., Valentino, M., Frost, H., O’Regan, P., Landers, D., & Freitas, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1041 · Cited by 54
Language model and its interpretability in biomedicine: A scoping review
Lyu, D., Wang, X., Chen, Y., & Wang, F. (2024) · iScience · doi:10.1016/j.isci.2024.109334 · Cited by 26
Kharitonova, K., Pérez‐Fernández, D., Gutiérrez-Hernando, J., Gutiérrez-Fandiño, A., Callejas, Z., & Griol, D. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2321959 · Cited by 21
“Previous work has used LLMs to streamline the production of systematic reviews of the medical literature, for example, by helping to identify relevant studies (Lee et al. 2020; Miwa et al. 2014) or by automating data extraction (Gu et al. 2021; Wallace et al. 2016).”
Qin, J., Liu, C., Cheng, S. L., Guo, Y., & Arcucci, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447326 · Cited by 23
“…we evaluated its performance using different combinations among three pre-trained image encoders, namely ResNet autoencoder [15, 16], DINOv2-small and DINOv2-base [17], and five pre-trained language models, which are BERT [18], BioBERT [19], ClinicalBERT [20], CXR-BERT [12] and PubMedBERT [21].”
MethodEntity linking for English and other languages: a survey
Guellil, I., García‐Domínguez, A., Lewis, P. R., Hussain, S., & Smith, G. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-023-02059-2 · Cited by 21
Vakili, T., Henriksson, A., & Dalianis, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02546-8 · Cited by 24
“…domain – e.g., Wikipedia and other data sources on the Internet – research suggests that using generic PLMs in highly specialized domains such as healthcare may be suboptimal due to significant domain differences [11, 12], even in the presence of large language models like T5-XL and GPT-3 [13].”
BackgroundJozani, M., Williams, J. A., AlEroud, A. F., & Bhagat, S. (2025) · Information Systems Frontiers · doi:10.1007/s10796-024-10566-y · Cited by 17
Adaptive Compressed-Based Privacy-Preserving Large Language Model for Sensitive Healthcare
Gong, X., Gao, J., Sun, S. M., Zhong, Z., Shi, Y., Zeng, H., & Yang, K. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3558935 · Cited by 22
Self-citationHiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
Gökdemir, O., Siebenschuh, C., Brace, A., Wells, A. I., Hsu, B., Hippe, K., Setty, P. V., Ajith, A., Pauloski, J. G., Sastry, V. K., et al. (2025) · Platform for Advanced Scientific Computing Conference (PASC) · doi:10.1145/3732775.3733586 · Cited by 27
“We encode this corpus with PubMedBERT, an encoder-based LLM specifically fine-tuned on PubMed articles [34].”
Clinical QA 2.0- Multi-Task Learning for Answer Extraction and Categorization
Pattnayak, P., Patel, H., Agarwal, A., Panda, S., Kumar, B., & Kumar, T. (2025) · IEEE International Conference on Electro Information Technology · doi:10.1109/eit64391.2025.11103631 · Cited by 17
“PubMedBERT [9] eliminated domain mismatches by pre-training exclusively on PubMed abstracts.”
WNUT-2020 Task 1 Overview: Extracting Entities and Relations from Wet Lab Protocols
Tabassum, J., Xu, W. H., & Ritter, A. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.33 · Cited by 17
Knafou, J., Naderi, N., Copara, J., Teodoro, D., & Ruch, P. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.40 · Cited by 17
Identifying Topics of Scientific Articles with BERT-Based Approaches and Topic Modeling
Glazkova, A. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75015-2_10 · Cited by 17
Sun, C. H., Yang, Z., Wang, L., Zhang, Y., Lin, H., & Wang, J. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103956 · Cited by 20
ProTranslator: Zero-Shot Protein Function Prediction Using Textual Description
Xu, H., & Wang, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04749-7_17 · Cited by 30
Sarrouti, M., Tao, C., & Randriamihaja, Y. M. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.37 · Cited by 29
“On average (micro), T5 which was only pre-trained on the general domain corpus, obtained a higher F1 score than BioBERT and PubMedBERT.”
BackgroundMethodInfluentialDeep learning models are not robust against noise in clinical textPreprint
Moradi, M., Blagec, K., & Samwald, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12242 · Cited by 18
“However, discussing how well the language models can perform on the benchmarks is out of the scope of this paper; we refer the reader to the respective papers for extensive discussion [10-13].”
BackgroundYang, R., Zeng, Q., You, K., Qiao, Y., Huang, L., Hsieh, C.-C., Rosand, B., Goldwasser, J., Dave, A., Keenan, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/60601 · Cited by 16
“…the use of domain-specific pretrained language models, transitioning from shallow embeddings such as BioWordVec [5] and BioSentVec [6] to advanced architectures like Bidirectional Encoder Representations from Transformers (BERT) [7], such as BioBERT [8], ClinicalBERT [9], and PubMedBERT [10].”
Deep learning approach to detection of colonoscopic information from unstructured reports
Seong, D., Choi, Y. H., Shin, S.-Y., & Yi, B.-K. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02121-7 · Cited by 16
“For the BioBERT model, we used an existing pre-trained contextualized word embedding, BiomedNLP-PubMedBERT, which was pre-trained using abstracts from PubMed and full-text articles from PubMedCentral [61, 69].”
MethodInfluential
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.