Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 301–400 of 1,030
Automated ICD coding using extreme multi-label long text transformer-based models
Liu, L., Perez‐Concha, O., Nguyen, A. N., Bennett, V., & Jorm, L. R. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102662 · Cited by 52
“In the biomedical domain, many domain-specific Transformer models have been continually pretrained or pretrained from scratch using the collection of PubMed 1 biomedical abstracts and/or clinical notes from MIMIC-III, including ClinicalBERT[38], PubMedBERT[39], ClinicalplusXLNet[11] and RoBERTa-PM-M3[40].”
MethodSim, J.-A., Huang, X., Horan, M. R., Baker, J. N., & Huang, I. (2024) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2024.2322664 · Cited by 43
“Finally, pretraining efforts still rely on the corpora of generic domains (e.g. news or Wikipedia articles) that may not be generalizable to cancer populations [55].”
BackgroundMethodKET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAG
Huang, Y., Zhang, S., & Xiao, X. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737012 · Cited by 67
“Akin to Algorithm 1, Delile et al. [7] employ Pubmed-BERT [12] to extract triplets from biomedical texts and link entities to the text chunks in which they appear.”
Narayanan, S., Mannam, K., Achan, P., Ramesh, M. V., Rangan, P. V., & Rajan, S. P. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103960 · Cited by 43
Hierarchical shared transfer learning for biomedical named entity recognition
Chai, Z., Han, J., Shi, S., Zhan, S., Zhuo, L., & Yang, Y. (2022) · BMC Bioinformatics · doi:10.1186/s12859-021-04551-4 · Cited by 46
“The BioBERT model and the PubMedBert model have achieved significant improvement in many biomedical tasks by pre-training BERT models using medical materials and fine-tuning them [26, 27].”
MethodAutomating Materials Exploration with a Semantic Knowledge Graph for Li‐Ion Battery Cathodes
Nie, Z., Zheng, S., Liu, Y., Chen, Z., Li, S., Lei, K., & Pan, F. (2022) · Advanced Functional Materials · doi:10.1002/adfm.202201437 · Cited by 45
Boosting Transformers and Language Models for Clinical Prediction in Immunotherapy
Chen, Z., Balan, M. M., & Brown, K. M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.32 · Cited by 43
“However, domain-agnostic LLMs like GPT-3 have so far been unable to achieve competitive results on biomedical NLP tasks (Moradi et al., 2021; Gutierrez et al., 2022), revealing the fact that the relevance and the knowledge reservation of pretraining sources have a significant impact to the knowledge…”
BackgroundKnowledge-based dynamic prompt learning for multi-label disease diagnosis
Xie, J., Li, X., Yuan, Y., Guan, Y., Jiang, J., Guo, X., & Peng, X. X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111395 · Cited by 44
Evaluating large language models in echocardiography reporting: opportunities and challenges
Chao, C., Banerjee, I., Arsanjani, R., Ayoub, C., Tseng, A. S., Delbrouck, J.-B., Kane, G. C., López-Jiménez, F. A., Attia, Z. I., Oh, J. K., et al. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae086 · Cited by 44
“31”
Towards robust foundation models for digital pathology
Kömen, J., de Jong, E. D., Hense, J., Marienwald, H., Dippel, J., Naumann, P., Marcus, E., Ruff, L., Alber, M., Teuwen, J., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-73923-2 · Cited by 58
Pabón, O. S., Montenegro, O., Torrente, M., Rodríguez‐González, A., Provencio, M., & Ernestina, M. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.913 · Cited by 39
“, 2019) have shown that such representations are able to improve performance on a wide range of natural language processing tasks (Liu, Kusner & Blunsom, 2020; Gu et al., 2020; Pires, Schlinger & Garrette, 2019).”
BackgroundMahbod, A., Saeidi, N., Hatamikia, S., & Woitek, R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110571 · Cited by 44
“BioMedCLIP employs a vision transformer (ViT-B/16-224) [36] as the image encoder and PubMed-BERT [49] with an extended token size of 256 as the text encoder.”
Knowledge-enhanced pretraining for vision-language pathology foundation model on cancer diagnosis
Zhou, X., Sun, L., He, D., Guan, W., Wang, G., Wang, R., Wang, L., Yuan, X., Sun, X., Zhang, Y., et al. (2026) · Cancer Cell · doi:10.1016/j.ccell.2026.01.019 · Cited by 50
Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model
Blanchard, A. E., Chandrashekar, M., Gao, S. B., Gounley, J. P., Lyngaas, I., Gläser, J., & Bhowmik, D. (2022) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2022.3144045 · Cited by 41
Ontology-driven and weakly supervised rare disease identification from clinical notes
Dong, H., Suárez-Paniagua, V., Zhang, H., Wang, M., Casey, A., Davidson, E., Chen, J., Alex, B., Whiteley, W. N., & Wu, H. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02181-9 · Cited by 46
“The notable models include but are not limited to BlueBERT [19] (BERT further pre-trained with PubMed abstracts and MIMIC-III clinical notes), PubMedBERT [32] (pre-trained from scratch with PubMed abstracts and full texts), SapBERT [33] (PubMed-BERT further pre-trained with UMLS concepts), etc.”
Bioformer: an efficient transformer language model for biomedical text miningPreprint
Li, F., Chen, Q., Wei, C.-H., Lu, Z., & Wang, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01588 · Cited by 57
“Consistent with the existing studies[12], we select the best batch size among 8, 16, and 32, and learning rate among 1e-5, 3e-5, and 5e-5 on the development set.”
BackgroundMethodInfluentialClinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“…[171] CASI [150], n2c2 [111], i2b2 [93], PubMedQA [108], MedMCQA [118], emrQA [113], BIOASQ [103] quEHRy [172], BiomedBERT [85], PubMedBERT [86], BioGPT [89], Llava-med [90] Summarization and Paraphrasing Summarizing Clinical Study Reports [173], Summarizing Patient-Provider Dialogues…”
Aviation-BERT: A Preliminary Aviation-Specific Natural Language Model
Chandra, C., Jing, X., Bendarkar, M. V., Sawant, K., Elias, L. R., Kirby, M. R., & Mavris, D. N. (2023) · AIAA AVIATION Forum · doi:10.2514/6.2023-3436 · Cited by 37
“…is called pre-training from scratch, where the LLM is pre-trained with no text data other than the domain-specific text This latter approach generally results in better outcomes for downstream tasks like classification, question answering (QA), etc. but requires a large available dataset [24 .”
BackgroundMethodInfluentialBioinformatics and biomedical informatics with ChatGPT: Year one review
Wang, J., Cheng, Z., Yao, Q., Liu, L., Xu, D. H., & Hu, G. (2024) · Quantitative Biology · doi:10.1002/qub2.67 · Cited by 41
“While its performance in question answering closely matched SOTA models like PubmedBERT [24], ChatGPT ‐ 3.5 showed limitations in other tasks, with similar observations made for ChatGPT ‐ 4 [7,25,26].”
InfluentialUse of SNOMED CT in Large Language Models: Scoping Review
Chang, E., & Sung, S. (2024) · JMIR Medical Informatics · doi:10.2196/62924 · Cited by 50
“%) used LLMs trained on large-scale biomedical corpora, such as BioBERT [14] and PubMedBERT [60], which were trained on PubMed articles, and ClinicalBERT [61] and EHRBERT [23], which were trained on clinical notes.”
MethodArtificial intelligence for drug delivery: Yesterday, today and tomorrow
Wu, Y., Wang, N., Xiong, P., Wang, R., Deng, J., & Ouyang, D. (2025) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2025.09.022 · Cited by 45
“Biomedical LLMs such as BioGPT 157 and PubMedBERT 158 excel at understanding medical literature and biological concepts.”
Biology-driven insights into the power of single-cell foundation models
Wu, J., Ye, Q., Wang, Y., Hu, R., Zhu, Y., Yin, M., Wang, T., Wang, J., Hsieh, C., & Hou, T. (2025) · Genome biology · doi:10.1186/s13059-025-03781-6 · Cited by 49
“LangCell LangCell [9] is the first scFM that employs the language-cell pretraining framework, which consists of a cell encoder and a text encoder initialized from Gen-eformer and PubMedBERT [65], respectively.”
Ding, J.-E., Thao, P. N. M., Peng, W.-C., Wang, J.-Z., Chug, C.-C., Hsieh, M.-C., Tseng, Y.-C., Chen, L., Luo, D., Wu, C., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-71020-2 · Cited by 36
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
Sun, Y., Wu, H., Zhu, C., Zheng, S., Chen, Q., Zhang, K., Zhang, Y., Wan, D., Lan, X., Zheng, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_4 · Cited by 99
Generative AI Models in Time-Varying Biomedical Data: Scoping Review
He, R., Sarwal, V., Qiu, X., Zhuang, Y., Zhang, L., Liu, Y., & Chiang, J. N. (2024) · Journal of Medical Internet Research · doi:10.2196/59792 · Cited by 43
“Researchers developed the BERT for Biomedical Text Mining [37] and PubMedBERT [38] transformer models using biomedical literature from PubMed.”
Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study
Yang, Z., Yao, Z., Tasmin, M., Vashisht, P., Jang, W. S., Ouyang, F., Wang, B., McManus, D. D., Berlowitz, D. R., & Yu, H. Q. (2025) · Journal of Medical Internet Research · doi:10.2196/65146 · Cited by 49
“Early models such as PubMedBERT [39] with 100 million parameters score around 38.3% in USMLE.”
InfluentialEventBERT: A Pre-Trained Model for Event Correlation Reasoning
Zhou, Y., Geng, X., Shen, T., Long, G., & Jiang, D. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511928 · Cited by 108
“Third, we only access limited computation resources and perform continual pre-training from RoBERTa, while previous works (Gu et al., 2020) verify learning with task-specific corpora and objectives from scratch greatly improves.”
BackgroundIs Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks
Peng, B., Chersoni, E., Hsu, Y.-Y., & Huang, C. (2021) · Workshop on Economics and Natural Language Processing (ECONLP) · doi:10.18653/v1/2021.econlp-1.5 · Cited by 54
Zhou, D., Gan, Z., Shi, X., Patwari, A., Rush, E. N., Bonzel, C.-L., Panickan, V. A., Hong, C., Ho, Y.-L. A., Cai, T. A., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104147 · Cited by 47
Preston, S., Wei, M., Rao, R., Tinn, R., Usuyama, N., LUCAS, M. A., 池谷, 裕., Weerasinghe, R. K., Lee, S., Piening, B., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100726 · Cited by 43
Self-citationPrompt Tuning in Biomedical Relation Extraction
He, J., Li, F., Li, J., Hu, X., Nian, Y., Xiang, Y., Wang, J., Wei, Q., Li, Y., Xu, H., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00162-9 · Cited by 42
Benchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35
Lyu, W., Dong, X., Wong, R., Zheng, S., Abell-Hart, K., Wang, F., & Chen, C. (2022) · arXiv · doi:10.48550/arxiv.2208.10240 · Cited by 46
“BLURB(Gu et al., 2021) benchmark is a recent work that released state-of-the-art pretrained and task-specific models for the community.”
BackgroundCesario, E., Comito, C., & Zumpano, E. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127079 · Cited by 39
Toward Unified AI Drug Discovery with Multimodal Knowledge
Luo, Y., Liu, X. Y., Yang, K., Huang, K., Hong, M., Zhang, J., Wu, Y., & Nie, Z. (2024) · Health Data Science · doi:10.34133/hds.0113 · Cited by 43
“To encode the unstructured knowledge D UK and P UK , we adopt PubMedBERT [28], a language model pretrained on biomedical corpus.”
MethodSelf-citationKnowledge graph construction for heart failure using large language models with prompt engineering
Xu, T., Gu, Y., Xue, M., Gu, R., Li, B., & Gu, X. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1389475 · Cited by 40
Self-citationWithers, C. A., Rufai, A. M., Venkatesan, A., Tirunagari, S., Lobentanzer, S., Harrison, M., & Zdrazil, B. (2025) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2025.2490835 · Cited by 40
“In drug discovery, BERT and its biomedical variants (e.g. BioBERT, PubMedBERT) [36,59–61] have proven highly effective for NER tasks.”
Large language models for batteries
Zuo, W., Zheng, H., He, T., Vishwanath, V., Chan, M. K. Y., Stevens, R. L., Amine, K., & Xu, G. (2025) · Joule · doi:10.1016/j.joule.2025.102037 · Cited by 39
ChatGPT in Medical Education: Bibliometric and Visual Analysis
Zhang, Y., Xie, X., & Xu, Q. (2025) · JMIR Medical Education · doi:10.2196/72356 · Cited by 42
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation
Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021) · doi:10.18653/v1/2021.acl-long.259 · Cited by 61
Literature-Augmented Clinical Outcome Prediction
Naik, A., Parasa, S., Feldman, S., Wang, L. L., & Hope, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.33 · Cited by 61
Multimodal learning on graphs for disease relation extraction
Lin, Y., Lu, K., Yu, S., Cai, T., & Žitnik, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104415 · Cited by 45
Knowledge-Rich Self-Supervision for Biomedical Entity Linking
Zhang, S., Cheng, H., Vashishth, S., Wong, C., Xiao, J., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.61 · Cited by 63
Self-citationOne Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial SupportPreprint
Stern, R., Rasiah, V., Matoshi, V., Bose, S. B., Stuermer, M., Chalkidis, I., Ho, D. E., & Niklaus, J. (2023) · arXiv (Cornell University) · doi:10.24451/arbor.19713 · Cited by 36
ELECTRAMed: a new pre-trained language representation model for biomedical NLP
Miolo, G., Mantoan, G., & Orsenigo, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09585 · Cited by 18
“However, it has been recently observed that for domains in which large corpora exist, like the biomedical field, pre-training language models from scratch yields better results than feeding the pre-training phase with general-domain knowledge [6].”
BackgroundEntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Lin, C., Miller, T., Dligach, D., Bethard, S. J., & Savova, G. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.21 · Cited by 55
Scancar, B., Byrne, J. A., Causeur, D., & Barnett, A. G. (2026) · BMJ · doi:10.1136/bmj-2025-087581 · Cited by 39
GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
Gao, Y., Xiong, Y., Wang, S., & Wang, H. (2022) · Applied Sciences · doi:10.3390/app122412942 · Cited by 34
Bridging the Worlds of Pharmacometrics and Machine Learning
Stankevičiūtė, K., Woillard, J., Peck, R., Marquet, P., & van der Schaar, M. (2023) · Clinical Pharmacokinetics · doi:10.1007/s40262-023-01310-x · Cited by 35
“Another relevant line of work in machine learning for biomarker identification and knowledge transfer includes language models for large-scale medical text preprocessing [56] and knowledge graph development [57–61], where link prediction could be used to quantify the relationships of interest.”
BackgroundBridging the Data Provenance Gap Across Text, Speech and VideoPreprint
Longpre, S., Singh, N., Cherep, M., Tiwary, K., Materzyńska, J., Brannon, W., Mahari, R., Obeng-Marnu, N., Dey, M., Hamdy, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17847 · Cited by 51
GatorTron: A Large Language Model for Clinical Natural Language Processing
Yang, X., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2022) · doi:10.1101/2022.02.27.22271257 · Cited by 33
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 44
Xiao, Y., Zhang, S., Zhou, H., Li, M., Yang, H., & Zhang, R. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104730 · Cited by 39
WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering
Chen, P., Zhu, C., Zheng, S., Li, H., & Yang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72764-1_23 · Cited by 81
HERGen: Elevating Radiology Report Generation with Longitudinal Data
Wang, F., Du, S., & Yu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73001-6_11 · Cited by 69
Biomedical Relation Extraction With Knowledge Graph-Based Recommendations
Sousa, D., & Couto, F. M. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3173558 · Cited by 39
“We chose BiOnt over other biomedical or even non-biomedical [25], [26] RE systems [27], due to its use of similar external entity knowledge.”
MethodGraph-Based Retriever Captures the Long Tail of Biomedical KnowledgePreprint
Delile, J., Mukherjee, S., Pamel, A. V., & Евгеньевич, Ж. Л. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12352 · Cited by 42
“Finally, it disam-biguates and/or merges overlapping candidate entities in input text chunks. ii) The PubmedBERT model (Gu et al., 2020),a BERT architecture based encoder pre-trained on Pubmed abstracts and PMC full-text articles and fine-tuned on the BioRED dataset, was used for RE from scientific…”
The changing landscape of text mining: a review of approaches for ecology and evolution
Farrell, M. J., Guillarme, N. L., Brierley, L., Hunter, S. B., Scheepens, D. R., Willoughby, A. R., Yates, A., & Mideo, N. (2024) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2024.0423 · Cited by 42
Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review
Yang, E., MacAvaney, S., Lewis, D. D., & Frieder, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_34 · Cited by 32
Analyzing the vast coronavirus literature with CoronaCentral
Lever, J., & Altman, R. B. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2100766118 · Cited by 32
BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
Monajatipoor, M., Rouhsedaghat, M., Li, L. H., Kuo, C. ‐. J., Chien, A., & Chang, K.-W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_69 · Cited by 48
Xiong, Y., Peng, H., Xiang, Y., Wong, K., Chen, Q., Yan, J., & Tang, B. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104035 · Cited by 32
Combining human and machine intelligence for clinical trial eligibility querying
Fang, Y., Idnay, B., Sun, Y., Liu, H., Chen, Z., Marder, K. S., Xu, H., Schnall, R., & Weng, C. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac051 · Cited by 34
How Do Your Biomedical Named Entity Recognition Models Generalize to Novel Entities?
Kim, H., & Kang, J. (2022) · IEEE Access · doi:10.1109/access.2022.3157854 · Cited by 34
“SOTA Models We use BioBERT [25], BlueBERT [35], and PubMedBERT [15].”
BackgroundMethodDR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing
Gao, Y., Dligach, D., Miller, T. A., Caskey, J., Sharma, B., Churpek, M. M., & Afshar, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104286 · Cited by 54
“BENCH has a fine-tuning and multi-tasking setup similar to the GLUE and SQuAD benchmarks in general domain NLP [51, 52], and BLURB in biomedical NLP [53], but stands out with a distinct focus and the first cNLP benchmark on promoting clinical diagnostic reasoning.”
Bernice: A Multilingual Pre-trained Encoder for Twitter
DeLucia, A., Wu, S., Mueller, A., Aguirre, C. A., Resnik, P., & Dredze, M. H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.415 · Cited by 61
“Studies have shown that pre-training data should match the intended deployment domain to achieve optimal task performance (Barbieri et al., 2021; Nguyen et al., 2020; Huang et al., 2020a; Gururangan et al., 2020; Gu et al., 2021; Zhang et al., 2022).”
BackgroundPre-training technique to localize medical BERT and enhance biomedical BERTPreprint
Wada, S., Takeda, T., Manabe, S., Konishi, S., Kamohara, J., & Matsumura, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07202 · Cited by 32
Liu, L., Yang, X., Lei, J., Shen, Y., Wang, J., Wei, P., Chu, Z., Qin, Z., & Ren, K. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3709941 · Cited by 31
“A synchronized training mechanism is formulated to retrieve clinical guidelines and leverage medical knowledge, significantly improving the clinical reasoning capabilities of LLMs and decrease the requirements for computational resources.”
Missing-modality enabled multi-modal fusion architecture for medical data
Wang, M., Fan, S., Li, Y., Xie, Z., & Chen, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104796 · Cited by 31
UniChest: Conquer-and-Divide Pre-Training for Multi-Source Chest X-Ray Classification
Dai, T., Zhang, R., Hong, F., Yao, J., Zhang, Y., & Wang, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3381123 · Cited by 32
“1 After the entity extraction, we use PubMedBERT [38] pre-trained on Unified Medical Language System (UMLS) data [39] as textual encoder backbone (cid:56) text for generating text representation [7].”
MethodLi, J., Li, Y., Pan, Y., Guo, J., Sun, Z., Li, F., He, Y. O., & Tao, C. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00318-x · Cited by 33
“PubMedBERT is a domain-specific language model pre-trained on large-scale biomedical corpora [64, 65].”
BackgroundContrastive Learning with Counterfactual Explanations for Radiology Report Generation
Li, M., Lin, H., Qiu, L., Liang, X., Chen, L., Saddik, A. E., & Chang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72775-7_10 · Cited by 31
Bonfigli, A., Bacco, L., Merone, M., & Dell’Orletta⋄, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103003 · Cited by 36
Ivanisenko, T., Сергеевич, Д. П., & Иванисенко, В. А. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252111811 · Cited by 40
Hu, M., Yuan, K., Shen, Y., Tang, F., Xu, X., Zhou, L., Li, W., Chen, Y., Xu, Z., Peng, Z., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01845 · Cited by 40
Rajendran, P., Yang, Y., Niedermayr, T. R., Gensheimer, M. F., Beadle, B. M., Le, Q., Xing, L., & Dai, X. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2025.110740 · Cited by 34
GLiNER-BioMed: a suite of efficient models for open biomedical named entity recognition
Yazdani, A., Stepanov, I., & Teodoro, D. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag322 · Cited by 33
“Despite GLiNER’s promising performance in open NER, directly applying it to biomedical texts remains challenging due to specialized vocabulary, sheer volume of entities, and complex semantic structures unique to biomedical corpora (Lee et al. 2020, Gu et al. 2022).”
Jeong, Y., & Kim, E. (2022) · IEEE Access · doi:10.1109/access.2022.3180830 · Cited by 45
“BioBERT [28] and PubMedBERT [29], which are state-of-the-art models for biomedical NLP tasks, were both pretrained with biomedical domain text data collected from PubMed and PubMed Central (PMC).”
BackgroundMethodClinicalRadioBERT: Knowledge-Infused Few Shot Learning for Clinical Notes Named Entity Recognition
Rezayi, S., Dai, H., Liu, Z., Wu, Z., Hebbar, A., Burns, A. H., Zhao, L., Zhu, D., Li, Q., Liu, W., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21014-3_28 · Cited by 61
Jantscher, M., Gunzer, F., Kern, R., Hassler, E. M., Tschauner, S., & Reishofer, G. (2023) · Scientific Reports · doi:10.1038/s41598-023-29323-3 · Cited by 34
“Multiple studies have demonstrated the advantage of adaptive pre-training under both high-and low-resource settings 31,32 .”
BackgroundKEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 43
High-level implementable methods for automated building code compliance checking
Lee, J.-K., Cho, K., Choi, H., Choi, S., Kim, S., & Cha, S. H. (2023) · Developments in the Built Environment · doi:10.1016/j.dibe.2023.100174 · Cited by 39
MU-Bench: A Multitask Multimodal Benchmark for Machine UnlearningPreprint
Cheng, J., & Amiri, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.14796 · Cited by 31
“For DDI, we train BioBERT (Lee et al., 2019), PubMedBERT (Gu et al., 2021) (abstract only and full text).”
MethodA Field Guide to Automatic Evaluation of LLM-Generated Summaries
van Schaik, T. A., & Pugh, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661346 · Cited by 30
“We considered a sentence in the ground truth summary to be "found" in the AI summary if the cosine similarities between a domain-relevant PTM (PubMedBERT [41]) embedding of those sentences was above a certain threshold.”
Perlman‐Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1636 · Cited by 30
“This results in broadly improved performance.(17,18) The development of screening technologies has led to a growing body of literature on their utility.”
BackgroundMethodLanguage models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30
Rajendran, P., Chen, Y., Qiu, L., Niedermayr, T., Liu, W., Buyyounouski, M., Bagshaw, H., Han, B., Yang, Y., Kovalchuk, N., et al. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.07.2149 · Cited by 36
Košprdić, M., Prodanović, N., Ljajić, A., Bašaragin, B., & Milošević, N. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102970 · Cited by 39
“The pre-trained initial model for zero-shot learning is based on transformer architecture and utilizes domain-specific BERT-based language models (BioBERT [18] and PubMedBERT [19]).”
BackgroundMethodInfluentialTransformers for Multi-label Classification of Medical Text: An Empirical Comparison
Yogarajan, V., Montiel, J., Smith, T. C., & Pfahringer, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77211-6_12 · Cited by 41
Lai, T., Ji, H., & Zhai, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.140 · Cited by 40
PICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
“…trains BERT on the combination of BookCorpus, English Wikipedia, PubMed abstracts and PubMed Central full-text articles [20]; (3) PubMedBERT-abs, which trains BERT on PubMed abstracts only, and (4) PubMedBERT-full on a combination of PubMed abstracts and PubMed Central full-text articles [21].”
MethodTripathi, D., Hajra, K., Mulukutla, A., Shreshtha, R., & Maity, D. (2025) · Bioengineering · doi:10.3390/bioengineering12020163 · Cited by 41
“They use patient information, like symptoms from across clinical areas, to mine, categorize, analyze, and deliver data appropriately [27].”
Tavakoli, N., Abad, Z. S. H., Gowda, V., Samsel, K., Bedayat, A., Ghasemiesfe, A., Bağcı, U., Hsiao, A., Leiner, T., Carr, J., et al. (2025) · Radiology · doi:10.1148/radiol.242961 · Cited by 38
Portelli, B., Lenzi, E., Chersoni, E., Serra, G., & Santus, E. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.149 · Cited by 29
Structure-inducing pre-training
McDermott, M. B. A., Yap, B., Szolovits, P., & Žitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00647-z · Cited by 33
Chen, Y., Hao, L., Zou, V. Z., Hollander, Z., Ng, R. T., & Isaac, K. V. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01583-z · Cited by 31
“For example, state-of-the-art bio-medical language models [23–25] were trained on billions of words from Wikipedia and PubMed.”
The New Version of the ANDDigest Tool with Improved AI-Based Short Names Recognition
Ivanisenko, T., Сергеевич, Д. П., Kolchanov, N. A., & Иванисенко, В. А. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232314934 · Cited by 29
“Classification models were constructed by fine-tuning the pre-trained PubMedBERT transformer model [31,33] for the sequence classification task using Python’s transformers v4.”
MethodTCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing
Ren, M., Huang, H., Zhou, Y., Cao, Q., Bu, Y., & Gao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_16 · Cited by 46
“5B tokens PubMedBERT (Gu et al., 2021) PubMed Medical EN 3.”
Fabregat, H., Duque, A., Martínez-Romo, J., & Araujo, L. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104279 · Cited by 34
Multilingual translation for zero-shot biomedical classification using BioTranslator
Xu, H., Woicik, A., Poon, H., Altman, R. B., & Wang, S. (2023) · Nature Communications · doi:10.1038/s41467-023-36476-2 · Cited by 48
Self-citation
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.