Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 201–300 of 1,030
Self-eXplainable AI for medical image analysis: A survey and new outlooks
Hou, J., Liu, S., Bie, Y., Wang, H., Tan, A., Luo, L., & Chen, H. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104791 · Cited by 66
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 83
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards
Delbrouck, J.-B., Chambon, P., Bluethgen, C., Tsai, E., Almusa, O., & Langlotz, C. P. (2022) · arXiv · doi:10.48550/arxiv.2210.12186 · Cited by 63
“Each of our rewards leverages the outputs of the released fine-tuned PubMedBERT model on RadGraph, namely the entities and the relations, on both a generated report and its reference.”
MethodA large-scale dataset of patient summaries for retrieval-based clinical decision support systems
Zhao, Z., Jin, Q., Chen, F., Peng, T., & Yu, S. (2023) · Scientific Data · doi:10.1038/s41597-023-02814-8 · Cited by 63
Su, P., & Vijay‐Shanker, K. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04642-w · Cited by 71
“We employ the the same spilt of training, development, and test sets with the PubMed-BERT model [20] during the model evaluation.”
MethodInfluentialGPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors
Jang, D., Yun, T.-R., Lee, C.-Y., Kwon, Y. K., & Kim, C.-E. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000416 · Cited by 66
Medical large language models are susceptible to targeted misinformation attacks
Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01282-7 · Cited by 62
nach0: multimodal natural and chemical languages foundation model
Livne, M., Miftahutdinov, Z., Tutubalina, E. V., Kuznetsov, M., Polykovskiy, D. A., Brundyn, A., Jhunjhunwala, A., Costa, A. B., Aliper, A., Aspuru‐Guzik, A., et al. (2024) · Chemical Science · doi:10.1039/d4sc00966e · Cited by 101
“The table contains metrics for each task and model, with the results of the top-performing base model emphasized in bold.”
MethodMultiVerS: Improving scientific claim verification with weak supervision and full-document context
Wadden, D., Lo, K., Wang, L., Cohan, A., Beltagy, I., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.6 · Cited by 60
“Domain adaptation for scientific text has been studied in a number of works, including Gururangan et al. (2020); Beltagy et al. (2019); Lee et al. (2020); Gu et al. (2021).”
BackgroundA comprehensive survey for generative data augmentation
Chen, Y., Yan, Z., & Zhu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128167 · Cited by 76
CellVoyager: AI CompBio agent generates new insights by autonomously analyzing biological data
Alber, S., Chen, B., Sun, E., Isakova, A., Wilk, A. J., & Zou, J. (2026) · Nature Methods · doi:10.1038/s41592-026-03029-6 · Cited by 79
Radiology-GPT: A Large Language Model for Radiology
Liu, Z., Zhong, A., Li, Y., Yang, L., Ju, C., Wu, Z., Ma, C., Shu, P., Chen, C., Kim, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.08666 · Cited by 59
“Domain-Specific Language Models (DSLM) are models that are specifically trained or fine-tuned on text data pertaining to a particular domain or sector [28, 29].”
Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking
Liu, F., Vulić, I., Korhonen, A., & Collier, N. (2021) · doi:10.18653/v1/2021.acl-short.72 · Cited by 94
BioREx: Improving biomedical relation extraction by leveraging heterogeneous datasets
Lai, P., Wei, C.-H., Luo, L., Chen, Q., & Lu, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104487 · Cited by 70
Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model
Soong, D., Sridhar, S., Si, H., Wagner, J.-S., Sá, A. C. C., Yu, C. Y., Karagoz, K., Guan, M., Kumar, S., Hamadeh, H. K., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000568 · Cited by 70
“For example, BioBERT [25] targets biomedical text mining tasks, SciBERT [26] and PubMedBERT [27] address scientific literature, and Legal-BERT [28] specializes in legal text processing.”
BackgroundImproved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News
Deka, P., Jurek, A., & Deepak, P. (2022) · Journal of Data Intelligence · doi:10.26421/jdi3.4-5 · Cited by 87
Chambon, P., Wu, C., Steinkamp, J., Adleberg, J., Cook, T. S., & Langlotz, C. P. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac219 · Cited by 69
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.30 · Cited by 79
Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges
Reddy, C. K. A., & Shojaee, P. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i27.35084 · Cited by 128
Extending the WILDS Benchmark for Unsupervised AdaptationPreprint
Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., Kumar, A., Hu, W., Yasunaga, M., Marklund, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.05090 · Cited by 99
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66
“PubMedBERT [11] is an encoder-based BERT-like model that is trained from scratch using abstracts from PubMed and full-text articles from PubMedCentral in the corpus “The Pile” [10].”
MethodFaithful AI in Medicine: A Systematic Review with Large Language Models and Beyond
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · doi:10.21203/rs.3.rs-3661764/v1 · Cited by 57
Peng, L., Luo, G., Zhou, S., Chen, J., Xu, Z., Sun, J., & Zhang, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01126-4 · Cited by 79
Waldock, W. J., Zhang, J., Guni, A., Nabeel, A., Darzi, A. W., & Ashrafian, H. (2024) · Journal of Medical Internet Research · doi:10.2196/56532 · Cited by 65
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. (2023) · arXiv · doi:10.48550/arxiv.2306.04504 · Cited by 57
“The effective utilization of transformer-based (Vaswani et al., 2017) NLP models like BERT (Devlin et al., 2019) have also led to significant progress in the biomedical domain (Lee et al., 2020; Alsentzer et al., 2019; Beltagy et al., 2019; Gu et al., 2020; Peng et al., 2019) in recent years.”
BackgroundMethodZhang, J., Whebell, S. F., Gallifant, J., Budhdeo, S., Mattie, H., Lertvittayakumjorn, P., Lopez, M. D. P. A., Tiangco, B. J., Gichoya, J. W., Ashrafian, H., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00032-2 · Cited by 64
PharmBERT: a domain-specific BERT model for drug labels
ValizadehAslani, T., Shi, Y., Ren, P., Wang, J., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad226 · Cited by 60
Generative AI and DT integrated intelligent process planning: a conceptual framework
Xu, Q.-F., Zhou, G.-H., Zhang, C., Chang, F.-T., Cao, Y., & Zhao, D. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-13861-9 · Cited by 35
Boers, T. G. W., Fockens, K. N., van der Putten, J. A., Jaspers, T. J. M., Kusters, C. H. J., Jukema, J. B., Jong, M. R., Struyvenberg, M. R., de Groof, J., Bergman, J. J. G. H. M., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103298 · Cited by 68
“In Natural language processing, Gu et al. (2021) have shown that in-domain pre-training outperforms out-of-domain pre-training (pre-training on data that is not from the target domain).”
BackgroundYan, C., Fu, X., Liu, X., Zhang, Y., Gao, Y., Wu, J. Y., & Li, Q. (2022) · Intelligent Medicine · doi:10.1016/j.imed.2022.03.003 · Cited by 69
Self-citationClinical Prompt Learning With Frozen Language Models
Taylor, N., Zhang, Y., Joyce, D. W., Gao, Z., Kormilitzin, A., & Nevado‐Holgado, A. J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294633 · Cited by 68
Information extraction from clinical notes: are we ready to switch to large language models?
Hu, Y., Zuo, X., Zhou, Y., Peng, X., Huang, J., Keloth, V. K., Zhang, V., Weng, R.-L., Shyr, C., Chen, Q., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf213 · Cited by 69
“18,46 Based on our preliminary experiments, these 2 distinct BERT models demonstrated superior performance for their respective tasks, motivating our decision to use them separately.”
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Rodriques, S. G., & White, A. D. (2023) · arXiv · doi:10.48550/arxiv.2312.07559 · Cited by 55
“LLMs trained on biomedical [30, 31, 32, 33, 34] or more generalized scientific literature [9, 35] have demonstrated impressive performance on current scientific QA benchmarks.”
BackgroundHierarchical label-wise attention transformer model for explainable ICD coding
Liu, L., Perez‐Concha, O., Nguyen, A. N., Bennett, V., & Jorm, L. R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104161 · Cited by 72
“Pascual et al. [32] investigated the fine-tuning of another BERT variant (PubMedBERT) for the ICD coding task, called BERT-ICD.”
InfluentialSeveral Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 58
“0 12 billion [19] BLOOM Adam,70 layers 176 billion [34] LLaMA AdamW,65 billion [9] Jurassic-1 178 billion [137] GLM AdamW,130 billion [138] PaLM Adafactor,540 billion [139] OPT 175B AdamW, 96 175 billion [13] Chinchilla Adam,80 layers 70 billion [67] BERT-base Adam,12 layers 100 million [14] BERT-large Adam,24 layers…”
BackgroundMethodInfluentialSong, Y., Miret, S., & Liu, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.201 · Cited by 82
Deep learning methods for protein function prediction
Boadu, F., Lee, A., & Cheng, J. (2025) · PROTEOMICS · doi:10.1002/pmic.202300471 · Cited by 53
“The embedding is performed using PubMedBert [83], a language model pretrained on PubMed abstracts and full-text articles proteins are projected into the same low-dimensional space using a bilinear layer.”
RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
Yuan, Z. P., Jin, Q., Tan, C., Zhao, Z., Yuan, H., Huang, F., & Huang, S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611830 · Cited by 85
“In the biomedical NLP community, previous research has shown that further continue training in the in-domain corpus can enhance the performance of pretrained language models [1, 14, 23, 44, 45].”
BackgroundMethodKovačević, A., Bašaragin, B., Milošević, N., & Nenadić, G. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102845 · Cited by 73
Carrino, C. P., Armengol-Estapé, J., Gutiérrez-Fandiño, A., Palao, J. L., Pàmies, M., González-Agirre, A., & Villegas, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03570 · Cited by 77
Almanac: Retrieval-Augmented Language Models for Clinical MedicinePreprint
Zakka, C., Chaurasia, A., Shad, R., Dalal, A. R., Kim, J. L., Moor, M., Alexander, K. M., Ashley, E. A., Boyd, J. H., Boyd, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.01229 · Cited by 71
Gupta, S., Basu, A., Nievas, M., Thomas, J., Wolfrath, N., Ramamurthi, A., Taylor, B., Kothari, A. N., Schwind, R., Miller, T. M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01274-7 · Cited by 93
Conversational health agents: a personalized large language model-powered agent framework
Abbasian, M., Azimi, I., Rahmani, A. M., & Jain, R. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf067 · Cited by 52
“12 This category emerged in response to research indicating that general-domain LLMs often struggle with healthcare-specific tasks due to domain shift, 25,26 and relying solely on prompt engineering may not significantly improve their healthcare-specific performance.”
The Diminishing Returns of Masked Language Models to Science
Hong, Z., Ajith, A., Pauloski, J. G., Duede, E., Chard, K., & Foster, I. T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.82 · Cited by 72
“We consider 14 BERT models: seven from existing literature (BERT-Base, BERT-Large, SciBERT, PubMedBERT, BioBERT v1.”
BackgroundMethodInfluentialAdvances in materials informatics: a review
Sivan, D., Kumar, K. S., Abdullah, A., Raj, V., Misnon, I. I., Ramakrishna, S., & Jose, R. (2024) · Journal of Materials Science · doi:10.1007/s10853-024-09379-w · Cited by 58
CPMI-ChatGLM: parameter-efficient fine-tuning ChatGLM with Chinese patent medicine instructions
Liu, C., Sun, K., Zhou, Q., Duan, Y., Shu, J., Kan, H., Gu, Z., & Hu, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-56874-w · Cited by 65
“Currently, large-scale models in the medical field primarily focus on extracting symptoms and clinical entities from the input to ensure the generation of healthcare recommendations 35–37 .”
BackgroundChatGPT: Transforming Healthcare with AI
Neha, F., Bhati, D., Shukla, D. K., & Amiruzzaman, M. (2024) · AI · doi:10.3390/ai5040126 · Cited by 59
ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
Huang, J., Wang, H., Sun, Y., Shi, Y., Huang, Z., Zhuo, A., & Feng, S. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539021 · Cited by 77
“Existing domain-specific PTMs mainly lie in the domain of healthcare [1, 14], biomedical [8, 10, 16], and academic & research [2].”
BackgroundBioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 71
A survey on membership inference attacks and defenses in machine learning
Niu, J., Liu, P., Zhu, X., Shen, K., Wang, Y., Chi, H., Shen, Y., Jiang, X., Ma, J., & Zhang, Y. (2024) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2024.02.001 · Cited by 81
Quo Vadis ChatGPT? From large language models to Large Knowledge Models
Venkatasubramanian, V., & Chakraborty, A. (2024) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2024.108895 · Cited by 54
“With the rapid increase in the number of applications of LLMs in the scientific and engineering domains, there has been a rise in scientific LLMs such as PubMedBERT[106], MolGPT[107], SciBERT[108] – to name a few.”
BackgroundA multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 62
Chakraborty, C., Bhattacharya, M., Pal, S., Chatterjee, S., Das, A., & Lee, S. (2025) · Journal of Advanced Research · doi:10.1016/j.jare.2025.02.011 · Cited by 57
ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure UnderstandingPreprint
Xiao, Y., Sun, E., Jin, Y., Wang, Q., & Wang, W. W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.11363 · Cited by 70
“Stage 1 requires approximately one week, while stage 2 demands around 60 hours.”
Comparison of biomedical relationship extraction methods and models for knowledge graph creation
Milošević, N., & Thielemann, W. (2022) · Journal of Web Semantics · doi:10.1016/j.websem.2022.100756 · Cited by 55
The architecture of language: Understanding the mechanics behind LLMs
Ferraris, A. F., Audrito, D., Di, L., & Poncibò, C. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2024.16 · Cited by 66
Integrating heterogeneous knowledge graphs into drug–drug interaction extraction from the literature
Asada, M., Miwa, M., & Sasaki, Y. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac754 · Cited by 57
Evaluating the robustness and readiness of large frontier models in health AI applications
池谷, 裕., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04501-8 · Cited by 58
“As shown in ‘Results’, our stress tests 1–5 revealed distinct vulnerabilities: models often succeed for the wrong reasons, retaining high accuracy even when visual input is removed (stress test 1 and stress test 2; see examples in Extended Data Figs.”
Self-citationZhang, T., Cai, Z., Wang, C., Qiu, M., Yang, B., & He, X. (2021) · doi:10.18653/v1/2021.acl-long.457 · Cited by 61
“…2020), BlueBERT (Peng et al., 2019), SCIBERT ( Beltagy et al., 2019) and ClinicalBert (Huang et al., 2019) apply continual learning on (2) PubMedBERT (Gu et al., 2020) learns weights from scratch using PubMed data to obtain an in-domain vocabulary, alleviating the out-of-vocabulary (OOV) problem.”
Efficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 77
Mekkes, N., Groot, M., Hoekstra, E., de Boer, A., Dagkesamanskaia, E., Bouwman, S., Wehrens, S. M. T., Herbert, M. K., Wever, D. D., Rozemüller, A. J. M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02843-9 · Cited by 52
“36) model from HuggingFace: PubMedBERT 37 , pretrained on PubMed abstracts, and Bio_ClinicalBERT 38 , pretrained on electronic health records.”
MethodLEAP: LLM instruction-example adaptive prompting framework for biomedical relation extraction
Zhou, H., Li, M., Xiao, Y., Yang, H., & Zhang, R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae147 · Cited by 53
Large language models to process, analyze, and synthesize biomedical texts: a scoping review
Doneva, S. E., Qin, S., Sick, B., Ellendorff, T., Goldman, J.-P., Schneider, G., & Ineichen, B. V. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00197-2 · Cited by 49
“To improve its performance in BioNLP tasks, the models can be trained on biomedical text corpus in two ways [102]: 1.”
Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
Su, P., Peng, Y., & Vijay‐Shanker, K. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.1 · Cited by 53
Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data
Jiang, H., Zhang, D., Cao, T., Yin, B., & Zhao, T. (2021) · doi:10.18653/v1/2021.acl-long.140 · Cited by 49
A Search Engine for Discovery of Scientific Challenges and Directions
Lahav, D., Saad-Falcon, J., Kuehl, B., Johnson, S., Parasa, S., Shomron, N., Chau, D. H., Yang, D., Horvitz, E., Weld, D. S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21456 · Cited by 55
Scientific Fact-Checking: A Survey of Resources and Approaches
Vladika, J., & Matthes, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.387 · Cited by 96
SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data
Jullien, M., Valentino, M., Frost, H., O’Regan, P., Landers, D., & Freitas, A. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.307 · Cited by 91
Ma, C., Zhou, Z., Liu, H., & Koslicki, D. (2022) · GigaScience · doi:10.1093/gigascience/giad057 · Cited by 51
“To capture the node attributes information, we utilize the PubMedBERT model [Gu et al., 2022], a pre-trained language model designed for biomedical texts, to generate a node attribute embedding for each node based on the concatenation of the node’s name and category.”
An evaluation of GPT models for phenotype concept recognition
Groza, T., Caufield, H., Gration, D., Baynam, G. S., Haendel, M. A., Robinson, P. N., Mungall, C., & Reese, J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02439-w · Cited by 55
“A user can “prompt” ChatGPT to perform a variety of tasks, with or without the need to provide examples to support them.”
BackgroundTen quick tips for deep learning in biology
Lee, B. D., Gitter, A., Greene, C. S., Raschka, S., Maguire, F., Titus, A., Kessler, M. D., Lee, A. J., Chevrette, M. G., Stewart, P. A., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009803 · Cited by 50
Accurate Clinical and Biomedical Named Entity Recognition at Scale
Kocaman, V., & Talby, D. (2022) · Software Impacts · doi:10.1016/j.simpa.2022.100373 · Cited by 66
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4Preprint
Kalyan, K. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12321 · Cited by 55
CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts
Yu, P., Chen, J., Feng, X., & Xia, Z. (2023) · arXiv · doi:10.48550/arxiv.2304.12008 · Cited by 45
GNorm2: an improved gene name recognition and normalization system
Wei, C.-H., Luo, L., Islamaj, R., Lai, P., & Lu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad599 · Cited by 44
Lu, Y., Huang, W., Li, Y., Xu, Y., Wei, Q., Sha, C., & Guo, P. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01159-2 · Cited by 54
Chen, Q., Du, J., Allot, A., & Lu, Z. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3173562 · Cited by 49
“, the HoC dataset [30] is the only multi-label text dataset among commonly-used benchmark datasets for biomedical text mining [16], [17] and it only has about 1500 PubMed documents; and (3)”
BackgroundMethodInfluentialChen, Q., Allot, A., Leaman, R., Islamaj, R., Du, J., Li, F., Wang, K., Xu, S., Zhang, Y., Bagherzadeh, P., et al. (2022) · Database · doi:10.1093/database/baac069 · Cited by 53
ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 90
“Besides, BERT has been validated in various domains (Lee et al., 2019; Beltagy et al., 2019; Peng et al., 2019; Chalkidis et al., 2020; Gu et al., 2021) shown in Table 1.”
BackgroundMethodInfluentialKedia, N., Sanjeev, S., Ong, J., & Chhablani, J. K. (2024) · Eye · doi:10.1038/s41433-023-02915-z · Cited by 49
“Other biomedical language models include DRAGON [22], BioLinkBERT [23], PubMedBERT [24], BioMegatron [25], and BioGPT [26].”
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yao, Y., Huang, S., Wang, W., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.40 · Cited by 66
Mahbub, M., Srinivasan, S., Danciu, I., Peluso, A., Begoli, E., Tamang, S. R., & Peterson, G. D. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262182 · Cited by 57
“As such, PubMedBERT is not vulnerable to possible data leakage. ii) PubMedBERT has achieved state-of-the-art performance in several biomedical NLP tasks on Biomedical Language Understanding and Reasoning Benchmark [21], in comparison to BioBERT, SciBERT, CLinicalBERT.”
MethodResultInfluentialLeveraging pre-trained language models for mining microbiome-disease relationships
Karkera, N., Acharya, S., & Palaniappan, S. K. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05411-z · Cited by 44
“Figure 5 shows the mechanism in which the evidence-question are taken in pairs for the fine-tuning purpose for the discriminative class of models namely BERT [30], BioMega-tron [33], PubMedBERT [32], BioClinicalBERT [28] and BioLinkBERT [34].”
BackgroundMethodExploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 58
Self-citationSevgi, M., Antaki, F., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325046 · Cited by 47
Pashangpour, S., & Nejat, G. (2024) · Robotics · doi:10.3390/robotics13080112 · Cited by 49
“The potential applications of the KnowledgeBot framework in healthcare could involve using a model with domain-specific knowledge such as MED-PaLM 2 [41] acting as the object generator, where the LLM interprets patient conditions or staff instructions to generate lists of relevant objects and…”
MethodWang, H., Zhao, S., Qiang, Z., Li, Z., Liu, C., Xi, N., Du, Y., Qin, B., & Liu, T. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3686807 · Cited by 65
“…Yan, and Lu 2019; Lee et al. 2020; Huang, Altosaar, and Ranganath 2019; Luo et al. 2022), learn domain-specific vocabulary for better representation (Lewis et al. 2020; Gu et al. 2021), or provide the LLMs with medical knowledge (Zhang et al. 2021; Michalopoulos et al. 2021; Wang et al. 2022a).”
Delbrouck, J.-B., Chambon, P., Chen, Z. H., Varma, M., Johnston, A., Blankemeier, L., Veen, D. V., Bui, T., Truong, S. Q. H., & Langlotz, C. P. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.765 · Cited by 89
“To achieve this, we evaluate two transformer-based libraries under MIT License for Entity and Relation Extraction: DyGIE++ (Wadden et al., 2019) and SpERT (Eberts and Ulges, 2020) with three proven transformer architectures, namely BERT (Kenton and Toutanova, 2019), BiomedBERT (Gu et al., 2020) and BiomedVLP-CXR-BERT…”
MethodGenerative Large Language Models Trained for Detecting Errors in Radiology Reports
Sun, C. H., Teichman, K., Zhou, Y., Critelli, B., Nauheim, D. O., Keir, G., Wang, X., Zhong, J., Flanders, A. E., Shih, G., et al. (2025) · Radiology · doi:10.1148/radiol.242575 · Cited by 53
Yan, S., Hu, M., Jiang, Y., Li, X., Fei, H., Tschandl, P., Kittler, H., & Ge, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01178 · Cited by 63
Zhang, X., Talukdar, N., Vemulapalli, S., Ahn, S., Wang, J., Han, M., Murtaza, S. M. B., Leshchiner, D. R., Dave, A. A., Joseph, D. F., et al. (2024) · medRxiv · doi:10.1101/2024.02.07.24302444 · Cited by 76
“The PubMedBERT model was implemented utilizing the MetBERT code repository from GitHub 11 .”
MethodInfluentialRadGraph: Extracting Clinical Entities and Relations from Radiology ReportsPreprint
Jain, S., Agrawal, A., Saporta, A., Truong, S. Q., Nguyen Duong, D., Bui, T., Chambon, P., Lungren, M. P., Ng, A. Y., Langlotz, C. P., et al. (2021) · arXiv (Cornell University) · doi:10.13026/2f5n-pb92 · Cited by 226
Wu, J., Liu, X., Li, M., Li, W., Su, Z., Lin, S., Garay, L., Zhang, Z., Zhang, Y., Zeng, Q., et al. (2024) · NEJM AI · doi:10.1056/aira2400012 · Cited by 41
Self-citationWu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41
“26 Given that approximately 90% of data for training LLMs is in English 27 and non-English medical corpora are even more scarce, 28 this language imbalance of training corpora poses challenges for the extensions of LLMs in non-English clinical scenarios.”
BackgroundTan, R. S. Y. C., Lin, Q., Low, G. H., Lin, R., Goh, T. C., Chang, C. C. E., Lee, F. F., Chan, W. Y., Tan, W. C., Tey, H. J., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad133 · Cited by 41
A question-answering framework for automated abstract screening using large language models
Akinseloyin, O., Jiang, X., & Palade, V. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae166 · Cited by 48
“71”
Labels in a haystack: Approaches beyond supervised learning in biomedical applications
Yakimovich, A., Beaugnon, A., Huang, Y., & Özkırımlı, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100383 · Cited by 50
Shen, Z., Schutte, D., Yi, Y., Bompelli, A., Yu, F., Wang, Y., & Zhang, R. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01819-4 · Cited by 45
“In general, these domain-specific BERT models have shown promising performance in clinical applications [18–20].”
BackgroundMethodLeveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness
Zhang, G., Jin, Q., Jered McInerney, D., Chen, Y., Wang, F., Cole, C. L., Yang, Q., Wang, Y., Malin, B. A., Peleg, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104640 · Cited by 40
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.