Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 401–500 of 1,030
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.896 · Cited by 74
BioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature
Frisoni, G., Mizutani, M., Moro, G., & Valgimigli, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.390 · Cited by 21
“The first category includes BIOBERT (Lee et al., 2020), SCIBERT (Beltagy et al., 2019), BLUEBERT (Peng et al., 2019), CLINICALBERT (Alsentzer et al., 2019), PUBMEDBERT (Gu et al., 2022), PUBMEDELECTRA (Tinn et al., 2021), BIOLINKBERT (Yasunaga et al., 2022), and BIOMEGATRON (Shin et al., 2020).”
BackgroundMethodInfluentialTürkmen, H., Dikenelli, O., Eraslan, C., Çallı, M. C., & Özbek, S. S. (2023) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00140-7 · Cited by 34
“PubMedBERT [5] evaluated this approach by pretraining a BERT model and creating the vocabulary from scratch using”
MethodEnsemble pretrained language models to extract biomedical knowledge from literature
Li, Z., Wei, Q., Huang, L., Li, J., Hu, Y., Chuang, Y.-S., He, J., Das, A., Keloth, V. K., Yang, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae061 · Cited by 29
“…challenge as follows: 1) We leveraged ensemble learning with the predictions from three pretrained domain-specific models, BioBERT, 18 Pub-MedBERT, 19 and BioM-ELECTRA, 20 built a rule-based detection method for cell line and taxonomy names and engaged domain experts to manually annotated 70 more…”
MethodInfluential(2023) · doi:10.1038/s42256-023-00764-9 · Cited by 28
“Pretraining frameworks 4,5 facilitate the seamless integration of new tasks, thereby expediting the modelling process, especially for scenarios with limited labelled data.”
Hsu, E., & Roberts, K. E. (2025) · Scientific Reports · doi:10.1038/s41598-024-68168-2 · Cited by 44
“One explanation is that since Llama2 is a general-domain model while PubMedBERT is a biomedical model, the former might have advantages in solving non-medical problems such as PHI identification while the latter has advantages in solving medical problems like medication terms.”
InfluentialDALL-M: Context-aware clinical data augmentation with large language models
Hsieh, C., Moreira, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Jorge, J. A., & Nascimento, J. C. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110022 · Cited by 32
Rahujo, A., Atif, D., Inam, S. A., Khan, A. A., & Ullah, S. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00339-0 · Cited by 28
Tong, Y., Chen, Y., & Shi, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.424 · Cited by 37
A review on Natural Language Processing Models for COVID-19 research
Hall, K., Chang, V. I. C., & Jayne, C. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100078 · Cited by 33
“BLURB (Biomedical Language Understanding & Reasoning Benchark) [29] is the most comprehensive benchmark for assessing the NLP apabilities of Biomedical NLP models.”
Natural language processing (NLP) aided qualitative method in health research
Cheligeer, C., Yang, L., Nandi, T., Doktorchik, C. T. A., Quan, H., Zeng, Y., & Singh, S. (2022) · Journal of Integrated Design and Process Science · doi:10.3233/jid-220013 · Cited by 30
“…BioBERT, UmlsBERT, PubMedBERT, and Clinical Text BERT are specifically pre-trained with clinical notes, research publications from the medical field, and medical thesaurus to support medical term-intensive tasks (Alsentzer et al., 2019; Gu et al., 2022; Lee, 2020; Michalopoulos et al., 2021).”
Predicting unplanned readmissions in the intensive care unit: a multimodality evaluation
Sheetrit, E., Brief, M., & Elisha, O. (2023) · Scientific Reports · doi:10.1038/s41598-023-42372-y · Cited by 32
“text has shown that LLMs can achieve even better performance on downstream clinical tasks [10].”
BackgroundMethodRobust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network
Lovelace, J., Newman-Griffis, D. R., Vashishth, S., Lehman, J. F., & Rosé, C. P. (2021) · doi:10.18653/v1/2021.acl-long.82 · Cited by 43
“For SNOMED CT Core KG, we utilize PubMedBERT (Gu et al., 2020) which is better suited for the biomedical terminology in the UMLS.”
MethodKim, B., & Ganapathi, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10650 · Cited by 31
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Gupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27
“, 2019), and MedBERT(Gu et al., 2021) the classification outcome from the BERT model to drive the direction of further questioning with the patient in a more controlled manner.”
Leveraging foundation and large language models in medical artificial intelligence
Wong, I. N., Monteiro, O., Baptista‐Hon, D. T., Wang, K., Lu, W.-Y., Sun, Z., Nie, S., & Yin, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003302 · Cited by 31
Timilsina, M., Buosi, S., Razzaq, M. A., Haque, R., Judge, C. S., & Curry, E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.109925 · Cited by 32
Zhu, H., Peng, H. M., Lyu, Z., Hou, L., Li, J., & Xiao, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119369 · Cited by 41
“For the biomedical domain, there are BioBERT (Lee, Yoon, Kim, Kim, Kim, So and Kang (2020)), extBERT (Tai, Kung, Dong, Comiter and Kuo (2020)), PubMedBERT (Gu, Tinn, Cheng, Lucas, Usuyama, Liu, Naumann, Gao and Poon (2020)), ClinicalBERT (Huang, Altosaar and Ranganath (2019)), MT-ClinicalBERT (Mulyar and McInnes (2020…”
BackgroundChari, S., Acharya, P., Gruen, D. M., Zhang, O., Eyigöz, E., Ghalwash, M. F., Seneviratne, O. W., Suárez, F., Meyer, P., Chakraborty, P., et al. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102498 · Cited by 35
“mains by the pre-training of these models on large biomedical corpora [38, 56].”
BackgroundExtensive evaluation of transformer-based architectures for adverse drug events extraction
Scaboro, S., Portelli, B., Chersoni, E., Santus, E., & Serra, G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110675 · Cited by 30
Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?
Xu, J., Ma, M. D., & Chen, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.138 · Cited by 32
Conversational agents enhance women's contribution in online debates
Hadfi, R., Okuhara, S., Haqbeen, J. A., Sahab, S., Ohnuma, S., & Itō, T. (2023) · Scientific Reports · doi:10.1038/s41598-023-41703-3 · Cited by 27
“With the recent prowess of Machine Learning (ML), intelligent agents are built with advanced techniques, such as Deep Learning and Large Language Models (LLMs) 19,35,36 .”
MethodMiranda-Escalada, A., Mehryary, F., Luoma, J., Estrada, D., Gascó, L., Pyysalo, S., Valencia, A., & Krallinger, M. (2023) · Database · doi:10.1093/database/baad080 · Cited by 36
“Again, biomedical pretrained LMs are used for both frameworks, including, but not only, PubMedBERT.”
BackgroundInfluentialDeveloping ChatGPT for biology and medicine: a complete review of biomedical question answering
Li, Q., Li, L., & Li, Y. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240004 · Cited by 36
“For example, BioBERT (Lee et al, 2019), PubMedBERT (Gu et al, 2020), and BioLinkBERT (Yasunaga et al, 2022), have made notable contributions in the medical domain, demonstrating the potential of BERT-based models in enhancing the performance of medical question answering systems.”
BackgroundInfluentialProtein Representation Learning via Knowledge Enhanced Primary Structure ModelingPreprint
Zhou, H.-Y., Fu, Y., Zhang, Z., Bian, C., & Yu, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.26.525795 · Cited by 38
“We initialize the language encoder using PubMedBERT (Gu et al., 2021) and do not update it in the training phase.”
MethodA Pretrained Language Model for Cyber Threat Intelligence
Park, Y. H., & You, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.12 · Cited by 45
Shi, Y., He, M., Chen, J., Han, F., & Cai, Y. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011989 · Cited by 26
Anderson, L. N., Hoyt, C. T., Zucker, J., McNaughton, A. D., Teuton, J., Karis, K., Arokium-Christian, N. N., Warley, J. T., Stromberg, Z. R., Gyori, B. M., et al. (2025) · Frontiers in Immunology · doi:10.3389/fimmu.2025.1502484 · Cited by 31
Utilizing large language models for gastroenterology research: a conceptual framework
Berry, P., Dhanakshirur, R. R., & Khanna, S. (2025) · Therapeutic Advances in Gastroenterology · doi:10.1177/17562848251328577 · Cited by 29
Artificial intelligence in bioinformatics: a survey
Jiang, J., Li, Y., Cao, S., Shan, Y., Liu, Y., Fei, T., Yu, Y., Feng, Y., Li, Y., Li, Y., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf576 · Cited by 33
Chen, C. L. P., Li, L., Wang, T., Wang, Z. Z., Leng, J., Shao, H., & Cheng, L. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.02.008 · Cited by 30
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.383 · Cited by 38
Meng, Z., Liu, F., Shareghi, E., Su, Y., Collins, C., & Collier, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.329 · Cited by 39
Ye, M., Cui, S., Wang, Y., Luo, J., Xiao, C., & Ma, F. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482273 · Cited by 44
“To solve this problem, we employ a language model called PubMedBERT [9] pre-trained on biomedical publications in the PubMed database and a two-layer multilayer perceptrons denoted as MLP1 for finetuning in our datasets to learn an embedding for each segment.”
MethodA simple neural vector space model for medical concept normalization using concept embeddings
Xu, D., & Miller, T. A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104080 · Cited by 30
Evidence Extraction to Validate Medical Claims in Fake News Detection
Deka, P., Jurek, A., & Deepak, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20627-6_1 · Cited by 59
Precision information extraction for rare disease epidemiology at scale
Kariampuzha, W. Z., Alyea, G., Qu, S., Sanjak, J. S., Mathé, E. A., Sid, E., Chatelaine, H., Yadaw, A. S., Xu, Y., & Zhu, Q. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04011-y · Cited by 30
“Training this deep learning model for NER requires a dataset labeled at the token level.”
BackgroundMethodInfluentialOAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
Zhang, F., Shi, S., Zhu, Y., Chen, B., Cen, Y., Yu, J., Chen, Y., Wang, L., Zhao, Q., Cheng, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672354 · Cited by 38
“All - - - AceKG [62] Partial All Graph - - S2ORC [38] All NLP Partial BLURB [18] Biomed.”
BackgroundMethodWada, S., Takeda, T., Okada, K., Manabe, S., Konishi, S., Kamohara, J., & Matsumura, Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102889 · Cited by 29
Large Language Models in Genomics—A Perspective on Personalized Medicine
Ali, S. M., Qadri, Y. A., Ahmad, K., Lin, Z., Leung, M.-F., Kim, S. W., Vasilakos, A. V., & Zhou, T. (2025) · Bioengineering · doi:10.3390/bioengineering12050440 · Cited by 30
Foundation Models in Healthcare: Opportunities, Risks & Strategies Forward
Thieme, A., Nori, A., Ghassemi, M., Bommasani, R., Andersen, T. O., & Luger, E. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3583177 · Cited by 25
“In other words, the powerful FMs that evolve, for example Google’s language model BERT [13], then come to serve as the basis for new state-of-the art models as adaptations from that FM (e.g., CXR-BERT [3], Med-BERT [34] or PubMedBERT [18] in healthcare).”
Tang, X., Tran, A., Tan, J. T. C., & Gerstein, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae260 · Cited by 43
“…led to promising outcomes in the biomedical field, particularly in tasks such as molecular property prediction and chemical structure generation, indicating strong potential for artificial intelligence in biomolecular studies (Gu et al. 2021, Xia et al. 2022, Luo et al. 2022a, Singhal et al. 2023).”
Hyperdimensional computing: A fast, robust, and interpretable paradigm for biological data
Stock, M., Criekinge, W. V., Boeckaerts, D., Taelman, S., Haeverbeke, M. V., Dewulf, P., & Baets, B. D. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012426 · Cited by 28
“Large models, such as BioBERT [122], allow for the processing of large amounts of biomedical data, for example, for drug discovery or personalized medicine [123].”
MethodRummaGEO: Automatic mining of human and mouse gene sets from GEO
Marino, G. B., Clarke, D. J. B., Lachmann, A., Deng, E. Z., & Ma’ayan, A. (2024) · Patterns · doi:10.1016/j.patter.2024.101072 · Cited by 27
Natural Language Processing for Digital Health in the Era of Large Language Models
Sarker, A., Zhang, R., Wang, Y., Xiao, Y., Das, S., Schutte, D., Oniani, D., Xie, Q., & Xu, H. (2024) · Yearbook of Medical Informatics · doi:10.1055/s-0044-1800750 · Cited by 39
Zhao, X., Huang, H., Yang, T., Lu, Y., Zhang, L., Wang, R., Liu, Z., Zhong, T., & Liu, T. (2025) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2025.102332 · Cited by 25
Explainable ICD multi-label classification of EHRs in Spanish with convolutional attention
Trigueros, O., Blanco, A., Lebeña, N., Casillas, A., & Pérez, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104615 · Cited by 30
Zhu, Y., Mahale, A., Peters, K., Mathew, L., Giuste, F., Anderson, B. J., & Wang, M. D. (2022) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3535508.3545555 · Cited by 25
Wang, S., Dang, Y., Sun, Z., Ding, Y., Pathak, J., Tao, C., Xiao, Y., & Peng, Y. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad068 · Cited by 29
Classifying European Court of Human Rights Cases Using Transformer-Based Techniques
Imran, A. S., Hodnefjeld, H., Kastrati, Z., Fatima, N., Daudpota, S. M., & Wani, M. A. (2023) · IEEE Access · doi:10.1109/access.2023.3279034 · Cited by 25
“Y Gu. et al. in [18] challenges the assumption that pretraining general-domain language models continuously leads to better performance in certain domains, such as biomedicine.”
BackgroundIntegrating domain knowledge for biomedical text analysis into deep learning: A survey
Cai, L., Li, J., Lv, H., Liu, W., Niu, H., & Wang, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104418 · Cited by 27
Knowledge-Enhanced Graph Topic Transformer for Explainable Biomedical Text Summarization
Xie, Q., Tiwari, P., & Ananiadou, S. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3308064 · Cited by 33
Wang, G., Lou, X., Guo, F., Kwok, D., & Cao, C. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3346210 · Cited by 25
Shaitarova, A., Zaghir, J., Lavelli, A., Krauthammer, M. O., & Rinaldi, F. (2023) · Yearbook of Medical Informatics · doi:10.1055/s-0043-1768726 · Cited by 29
“Overall, in-domain pretraining is arguably the best option, given the availability of training data and computational resources [45–47].”
BackgroundXiao, Y., Hou, Y., Zhou, H., Diallo, G., Marcelo Fiszman, B., Wolfson, J. A., Zhou, L., Halil Kilicoglu, B., Chen, Y., Su, C., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-58604-8 · Cited by 30
Large Language Model and Text Generation
Wu, Y. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_10 · Cited by 34
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 31
Visual–language foundation models in medicine
Liu, C., Jin, Y., Guan, Z., Li, T., Qin, Y., Qian, B., Jiang, Z., Wu, Y.-L., Wang, X., Zheng, Y. F., et al. (2024) · The Visual Computer · doi:10.1007/s00371-024-03579-w · Cited by 44
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical StudyPreprint
Nadkarni, R., Wadden, D., Beltagy, I., Smith, N. A., Hajishirzi, H., & Hope, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09700 · Cited by 31
AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach
Amirizaniani, M., Martin, E., Roosta, T. G., Chadha, A. R., & Shah, C. S. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679222 · Cited by 33
“For example, LLMs have become increasingly integrated into critical sectors, such as health-care [11, 34, 39], education [7, 16, 19], legal [18] and financial services [13, 41].”
BackgroundA survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 27
Younesi, A., Oustad, E., Ansari, M., Fahringer, T., & Buyya, R. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101815 · Cited by 34
Sequence tagging for biomedical extractive question answering
Yoon, W., Jackson, R., Lagerberg, A., & Kang, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac397 · Cited by 29
Wang, Y., Liu, X., Hu, W., & Zhang, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.276 · Cited by 24
Self-citationApplication of artificial intelligence and machine learning in drug repurposing
Ghandikota, S. K., & Jegga, A. G. (2024) · Progress in molecular biology and translational science · doi:10.1016/bs.pmbts.2024.03.030 · Cited by 28
Qiao, G., Wang, G., & Li, Y. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae570 · Cited by 27
Han, Q., Tian, S., & Zhang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02998 · Cited by 24
“In biology and bioinformatics area, well-known pre-trained models include BioBERT (9), PubMedBERT (abstract only), and PubMedBERT (full text) (10).”
MethodAryffin, H. A. K., Sahbudin, M. A. B., Pitchay, S. A., Abhalim, A. H., & Sahbudin, I. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2874 · Cited by 29
Ponthongmak, W., Thammasudjarit, R., McKay, G. J., Attia, J. R., Theera-Ampornpunt, N., & Thakkinstian, A. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101227 · Cited by 29
Remy, F., Demuynck, K., & Demeester, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.104 · Cited by 46
Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
Yun, H., Marshall, I. J., Trikalinos, T. A., & Wallace, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.626 · Cited by 40
Domain-Specific Language Model Post-Training for Indonesian Financial NLP
Maharani, N. P. I., Purwarianti, A., Yustiawan, Y., & Rochim, F. C. (2023) · doi:10.1109/iceei59426.2023.10346625 · Cited by 26
“Previous studies have explored on specific domains such as biomedical [11]–[13], scientific domain [14], legal domain [15], and financial [3]–[5].”
BackgroundPotential merits and flaws of large language models in epilepsy care: A critical review
Diessen, E. G. A. L. V., van Amerongen, R. A., Zijlmans, M., & Otte, W. M. (2024) · Epilepsia · doi:10.1111/epi.17907 · Cited by 26
“20 The latter strategy is applied increasingly as it avoids the laborious work of supervised data labeling.”
BackgroundSymptom-BERT: Enhancing Cancer Symptom Detection in EHR Clinical Notes
Zeinali, N., Albashayreh, A., Fan, W., & White, S. G. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.05.015 · Cited by 27
Online Mendelian Inheritance in Animals (OMIA): a genetic resource for vertebrate animals
Tammen, I., Mather, M., Leeb, T., & Nicholas, F. W. (2024) · Mammalian Genome · doi:10.1007/s00335-024-10059-y · Cited by 28
“By far the most successful turned out to be a machine-learning model developed using Microsoft’s PubMedBERT (Gu et al. 2021 ), which was adapted for OMIA in February 2023, and has been in daily use ever since.”
MethodGuthrie, E., Levy, D., & del Carmen, G. (2024) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2024.02.016 · Cited by 22
Aspect-Based Sentiment Analysis of Patient Feedback Using Large Language Models
Alkhnbashi, O. S., Nassr, R. M., & Hammoudeh, M. A. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8120167 · Cited by 25
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 27
Self-citationLin, M., Holste, G., Wang, S., Zhou, Y., Wei, Y., Banerjee, I., Chen, P., Dai, T., Du, Y., Dvornek, N. C., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103739 · Cited by 30
Description-based Label Attention Classifier for Explainable ICD-9 Classification
Feucht, M., Wu, Z., Althammer, S., & Volker Tresp, C. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.8 · Cited by 28
Discovering novel drug-supplement interactions using SuppKG generated from the biomedical literature
Schutte, D., Vasilakes, J., Bompelli, A., Zhou, Y., Fiszman, M., Xu, H., Kilicoglu, H., Bishop, J. R., Adam, T. J., & Zhang, R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104120 · Cited by 23
“This included the base uncased BERT to use as a baseline and five biomedical domain-specific BERT models that had attained SotA results on various biomedical NLP tasks: PubMedBERT abstracts only, PubMedBERT abstracts and full text [32], BioBERT [33], BlueBERT [34], and BioClinicalBERT [35].”
MethodBertSRC: transformer-based semantic relation classification
Lee, Y., Son, J., & Song, M. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01977-5 · Cited by 25
“Since then, its variant models, which were pre-trained on domain-specific data, such as BioBERT [31], SciBERT [32], and PubMedBERT [33] as well as advanced versions of BERT with tweaked pre-training methods or a structure of layers, such as ALBERT [34] and RoBERTa [35], have been announced.”
MethodA role distinguishing Bert model for medical dialogue system in sustainable smart city
Wang, S., Wang, S., Liu, Z., & Zhang, Q. (2022) · Sustainable Energy Technologies and Assessments · doi:10.1016/j.seta.2022.102896 · Cited by 22
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 38
“There is a wealth of evidence to demonstrate the beneficial impact of in-domain pre-training on clinical (Alsentzer et al., 2019b; Lehman et al., 2023) and biomedical (Gu et al., 2021) Table 5: The impact of domain adaptation of metrics on correlation to human assessments.”
BackgroundLokker, C., Bagheri, E., Abdelkader, W., Parrish, R., Afzal, M., Navarro, T., Cotoi, C., Germini, F., Linkins, L. A., Haynes, R. B., et al. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104384 · Cited by 26
Multilingual Clinical NER: Translation or Cross-lingual Transfer?
Gaschi, F., Fontaine, X., Rastin, P., & Toussaint, Y. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.34 · Cited by 22
Can ChatGPT explain it? Use of artificial intelligence in multiple sclerosis communication
Inojosa, H., Gilbert, S., Kather, J. N., Proschmann, U., Akgün, K., & Ziemssen, T. (2023) · Neurological Research and Practice · doi:10.1186/s42466-023-00270-8 · Cited by 22
“Other LLMs have been developed based on biomedical literature or clinical notes, although using a smaller data-base such as e.g. BioBERT [12], BLURB [13] or BioMega-tron [14] but these are not as popular and accessible as ChatGPT.”
MethodBergman, E., Dürlich, L., Arthurson, V., Sundström, A., Larsson, M., Bhuiyan, S., Jakobsson, A., & Westman, G. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000409 · Cited by 28
Dong, G., Bate, A., Haguinet, F., Westman, G., Dürlich, L., Hviid, A. P., & Sessa, M. (2023) · Drug Safety · doi:10.1007/s40264-023-01381-6 · Cited by 23
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 26
Path-BigBird: An AI-Driven Transformer Approach to Classification of Cancer Pathology Reports
Chandrashekar, M., Lyngaas, I., Hanson, H. A., Gao, S. B., Wu, X., & Gounley, J. P. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci.23.00148 · Cited by 28
Iscoe, M. S., Socrates, V., Gilson, A., Chi, L., Li, H., Huang, T. Y., Kearns, T., Perkins, R., Khandjian, L., & Taylor, R. A. (2024) · Academic Emergency Medicine · doi:10.1111/acem.14883 · Cited by 23
Extracting patient lifestyle characteristics from Dutch clinical text with BERT models
Muizelaar, H., Haas, M., van Dortmont, K., van der Putten, P., & Spruit, M. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02557-5 · Cited by 19
“Within the existing literature in the clinical domain, domain-specific models are shown to outperform fine-tuned general BERT models virtually every time a direct comparison takes place, such as for the aforementioned ClinicalBERT [11, 16, 18, 20–22] and BioBERT [11, 19, 21, 23, 24].”
MethodCamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data
Touchent, R., & de la Clergerie, É. V. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2q9tffuv6zxt · Cited by 35
Clustering-based topic modeling for biomedical documents extractive text summarization
AbdelAziz, N. M., Ali, A. A., Naguib, S. M., & Fayed, L. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06640-6 · Cited by 24
“Overall, there are many PLMs pre-trained specifically for biomedical texts, such as BioBERT [43], PubMed BERT [44], SciBERT [45], BlueBERT [46], ClinicalBERT [47], and ALBERT [48].”
The Classification of Short Scientific Texts Using Pretrained BERT Model
Danilov, G., Ishankulov, T., Kotik, K., Orlov, Y. N., Shifrin, M., & Aa, P. (2021) · Studies in health technology and informatics · doi:10.3233/shti210125 · Cited by 22
NILINKER: Attention-based approach to NIL Entity Linking
Ruas, P., & Couto, F. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104137 · Cited by 33
Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2022) · Australasian Document Computing Symposium · doi:10.1145/3572960.3572980 · Cited by 31
“In the CLEF TAR dataset, documents are represented by their PubMed PMID (i.e., a document identifier).”
BackgroundMethodInfluentialBuilding Chinese Biomedical Language Models via Multi-Level Text DiscriminationPreprint
Wang, Q., Dai, S., Xu, B., Lyu, Y., Zhu, Y. T., Wu, H. R., & Wang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07244 · Cited by 22
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.