Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 701–800 of 1,030
Large language models for clinical artificial intelligence in healthcare a systematic review
Ghnemat, R., & Saleh, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00784-x · Cited by 17
“Pre-training allows the model to develop significant natural language understanding, which raises interest in researching encoder-only LMs on medical applications [71, 72].”
Denoising Graph Inference Network for Document-Level Relation Extraction
Wang, H., Qin, K., Duan, G., & Luo, G. (2023) · Big Data Mining and Analytics · doi:10.26599/bdma.2022.9020051 · Cited by 13
Retrieval augmented scientific claim verification
Liu, H., Soroush, A., Nestor, J. G., Park, E., Idnay, B., Fang, Y., Pan, J. Y., Liao, S., Bernard, M., Peng, Y., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae021 · Cited by 29
Question-answering system extracts information on injection drug use from clinical notes
Mahbub, M., Goethert, I., Danciu, I., Knight, K., Srinivasan, S., Tamang, S. R., Rozenberg‐Ben‐Dror, K., Solares, H., Martins, S. B., Trafton, J. A., et al. (2024) · Communications Medicine · doi:10.1038/s43856-024-00470-6 · Cited by 15
“After reviewing the hyperparameters utilized in various QA tasks as outlined in [62, 63, 64, 52, 65, 66, 67, 68, 53, 25], we set the document stride to 128 and opted for a batch size of 32, a learning rate of 3 e − 5 and a training epoch count of 5 for the training configurations.”
MethodEnhancing Relation Extraction from Biomedical Texts by Large Language Models
Asada, M., & Fukuda, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_1 · Cited by 22
ClinLinker: Medical Entity Linking of Clinical Concept Mentions in Spanish
Gallego, F., López-García, G., Gascó, L., Krallinger, M., & Veredas, F. J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63775-9_19 · Cited by 13
Heterogeneous biomedical entity representation learning for gene–disease association prediction
Meng, Z., Liu, S., Liang, S., Jani, B., & Meng, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae380 · Cited by 13
“• PubMedBERT and PubMedBERT-Fulltext [13] : Variants of the BERT model have been trained on an extensive corpus of biomedical literature to capture the nuances of biomedical texts.”
BackgroundMethodInfluentialIntroducing MagBERT: A language model for magnesium textual data mining and analysis
Kumar, S., Jaafreh, R., Singh, N., Hamad, K., & Yoon, D. H. (2024) · Journal of Magnesium and Alloys · doi:10.1016/j.jma.2024.08.010 · Cited by 13
Nentidis, A., Katsimpras, G., Krithara, A., López, S. L., Farré-Maduell, E., Krallinger, M., Loukachevitch, N., Davydova, V., Tutubalina, E. V., & Παλιούρας, Γ. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71908-0_1 · Cited by 13
“For English NNER, they used the pre-trained PubMedBERT for contextualized word embeddings [18]; for Russian NNER, they employed a pre-trained SBERT-Large-NLU-RU model; and for Bilingual NNER, they utilized BERT-Base-Multilingual-uncased [13].”
Augmenting biomedical named entity recognition with general-domain resources
Yin, Y., Kim, H., Xiao, X., Wei, C.-H., Kang, J., Lu, Z., Xu, H., Fang, M., & Chen, Q. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104731 · Cited by 16
“2) AIONER [43], a newly developed BioNER model built upon PubMedBERT [14], employed a flexible tagging scheme that integrates eleven BioNER training datasets into a unified resource, matching or surpassing the performance of previous SOTA methods, 3) PTC [42], a web-based system for recognizing…”
Multi-Modal CLIP-Informed Protein Editing
Yin, M., Zhou, H., Zhu, Y., Lin, M., Wu, Y., Wu, J., Xu, H., Hsieh, C., Hou, T., Chen, J., et al. (2024) · Health Data Science · doi:10.34133/hds.0211 · Cited by 18
Cost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications
Huang, D., & Cole, J. M. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02029 · Cited by 13
“31,32 In order to train an efficient embedding model for the optoelectronics literature, a data set OE-Ttl-Abs-303k of 303k pairs of journal titles and abstracts about optoelectronics was collected via the Scopus API, and then used to fine-tune the language models for text retrieval based on cosine…”
Choi, J., Yoon, J., & Lee, C. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111034 · Cited by 13
Yazdanpanah, O., Chang, M., & Bakhshi, E. A. (2024) · Integrated Computer-Aided Engineering · doi:10.1177/10692509241301240 · Cited by 13
Uncertainty-Aware Contrastive Learning for deep clustering
Chang, L., Chen, L., & Zhou, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130568 · Cited by 13
Chen, H., Alfred, M. C., & Cohen, E. (2025) · JMIR Medical Informatics · doi:10.2196/68955 · Cited by 18
“The inclusion of these models was based on the hypothesis that domain-specific pretraining may enhance the model’s ability to capture contextually relevant medical terminology and nuances, thereby improving its effectiveness in detecting stigmatizing language [58].”
Segment anything in pathology images with natural language
Chen, Z., Hou, J., Lin, L., Wang, Y., Bie, Y., Wang, X., Zhou, Y., Li, D., Tan, H., Liang, L., et al. (2026) · Nature Computational Science · doi:10.1038/s43588-026-01042-5 · Cited by 13
“The text encoder Φ text utilizes a PubMedBERT model 33 to process textual prompts specifying the target object and generate text features of length L : Joint feature interaction module .”
Flöge, K., Udayakumar, S., Sommer, J., Piraud, M., Kesselheim, S., Fortuin, V., Günnemann, S., van der Weg, K. J., Gohlke, H., Merdivan, E., et al. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013679 · Cited by 14
Exploiting ICD Hierarchy for Classification of EHRs in Spanish Through Multi-Task Transformers
Blanco, A., Pérez, A., & Casillas, A. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3112130 · Cited by 16
mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
Jørgensen, R. K., Hartmann, M., Dai, X., & Elliott, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.290 · Cited by 13
BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
Monajatipoor, M., Rouhsedaghat, M., Li, L. H., Chien, A., Kuo, C. ‐. J., Scalzo, F., & Chang, K.-W. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00372 · Cited by 14
“Recently, BlueBERT has been widely used in the bioNLP community for various NLP tasks [15, 13, 38] and a few V&L tasks, e.”
MethodErdengasileng, A., Han, Q., Zhao, T., Tian, S., Sui, X., Li, K., Wang, W., Wang, J., Hu, T., Pan, F., et al. (2022) · Database · doi:10.1093/database/baac066 · Cited by 16
“We tried different per-trained models on different tracks, including Bidirectional Encoder Representations from Transformers (BERT) (27) (Tracks 1, 2, 3 and 5), BioBERT (28) (Track 1, 2, and 3), PubMedBERT (29) abstract only (Track 5), PubMedBERT fulltext [Tracks 1, 2, 3 (abstract only and full text) and 5], Sentence…”
MethodInfluentialK-RET: knowledgeable biomedical relation extraction system
Sousa, D., & Couto, F. M. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad174 · Cited by 13
An Improved Model for Medical Forum Question Classification Based on CNN and BiLSTM
Mutabazi, E., Ni, J., Tang, G., & Cao, W. (2023) · Applied Sciences · doi:10.3390/app13158623 · Cited by 13
Precision Health in the Age of Large Language Models
Poon, H., Naumann, T., Zhang, S., & Hernández, J. G. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599568 · Cited by 16
Self-citationZhou, Y., Jin, Z., Li, M., Li, M., Liu, X., You, X., & Wu, J. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.234 · Cited by 13
Self-citationModelling Temporal Document Sequences for Clinical ICD Coding
Ng, B. L. C., Santos, D., & Rei, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.120 · Cited by 17
Incorporating entity-level knowledge in pretrained language model for biomedical dense retrieval
Tan, J., Hu, J., & Dong, S. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107535 · Cited by 13
Crema, C., Buonocore, T. M., Fostinelli, S., Parimbelli, E., Verde, F., Fundarò, C., Manera, M. R., Ramusino, M. C., Capelli, M., Costa, A. C., et al. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104557 · Cited by 18
Explanatory argument extraction of correct answers in resident medical exams
Goenaga, I., Atutxa, A., Gojenola, K., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102985 · Cited by 17
Abdelkader, W., Navarro, T., Parrish, R., Cotoi, C., Germini, F., Linkins, L., Iorio, A., Haynes, R. B., Ananiadou, S., Chu, L., et al. (2021) · JMIR Research Protocols · doi:10.2196/29398 · Cited by 13
“PubMedBERT is pretrained using domain-specific text data from a collection of 14 million PubMed abstracts, which were downloaded in February 2020, with abstracts under 128 words removed [47].”
MethodContinual knowledge infusion into pre-trained biomedical language models
Jha, K., & Zhang, A. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab671 · Cited by 13
A Dataset for N-ary Relation Extraction of Drug Combinations
Tiktinsky, A., Viswanathan, V., Niezni, D., Azagury, D. M., Shamay, Y., Taub-Tabib, H., Hope, T., & Goldberg, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.233 · Cited by 22
A Unified Framework for Generative Data Augmentation: A Comprehensive SurveyPreprint
Chen, Y., Yan, Z., & Zhu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00277 · Cited by 17
Text2MDT: Extracting Medical Decision Trees from Medical TextsPreprint
Zhu, W., Li, W., Tian, X., Wang, P., Wang, X., Chen, J., Wu, Y., Ni, Y., & Xie, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02034 · Cited by 17
“In this stage, researchers become less interested in modifying the model architecture, but instead trying to pretrain or further pre-train a PLM from the open domain to the medical domain Guo et al. (2021a); Zhu (2021b); Gu et al. (2020).”
BackgroundCroxford, E., Gao, Y., Patterson, B. W., To, D., Tesch, S., Dligach, D., Mayampurath, A. M., Churpek, M. M., & Afshar, M. (2024) · medRxiv · doi:10.1101/2024.03.20.24304620 · Cited by 17
Assessing citation integrity in biomedical publications: corpus annotation and NLP models
Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae420 · Cited by 12
“For this classification task, we fine-tuned the PubMedBERT model (Gu et al. 2021).”
MethodInfluentialEstablishing Strong Baselines For TripClick Health Retrieval
Hofstätter, S., Althammer, S., Sertkan, M., & Hanbury, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_17 · Cited by 12
Automatic extraction of ranked SNP-phenotype associations from text using a BERT-LSTM-based method
Bokharaeian, B., Dehghani, M., & Díaz, A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05236-w · Cited by 12
“Therefore, recent biomedical text mining models rely primarily on the adapted versions of word representations such as SciBERT [13] for scientific texts and PubMedBERT-LSTM [14] for biomedical texts.”
Automated Ontology Evaluation: Evaluating Coverage and Correctness using a Domain Corpus
Zaitoun, A., Sagi, T., & Hose, K. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587617 · Cited by 12
BELB: a biomedical entity linking benchmark
Garda, S., Weber, L., Martin, R., & Leser, U. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad698 · Cited by 21
“Although several benchmarks have been developed for biomedical text mining, e.g. BLUE (Peng et al., 2019) and BLURB (Gu et al., 2021), BEL is notably absent from all of them.”
BackgroundRepresentation Learning for Person or Entity-Centric Knowledge Graphs: An Application in Healthcare
Theodoropoulos, C., Mulligan, N., Stappenbeck, T. S., & Bettencourt‐Silva, J. H. (2023) · International Conference on Knowledge Capture · doi:10.1145/3587259.3627545 · Cited by 12
“Alternatively to the introduction of sparsity using the BOW approach, a language model, such as BioBERT [39], PubMedBERT [24], and CharacterBERT [20, 63], can be used for the node initialization.”
MethodA comparative analysis of Spanish Clinical encoder-based models on NER and classification tasks
Subies, G. G., Jiménez, Á. B., & Fernández, P. M. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae054 · Cited by 16
Park, C., Lee, H., & Jeong, O. (2024) · Future Internet · doi:10.3390/fi16080260 · Cited by 12
Jonker, R. A. A., de Almeida, T. D., Antunes, R., Almeida, J. R., & Matos, S. (2024) · Database · doi:10.1093/database/baae068 · Cited by 17
Yada, S., Nakamura, Y., Wakamiya, S., & Aramaki, E. (2024) · Methods of Information in Medicine · doi:10.1055/a-2405-2489 · Cited by 12
Framework for automation of short answer grading based on domain-specific pre-training
Bonthu, S., Sree, S. R., & Prasad, M. H. M. K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109163 · Cited by 12
MediAlbertina: An European Portuguese medical language model
Nunes, M., Boné, J., Ferreira, J. C., Chaves, P., & Elvas, L. B. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109233 · Cited by 18
An Evaluation of Standard Statistical Models and LLMs on Time Series Forecasting
Cao, R., & Wang, Q. (2024) · IEEE International Conference on Future Machine Learning and Data Science (FMLDS) · doi:10.1109/fmlds63805.2024.00098 · Cited by 12
Wang, Y., Tong, H., Zhu, Z., Hou, F., & Li, Y. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06086-4 · Cited by 13
“• PubMedBERT [48] is a biomedical language model with a BERT architecture.”
DiMB-RE: mining the scientific literature for diet-microbiome associations
Hong, G., Hindle, V., Veasley, N. M., Holscher, H. D., & Halil Kilicoglu, B. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf054 · Cited by 12
“The current state-of-the-art method on these datasets [21] use a joint modeling approach for named entity recognition (NER) and relation extraction (RE) based on PubMedBERT [22].”
Enhancing Data Quality in Medical Concept Normalization Through Large Language Models
Chen, H., Li, R., Cleveland, A. D., & Ding, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104812 · Cited by 16
Ovi, M. S. I., Anjum, N., Bithe, T. H., Rahman, M. M., & Smrity, M. S. A. (2024) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit64611.2024.11021727 · Cited by 16
“Tools like OpenAI’s ChatGPT [4], [5], GitHub Copilot [6], [7], and Codeium [8] have become popular for their abilities in code generation, real-time debugging, and problem-solving support [9]–[11].”
Isaradech, N., Riedel, A., Sirikul, W., Kreuzthaler, M., & Schulz, S. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103165 · Cited by 14
Enhancing AI for citation screening in literature reviews: Improving accuracy with ensemble models
Zhang, Z., Nezhad, M. J. M., Gupta, P., Zolnour, A., Azadmaleki, H., Topaz, M., & Zolnoori, M. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106035 · Cited by 14
Enhancing biomedical relation extraction with directionality
Lai, P., Wei, C.-H., Tian, S., Leaman, R., & Lu, Z. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf226 · Cited by 12
“2-11B model performs comparably to PubMedBERT in the Novelty metrics despite PubMedBERT's domain-specific pre-training.”
InfluentialVerification is All You Need: Prompting Large Language Models for Zero-Shot Clinical Coding
Li, S., Zheng, C., Wu, J., Xu, Q., Xu, X., Wang, H., Sun, Y., Bai, Z., Xu, Y., Zhu, L., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3593028 · Cited by 13
LUK: Empowering Log Understanding With Expert Knowledge From Large Language Models
Ma, L., Yang, W., Jiang, S., Fei, B., Zhou, M., Li, S., Zhao, M., Xu, B., & Xiao, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3594046 · Cited by 17
“Many studies [50]– [52] have demonstrated that pre-training on domain corpus and fine-tuning with supervised data can improve performance on a specific task, even outperforming LLMs [53], [54].”
Schmidt, L., Campbell, F., Craig, D., Walker, V. R., Rooney, A. A., Schmitt, C., Härtung, T., & Thomas, J. (2025) · Evidence-Based Toxicology · doi:10.1080/2833373x.2025.2546791 · Cited by 13
TRIALSCOPE — A Framework for Clinical Trial Simulation from Real-World Data
Hernández, J. G., Ueno, R., Wong, C., Gero, Z., Bagga, J., Chien, I., Oravkin, E., Kıcıman, E., Nori, A. V., Weerasinghe, R. K., et al. (2025) · NEJM AI · doi:10.1056/aioa2400859 · Cited by 13
Self-citationLee, H.-S., Song, S.-H., Park, C., Seo, J., Kim, W. H., Kim, J.-I., Kim, S., Han, K., & Lee, Y. H. (2025) · BMC Medical Ethics · doi:10.1186/s12910-025-01285-3 · Cited by 16
“To assess the quality of the transformed radiology reports, an evaluation process was conducted by the radiologists who created the original reports and PubMed-BERTScore [37].”
Data curation in cheminformatics: importance and implementation
Esaki, T., & Ikeda, K. (2026) · Journal of Cheminformatics · doi:10.1186/s13321-026-01174-w · Cited by 14
“…conditions for legacy data: for example, recent literature-mining pipelines such as ChemEx and domain-specific language-model workflows automatically extract assay descriptors, units, and protocol parameters from publications and database documentation and link them to numerical endpoints [82, 83].”
Open-Domain Question-Answering for COVID-19 and Other Emergent Domains
Levy, S., Mo, K., Xiong, W., & Wang, W. Y. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.30 · Cited by 12
BioCaster in 2021: automatic disease outbreaks detection from global news media
Meng, Z., Okhmatovskaia, A., Polleri, M., Shen, Y., Powell, G. A., Fu, Z., Ganser, I., Zhang, M., King, N. B., Buckeridge, D. L., et al. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac497 · Cited by 12
Lifelines for a drowning science ‐ improving findability and synthesis of hydrologic publications
Stein, L., Mukkavilli, S. K., & Wagener, T. (2022) · Hydrological Processes · doi:10.1002/hyp.14742 · Cited by 12
Pyne, Y., Wong, Y.-M., Fang, H.-S., & Simpson, E. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100659 · Cited by 9
“The consultation and patient demographics for the OIAM dataset are given in table 1, and the number of transcripts with multiple labels is shown in figure 1.”
How Good Is ChatGPT for Medication Evidence Synthesis?
Liu, H., Peng, Y., & Weng, C. (2023) · Studies in health technology and informatics · doi:10.3233/shti230347 · Cited by 18
Narganes-Carlón, D., Crowther, D. J., & Pearson, E. R. (2023) · Scientific Reports · doi:10.1038/s41598-023-35597-4 · Cited by 16
BERT for Complex Systematic Review Screening to Support the Future of Medical Research
Hasny, M., Vasile, A.-P., Gianni, M., Bannach‐Brown, A., Nasser, M., Mackay, M., donovan, D., Šorli, J., Domocos, I., Dulloo, M., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34344-5_21 · Cited by 12
CardioBERTpt: Transformer-based Models for Cardiology Language Representation in Portuguese
Schneider, E. T. R., Gumiel, Y. B., de Souza, J. V. A., Cintho, L. M. M., Oliveira, L. E. S. E., Rebelo, M. D. S., Gutierrez, M. A., Krieger, J. E., Teodoro, D., Moro, C. M. C., et al. (2023) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms58004.2023.00247 · Cited by 12
AliBERT: A Pre-trained Language Model for French Biomedical Text
Berhe, A., Draznieks, G., Martenot, V., Masdeu, V., Davy, L., & Zucker, J. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.19 · Cited by 24
“Our pre-trained language models trained on in-domain (biomedical) textual documents tend to out-perform models that are trained on general domain textual documents which is also seen on the literature review of pre-trained language models for English language such as BioBERT (Lee et al., 2020), PubMedBERT (Gu et al.,…”
BackgroundMethodInfluentialVarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
Hu, D., Hou, X., Du, X., Zhou, M., Jiang, L., Mo, Y., & Shi, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.468 · Cited by 21
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
Cai, P., Yao, Z., Liu, F., Wang, D., Reilly, M. H., Zhou, H., Li, L., Cao, Y., Kapoor, A., Bajracharya, A. S., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00616 · Cited by 37
Zheng, Y. (2024) · Entertainment Computing · doi:10.1016/j.entcom.2024.100716 · Cited by 13
Analyzing Research Trends in Inorganic Materials Literature Using NLP
Kuniyoshi, F., Ozawa, J., & Miwa, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86517-7_20 · Cited by 12
K-12BERT: BERT for K-12 Education
Goel, V., Sahnan, D., Venktesh, V., Sharma, G., Dwivedi, D., & Mohania, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_123 · Cited by 14
What Do You See in this Patient? Behavioral Testing of Clinical NLP Models
van Aken, B., Herrmann, S., & Löser, A. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.7 · Cited by 14
Evaluating word representations in a technical language processing pipeline
Nandyala, A. V., Lukens, S., Rathod, S., & Agarwal, P. (2021) · PHM Society European Conference · doi:10.36001/phme.2021.v6i1.2894 · Cited by 12
BIOSCAN-5M: A Multimodal Dataset for Insect BiodiversityPreprint
Gharaee, Z., Lowe, S. C., Gong, Z., Arias, P. M., Pellegrino, N., Wang, A. T., Haurum, J. B., Zarubiieva, I., Kari, L., Steinke, D., et al. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.2406.12723 · Cited by 13
“Advances in transformers pretrained with SSL at scale, known as foundation models [38, 85, 14, 86, 13, 26], have shown robust performance across diverse tasks.”
BackgroundGenerative Artificial Intelligence-Based Medical Entity Data Extractor Using Large Language ModelsPreprint
Ghali, M.-K., Farrag, A., Sakai, H., baz, H. E., Jin, Y., & Lam, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5063216 · Cited by 12
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 344
RobBERTje: a Distilled Dutch BERT ModelPreprint
Delobelle, P., Winters, T., & Berendt, B. (2022) · Computational Linguistics in the Netherlands Journal · doi:10.48550/arxiv.2204.13511 · Cited by 17
Continual Contrastive Finetuning Improves Low-Resource Relation Extraction
Zhou, W., Zhang, S., Naumann, T., Chen, M., & Poon, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.739 · Cited by 11
Self-citationAiding ICD-10 Encoding of Clinical Health Records Using Improved Text Cosine Similarity and PLM-ICD
Silva, H., Duque, V., Macedo, M., & Mendes, M. (2024) · Algorithms · doi:10.3390/a17040144 · Cited by 14
Manion, F. J., Du, J., Wang, D., He, L., Lin, B., Wang, J., Wang, S., Eckels, D., Cervenka, J., Fiduccia, P. C., et al. (2024) · JMIR Medical Informatics · doi:10.2196/54653 · Cited by 11
LKAN: LLM-Based Knowledge-Aware Attention Network for Clinical Staging of Liver Cancer
Li, Y., Zheng, X., Li, J., Dai, Q., Wang, C., & Chen, M. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3478809 · Cited by 16
Aftiss, A., Lamsiyah, S., Alaoui, S. O. E., & Schommer, C. (2024) · IEEE Access · doi:10.1109/access.2024.3514915 · Cited by 11
“In the same context, PubMedBERT [80] is also a domain-specific model based on the BERT architecture, pre-trained on biomedical literature, including research articles and PubMed abstracts.”
Wang, S., Bian, J., Huang, X., Zhou, H., & Zhu, S. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3520579 · Cited by 11
Jin, M., Choi, S., & Kim, G.-W. (2025) · Electronics · doi:10.3390/electronics14020328 · Cited by 11
“Pretrained language models (PLMs), such as PubMedBERT and BioBERT, trained on biomedical corpora, have shown remarkable success in managing complex medical terminology, abbreviations, and overlapping entities [18,19].”
Structured Extraction of Real World Medical Knowledge using LLMs for Summarization and Search
Kim, E., Shrestha, M., Foty, R. G., DeLay, T., & Seyfert‐Margolis, V. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825160 · Cited by 15
A Primer on Word Embeddings: AI Techniques for Text Analysis in Social Work
Perron, B. E., Rivenburgh, K. A., Victor, B. G., Qi, Z., & Luan, H. (2025) · Journal of the Society for Social Work and Research · doi:10.1086/735577 · Cited by 12
Summarizing Clinical Notes using LLMs for ICU Bounceback and Length-of-Stay Prediction
Choudhuri, A., Polgreen, P. M., Segre, A. M., & Adhikari, B. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00118 · Cited by 13
NLP and AI for Public Health Intelligence: Automating Disease Surveillance from Unstructured Data
Methuku, V. (2025) · ICCK Transactions on Emerging Topics in Artificial Intelligence · doi:10.62762/tetai.2025.222799 · Cited by 11
Developing a large language model for oil- and gas-related rock mechanics: Progress and challenges
Lin, B., Jin, Y., Cao, Q., Meng, H., Pang, H., & Wei, S. (2025) · Natural Gas Industry B · doi:10.1016/j.ngib.2025.03.007 · Cited by 11
Liu, J., Nguyen, A. N., Capurro, D., & Verspoor, K. M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3574254 · Cited by 12
Izhar, A., Idris, N., & Japar, N. B. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103220 · Cited by 19
Qin, M., Feng, L., Lü, J., Sun, Z., Yu, Z., & Han, L. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.109070 · Cited by 11
Domain-Specific Language Model Pre-Training for Korean Tax Law Classification
Gu, Y. H., Piao, X., Yin, H., Dong, J., Zheng, R., & Yoo, S. J. (2022) · IEEE Access · doi:10.1109/access.2022.3164098 · Cited by 11
“In a study on solving a domain-specific task using a BERTbased pre-trained language model, the performance of the model was compared against that of a model that recorded SOTA performance in the respective domain [7], [9], [30].”
BackgroundMethodResultInfluentialSelf-citationSide-Aware Meta-Learning for Cross-Dataset Listener Diagnosis With Subjective Tinnitus
Liu, Z., Li, Y., Yao, L., Lucas, M. V., Monaghan, J. J. M., & Zhang, Y. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3201158 · Cited by 11
“languages processing [34], applications in health domains [6], [35].”
BackgroundSelf-citation
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.