A pre-training and self-training approach for biomedical named entity recognition
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.
Abstract
Named entity recognition (NER) is a key component of many scientific literature mining tasks, such as information retrieval, information extraction, and question answering; however, many modern approaches require large amounts of labeled training data in order to be effective. This severely limits the effectiveness of NER models in applications where expert annotations are difficult and expensive to obtain. In this work, we explore the effectiveness of transfer learning and semi-supervised self-training to improve the performance of NER models in biomedical settings with very limited labeled data (250-2000 labeled samples). We first pre-train a BiLSTM-CRF and a BERT model on a very large general biomedical NER corpus such as MedMentions or Semantic Medline, and then we fine-tune the model on a more specific target NER task that has very limited training data; finally, we apply semi-supervised self-training using unlabeled data to further boost model performance. We show that in NER tasks that focus on common biomedical entity types such as those in the Unified Medical Language System (UMLS), combining transfer learning with self-training enables a NER model such as a BiLSTM-CRF or BERT to obtain similar performance with the same model trained on 3x-8x the amount of labeled data. We further show that our approach can also boost performance in a low-resource application where entities types are more rare and not specifically covered in UMLS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“self-training can propagate both knowledge and error from high confidence predictions on unlabelled samples [34]”
From PICO entity extraction for preclinical animal literature. (2022) · cites this paper as [34]
In the citing paper
“training from larger annotated corpora may reduce the error propagation and boost performance”
From PICO entity extraction for preclinical animal literature. (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Using a base NER model such as the popular Bidirectional Long Short Term Memory Conditional Random Field (BiLSTM-CRF) [10] or state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) [11], we show that the combination of transfer learning and semi-supervised learning can significantly reduce the…”
BackgroundMethodSemi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 1×
“In transfer learning, a model that is trained on one task is then retrained on and applied to a different related task; knowledge gained when training on the first task may boost performance on the second task, especially when labeled training data is scarce for the second task [27].”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“For our BERT model, we utilize the pre-trained WordPiece vocabulary and model weights from BlueBERT Base [53], which is the BERT Base model pre-trained on PubMed abstracts and MIMIC III clinical notes, as this model has shown superior performance on biomedical and medical NER tasks compared to other BERT-based models…”
MethodNeural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 1×
“Recent work has shown that supervised machine learning approaches, especially deep neural networks, achieve superior performance on various NER tasks [1, 2, 21].”
BackgroundTarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Miyato, T., Maeda, S.-I., Koyama, M., & Ishii, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858821 · Cited by 2,843
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
Research works printed but not linked: 4
- Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805. 2018;.
- JY Lee, 2018, LREC
- J Lee, 2020, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36, 1234, 10.1093/bioinformatics/btz682 · DOI · Paper page
- A Tarvainen, 2017, ICLR (Workshop)
Cited by
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
“For example, Gao et al. [63] fine-tuned BlueBERT on a large general biomedical NER corpus like MedMentions [64] or Semantic Medline before fine-tuning on the small target NER corpus.”
BackgroundMethodInfluentialPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
“BioBERT [121], BlueBERT [179], SciBERT [16], PubMedBERT [66], BioMegatron [215], Yuan et al [286], Wei et al [258], Gao et al [58], Mascio et al [148], Guo et al [69], [73], Sont et al [221].”
MethodInfluentialThe role of large language models in medical image processing: a narrative review
Tian, D., Jiang, S., Zhang, L., Lu, X., & Xu, Y. (2024) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-23-892 · Cited by 95
“…imaging field, the studies conducted by Shang Gao and J. Blair Christian’s team exploring the efficacy of early LLM Bidirectional Encoder Representations from Transformers (BERT) transfer learning and semi-supervised self-training in Named Entity Recognition (NER) could be a reference point (26).”
Large-scale application of named entity recognition to biomedicine and epidemiology
Raza, S., Reji, D. J., Shajan, F., & Bashir, S. R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000152 · Cited by 81
“However, the latter works show that adding the language models on top of traditional models (e.g., BiLSTM) shows good performance [42,43], which may be attributed to large-scale pre-training tasks on the top.”
BackgroundBiomedical named entity recognition based on fusion multi-features embedding
Li, M., Yang, H., & Liu, Y. (2023) · Technology and Health Care · doi:10.3233/thc- · Cited by 52
Klein, L., Seidl, D., Fulneček, J., Prokop, L., Mišák, S., & Dvorský, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118910 · Cited by 49
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05006 · Cited by 34
“BioBERT [121], BlueBERT [179], SciBERT [16], PubMedBERT [66], BioMegatron [215], Yuan et al [286], Wei et al [258], Gao et al [58], Mascio et al [148], Guo et al [69], [73], Sont et al [221].”
MethodInfluentialCesario, E., Comito, C., & Zumpano, E. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127079 · Cited by 39
PICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
“Previous studies show that self-training can propagate both knowledge and error from high confidence predictions on unlabelled samples [34] and that training from larger annotated corpora may reduce the error propagation and boost performance.”
BackgroundRaza, S., & Schwartz, B. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02117-3 · Cited by 41
“Recent developments in biomedical NER include the use of transfer learning [17], BERT-like [13], attention-based [18], multi-task learning [19], and hybrid models [20] to improve the performance of these models.”
MethodA review on Natural Language Processing Models for COVID-19 research
Hall, K., Chang, V. I. C., & Jayne, C. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100078 · Cited by 33
“n [47], a novel EE approach was taken by combining intermediate ine-tuning with semi-supervised learning.”
Integrating domain knowledge for biomedical text analysis into deep learning: A survey
Cai, L., Li, J., Lv, H., Liu, W., Niu, H., & Wang, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104418 · Cited by 27
Multi-Label Classification of Indonesian Online Toxicity using BERT and RoBERTa
Sagama, Y., & Alamsyah, A. (2023) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict59002.2023.10205892 · Cited by 29
“During pre-training, the model engages with a large text corpus, learning language structure through two unsupervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) [18].”
MethodSemi-supervised geological disasters named entity recognition using few labeled data
Lei, X., Song, W., Fan, R., Feng, R., & Wang, L. (2022) · GeoInformatica · doi:10.1007/s10707-022-00474-1 · Cited by 24
“In the bio-medical field, there are some NER approaches that combine both pre-trained model transferring and self-training strategies [8, 12, 24].”
BackgroundMethodRecent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative StudyPreprint
Keraghel, I., Morbieu, S., & Nadif, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10825 · Cited by 176
“Another method is self-training [Gao et al., 2021], where the initial model is trained on the labeled data, and then used to predict the labels for the unlabeled data.”
Arzideh, K., Schäfer, H., Allende‐Cid, H., Baldini, G., Hilser, T., Idrissi-Yaghir, A., Laue, K., Chakraborty, N., Doll, N., Antweiler, D., et al. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110665 · Cited by 23
Alamsyah, A., & Sagama, Y. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200394 · Cited by 33
Zhou, M., Tan, J., Song, Y., Wang, H., Wang, L., & Xiao, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3299824 · Cited by 19
Chen, Y., Zhang, L., & Dong, Q. (2024) · Journal of Cultural Heritage · doi:10.1016/j.culher.2024.09.011 · Cited by 20
Named Entity Recognition in Unstructured Medical Text Documents
Pearson, C., Seliya, N., & Dave, R. (2021) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet52533.2021.9698694 · Cited by 22
Hanslo, R. (2022) · Annals of Computer Science and Information Systems · doi:10.15439/2022f53 · Cited by 14
“Loubser and Puttkammer [1] used bi-LSTM-aux and bi-LSTM-aux with embeddings model variations for their study.”
BackgroundInfluentialMF-MNER: Multi-models Fusion for MNER in Chinese Clinical Electronic Medical Records
Du, H., Xu, J., Du, Z., Chen, L., Ma, S., Wei, D., & Wang, X. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00624-z · Cited by 11
“In recent years, pre-training methods utilizing massive-scale unlabeled data have provided new solutions [21].”
BackgroundImproving biomedical named entity recognition through transfer learning and asymmetric tri-training
Bhattacharya, M., Bhat, S., Tripathy, S., Bansal, A., & Choudhary, M. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.244 · Cited by 14
Biomedical named entity recognition based on fusion multi-features embedding
Li, M., Yang, H., & Liu, Y. (2023) · Technology and Health Care · doi:10.3233/thc-236011 · Cited by 11
Harnessing high-quality pseudo-labels for robust few-shot nested named entity recognition
Ming, H., Yang, J., Liu, S., Jiang, L., & An, N. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110992 · Cited by 8
A lightweight biomedical named entity recognition with pre-trained model
Gou, Y., & Cao, J. (2023) · IEEE International Conference on Data Science and Computer Application (ICDSCA) · doi:10.1109/icdsca59871.2023.10392374 · Cited by 6
“Tokenization[10] is essential because it breaks down the text into smaller units that can be processed by NLP models.”
BackgroundEnhanced semantic representation model for multisource point of interest attribute alignment
Li, P., Wang, Y., Liu, J., Luo, A., Xu, S., & Zhang, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101852 · Cited by 7
Zhang, S., Gharleghi, R., Singh, S., Shen, C., Adikari, D., Zhang, M., Moses, D. A., Vickers, D., Sowmya, A., & Beier, S. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01677-2 · Cited by 7
“Therefore, researchers continuously seek methods to leverage pre-annotated public datasets for model training and subsequent application to unannotated private datasets [8].”
Qi, R., Lv, P., Zhang, Q., & Wu, M. (2022) · Applied Sciences · doi:10.3390/app12178539 · Cited by 3
Rattanatamrong, P., Boonpalit, Y., & Boonnavasin, M. (2024) · Journal of Information Science · doi:10.1177/01655515231220164 · Cited by 3
“Such pre-trained NER models are particularly useful when resources, time constraints and data availability make it challenging to develop custom models immediately [43].”
An Empirical Cross Domain-Specific Entity Recognition with Domain Vector
Chen, W., Han, S., & Huang, H. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557545 · Cited by 3
“In representation transfer, a popular method is to pre-training language models [2, 9], such as Bert, using the corpus of the two domains, but it still needs a large amount of unlabeled data in the target domain.”
MethodSearching COVID-19 Clinical Research Using Graph Queries: Algorithm Development and Validation
Invernici, F., Bernasconi, A., & Ceri, S. (2024) · Journal of Medical Internet Research · doi:10.2196/52655 · Cited by 2
“Bio-ontologies are also applied in natural language processing tasks, such as NER [43].”
BackgroundResearch on Joint Extraction of Entity and Relation in Geological Hazard Text
周, 榆. (2021) · Computer Science and Application · doi:10.12677/csa.2021.1111271 · Cited by 1
Marxer, C., Rölke, H., Alfieri, A., & Halatsch, M. (2023) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds57534.2023.00009 · Cited by 1
“Especially in the field of bio-medical research, NER represents an important key tool in order to be able to keep up with the constant growth of newly discovered and defined concepts and entities from literature, such as new experimental active substances or drugs [23].”
Identification Semantic Text of Indonesian Medical Terms from Question-Answer Data
Purwitasari, D., Juanita, S., Purnama, I. K. E., Raihan, M., & Purnomo, M. H. (2022) · International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE) · doi:10.1109/icitisee57756.2022.10057601 · Cited by 1
“NER locates and classifies certain entities in one or more bodies of text [11].”
BackgroundYang, Y., Wang, Z., & Jiang, Z. (2021) · International Conference on Network and Information Systems for Computers (ICNISC) · doi:10.1109/icnisc54316.2021.00124 · Cited by 1
Accelerated variant curation from scientific literature using biomedical text mining
Mallick, R., Arnaboldi, V., Davis, P., Diamantakis, S., Zarowiecki, M., & Howe, K. (2022) · PubMed Central · doi:10.17912/micropub.biology.000578
“It has been previously observed that a BiLSTM-CRF performs comparatively or in certain cases even better than a BERT based NER (Gao et al. 2021) , thus our NER+RegEx solution seems suitable for good performance on the semi-structured data of protein changing mutations.”
MethodZhu, H., & Mitrofanova, O. A. (2024) · Philology Theory & Practice · doi:10.30853/phil20240430
Weakly-Supervised Learning Strategy for Domain-Specific Entity Recognition
Weber, L. J., Ramalingam, K. J., Liu, C., Beyer, M., & Zimmermann, A. P. (2024) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds62089.2024.10756418
“…employed in weakly-supervised training pipelines encompass teacher-student frameworks [21]–[23] with focus on models’ high confidence predictions [24], [25], contrastive learning approaches [26]–[28], the use of optimized loss functions [29]–[31], ensemble techniques [32]–[34], and the…”
Deep Learning Methods for Biomedical Named Entity Recognition: A Comprehensive Review
Sabitha, S., & Pillai, A. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-78937-3_11
Proposing an NLP-Based Model for Multi-Class Classification of Holland Personality
Anggraini, E. K., Alamsyah, A., & Dudija, N. (2025) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera66156.2025.11087300
“In the pre-training phase, BERT learns language representations by processing a large-scale text corpus through two self-supervised objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) [23].”
A Novel Low-Resource Graph Convolutional Neural Network Approach for Entity Recognition
Zhao, Z., & Li, X. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3614139
“Howevergoldstandardlabeleddatais expensive and difficult to obtain in a biomedical environment [17], as it usually requires expensive expert knowledge.”
Raza, S., & Schwartz, B. (2024) · Figshare · doi:10.6084/m9.figshare.26564418
Large Language Model Judged Self-Training for Named Entity Recognition
Chen, S., Wang, J., Yang, C., Xiao, Y., Li, Z., & Lin, X. A. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3778009
Tran, Q.-V.-A., Duong, G.-B., Tran, H.-H.-H., Vuong, T.-H.-Y., & Le, H.-Q. (2026) · Database · doi:10.1093/database/baag056
“Self-training is also widely adopted in low-resource scenarios [ 28 , 35 ].”
Metadata
- Authors
- Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B.
- Affiliations
- Oak Ridge National Laboratory (US)
- Corresponding author
- Gao, Shang
- Christian, J. Blair
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-02-09
- Volume
- 16
- Issue
- 2
- Pages
- e0246310
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7872256
- OpenAlex
- W3129160532
- Cited by
- 46
- References
- 71
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 9 · 2023: 14 · 2022: 10 · 2021: 5
- FWCI
- 2.98
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- U.S. Department of Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical named entity recognition, transfer learning, limited labeled data, BiLSTM-CRF, BERT, fine-tuning, UMLS, scientific literature mining, unlabeled data
- MeSH terms
- Supervised Machine Learning, Unsupervised Machine Learning, Artificial Intelligence, Humans, Models, Theoretical, Terminology as Topic, Transfer, Psychology, Unified Medical Language System, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.