The DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.
A pre-training and self-training approach for biomedical named entity recognition
Open accessAbstract
Named entity recognition (NER) is a key component of many scientific literature mining tasks, such as information retrieval, information extraction, and question answering; however, many modern approaches require large amounts of labeled training data in order to be effective. This severely limits the effectiveness of NER models in applications where expert annotations are difficult and expensive to obtain. In this work, we explore the effectiveness of transfer learning and semi-supervised self-training to improve the performance of NER models in biomedical settings with very limited labeled data (250-2000 labeled samples). We first pre-train a BiLSTM-CRF and a BERT model on a very large general biomedical NER corpus such as MedMentions or Semantic Medline, and then we fine-tune the model on a more specific target NER task that has very limited training data; finally, we apply semi-supervised self-training using unlabeled data to further boost model performance. We show that in NER tasks that focus on common biomedical entity types such as those in the Unified Medical Language System (UMLS), combining transfer learning with self-training enables a NER model such as a BiLSTM-CRF or BERT to obtain similar performance with the same model trained on 3x-8x the amount of labeled data. We further show that our approach can also boost performance in a low-resource application where entities types are more rare and not specifically covered in UMLS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“self-training can propagate both knowledge and error from high confidence predictions on unlabelled samples [34]”
From PICO entity extraction for preclinical animal literature. (2022) · cites this paper as [34]
In the citing paper
“training from larger annotated corpora may reduce the error propagation and boost performance”
From PICO entity extraction for preclinical animal literature. (2022) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Using a base NER model such as the popular Bidirectional Long Short Term Memory Conditional Random Field (BiLSTM-CRF) [10] or state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) [11], we show that the combination of transfer learning and semi-supervised learning can significantly reduce the…”
BackgroundMethodSemi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 1×
“In transfer learning, a model that is trained on one task is then retrained on and applied to a different related task; knowledge gained when training on the first task may boost performance on the second task, especially when labeled training data is scarce for the second task [27].”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“For our BERT model, we utilize the pre-trained WordPiece vocabulary and model weights from BlueBERT Base [53], which is the BERT Base model pre-trained on PubMed abstracts and MIMIC III clinical notes, as this model has shown superior performance on biomedical and medical NER tasks compared to other BERT-based models…”
MethodNeural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 1×
“Recent work has shown that supervised machine learning approaches, especially deep neural networks, achieve superior performance on various NER tasks [1, 2, 21].”
BackgroundTarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Miyato, T., Maeda, S.-I., Koyama, M., & Ishii, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858821 · Cited by 2,843
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
Research works printed but not linked: 4
- Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805. 2018;.
- JY Lee, 2018, LREC
- J Lee, 2020, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36, 1234, 10.1093/bioinformatics/btz682 · DOI · Paper page
- A Tarvainen, 2017, ICLR (Workshop)
Cited by
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
“For example, Gao et al. [63] fine-tuned BlueBERT on a large general biomedical NER corpus like MedMentions [64] or Semantic Medline before fine-tuning on the small target NER corpus.”
BackgroundMethodInfluentialPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
“BioBERT [121], BlueBERT [179], SciBERT [16], PubMedBERT [66], BioMegatron [215], Yuan et al [286], Wei et al [258], Gao et al [58], Mascio et al [148], Guo et al [69], [73], Sont et al [221].”
MethodInfluentialThe role of large language models in medical image processing: a narrative review
Tian, D., Jiang, S., Zhang, L., Lü, X., & Xu, Y. (2023) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-23-892 · Cited by 125
“…imaging field, the studies conducted by Shang Gao and J. Blair Christian’s team exploring the efficacy of early LLM Bidirectional Encoder Representations from Transformers (BERT) transfer learning and semi-supervised self-training in Named Entity Recognition (NER) could be a reference point (26).”
Large-scale application of named entity recognition to biomedicine and epidemiology
Raza, S., Reji, D. J., Shajan, F., & Bashir, S. R. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000152 · Cited by 60
“However, the latter works show that adding the language models on top of traditional models (e.g., BiLSTM) shows good performance [42,43], which may be attributed to large-scale pre-training tasks on the top.”
BackgroundBiomedical named entity recognition based on fusion multi-features embedding
Li, M., Yang, H., & Liu, Y. (2023) · Technology and Health Care · doi:10.3233/thc- · Cited by 51
Metadata
- Authors
- Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B.
- Affiliations
- Oak Ridge National Laboratory (US)
- Corresponding author
- Gao, Shang
- Christian, J. Blair
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-02-09
- Volume
- 16
- Issue
- 2
- Pages
- e0246310
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7872256
- OpenAlex
- W3129160532
- Cited by
- 46
- References
- 71
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 9 · 2023: 14 · 2022: 10 · 2021: 5
- FWCI
- 2.98
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- U.S. Department of Energy
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical named entity recognition, transfer learning, limited labeled data, BiLSTM-CRF, BERT, fine-tuning, UMLS, scientific literature mining, unlabeled data
- MeSH terms
- Supervised Machine Learning, Unsupervised Machine Learning, Artificial Intelligence, Humans, Models, Theoretical, Terminology as Topic, Transfer, Psychology, Unified Medical Language System, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A pre-training and self-training approach for biomedical named entity recognition. https://app.cochat.ai/papers/10.1371/journal.pone.0246310
- The paper
- Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B. (2021). A pre-training and self-training approach for biomedical named entity recognition. PLOS ONE. https://doi.org/10.1371/journal.pone.0246310
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.