CoChat

A pre-training and self-training approach for biomedical named entity recognition

Open access

The DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.

Abstract

Named entity recognition (NER) is a key component of many scientific literature mining tasks, such as information retrieval, information extraction, and question answering; however, many modern approaches require large amounts of labeled training data in order to be effective. This severely limits the effectiveness of NER models in applications where expert annotations are difficult and expensive to obtain. In this work, we explore the effectiveness of transfer learning and semi-supervised self-training to improve the performance of NER models in biomedical settings with very limited labeled data (250-2000 labeled samples). We first pre-train a BiLSTM-CRF and a BERT model on a very large general biomedical NER corpus such as MedMentions or Semantic Medline, and then we fine-tune the model on a more specific target NER task that has very limited training data; finally, we apply semi-supervised self-training using unlabeled data to further boost model performance. We show that in NER tasks that focus on common biomedical entity types such as those in the Unified Medical Language System (UMLS), combining transfer learning with self-training enables a NER model such as a BiLSTM-CRF or BERT to obtain similar performance with the same model trained on 3x-8x the amount of labeled data. We further show that our approach can also boost performance in a low-resource application where entities types are more rare and not specifically covered in UMLS.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0246310

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 NER workflow used for our experiments.
Fig 1 NER workflow used for our experiments.CC0 · PubMed Central
Fig 2 Performance of the BiLSTM-CRF after each iteration of self-training for three scenarios: S800 with 250 initial labeled sentences, Linnaeus with 250 initial labeled sentences, and BC2GM with 1000 initial labeled sentences. For each scenario, we also show the percentage of total available data (train + unsupervised) added to the training set after each iteration of self-training.
Fig 2 Performance of the BiLSTM-CRF after each iteration of self-training for three scenarios: S800 with 250 initial labeled sentences, Linnaeus with 250 initial labeled sentences, and BC2GM with 1000 initial labeled sentences. For each scenario, we also show the percentage of total available data (train + unsupervised) added to the training set after each iteration of self-training.CC0 · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 4

  1. Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805. 2018;.
  2. JY Lee, 2018, LREC
  3. J Lee, 2020, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36, 1234, 10.1093/bioinformatics/btz682 · DOI · Paper page
  4. A Tarvainen, 2017, ICLR (Workshop)

Cited by

See all 46

Metadata

Authors
Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B.
Affiliations
Oak Ridge National Laboratory (US)
Corresponding author
  • Gao, Shang
  • Christian, J. Blair
Venue
PLOS ONE
Published
2021-02-09
Volume
16
Issue
2
Pages
e0246310
ISSN
1932-6203 (ISSN-L)
OpenAlex
W3129160532
Cited by
46
References
71
Citations by year
2026: 2 · 2025: 6 · 2024: 9 · 2023: 14 · 2022: 10 · 2021: 5
FWCI
2.98
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, public-domain)
  • PubMed Central (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Funders
U.S. Department of Energy
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
biomedical named entity recognition, transfer learning, limited labeled data, BiLSTM-CRF, BERT, fine-tuning, UMLS, scientific literature mining, unlabeled data
MeSH terms
Supervised Machine Learning, Unsupervised Machine Learning, Artificial Intelligence, Humans, Models, Theoretical, Terminology as Topic, Transfer, Psychology, Unified Medical Language System, Recognition, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.