CoChat

BioGPT: generative pre-trained transformer for biomedical text generation and mining

Open access

The DOI matches its Crossref record; 73 of the 74 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bib/bbac409

Cites

73 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 8

  1. Wang, 2019, International Conference on Learning Representations
  2. Devlin, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
  3. Clark, 2019, International Conference on Learning Representations
  4. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  5. Nentidis, Joint European Conference on Machine Learning and Knowledge Discovery in Databases
  6. Liu, 2020, Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 3787
  7. Tsu-Jui, 2019, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1409
  8. Kingma, 2015, International Conference on Learning Representations

Cited by

Items 901–1,000 of 1,017

Metadata

Authors
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y.
Affiliations
  • Microsoft Research Asia (China) (CN)
  • Peking University (CN)
  • Microsoft (United States) (US)
Corresponding author
  • Xia, Yingce
  • Qin, Tao
Venue
Briefings in Bioinformatics
Published
2022-09-24
Volume
23
Issue
6
ISSN
  • 1467-5463 (ISSN-L)
  • 1477-4054
OpenAlex
W4297253404
Cited by
1,150
References
65
Citations by year
2026: 329 · 2025: 375 · 2024: 287 · 2023: 159
FWCI
67.11
Type
journal article
Language
en
Open access
Locations
  • Briefings in Bioinformatics (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
BioGPT, generative pre-trained transformer, biomedical natural language processing, relation extraction, drug-drug interactions, PubMedQA, biomedical literature, domain-specific language models
MeSH terms
Natural Language Processing, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.