BioGPT: generative pre-trained transformer for biomedical text generation and mining
Open accessThe DOI matches its Crossref record; 73 of the 74 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our language module uses BioGPT23, an autoregressive generative model based on the Transformer24 decoder architecture of GPT-325, trained on 15 million biomedical articles from PubMed with a vocabulary size of 42,384.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“BioGPT is based on GPT-246 and trained using biomedical literature and has shown improved performance on text classification tasks over BERT-based architectures23.”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“fine-tuned BioGPT23 has shown superior performance to BERT-based models in document classification”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [23]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Research works printed but not linked: 8
- Wang, 2019, International Conference on Learning Representations
- Devlin, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
- Clark, 2019, International Conference on Learning Representations
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Nentidis, Joint European Conference on Machine Learning and Knowledge Discovery in Databases
- Liu, 2020, Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 3787
- Tsu-Jui, 2019, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1409
- Kingma, 2015, International Conference on Learning Representations
Cited by
Items 1,001–1,017 of 1,017
Automatic Medical Generated Analysis Based on Histopathology Image Segmentation Half UNet
Veraldo, N. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.70521
Atalor, S. I., & Omachi, A. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i8.495 · Cited by 1
VisArena: A testbed for visualization with synthetic data tables
Fu, S., Jin, J., Ma, Q., Luo, Z., Xiong, K., & Wu, Y. (2025) · Journal of Visualization · doi:10.1007/s12650-025-01069-2
Optimizing LLaMa2 Fine-Tuning with LoRA and Quantization for Medical Question Answering
Bui, T.-T., Truong, D., & Ngo, L. T. (2025) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-90194-2_34
Bi-Encoder-Based Approach to Biomedical Document-Level Entity Recognition and Relation Extraction
Nie, P.-Y., Lin, M.-X., Ning, J.-Z., Yang, Z.-H., & Wang, L. (2025) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2025.3582790
Benrubi, L., Sato, T., Westgard, L. K., Zollo‐Venecek, K., Socrates, B., Sweigart, B., Ridgway, J. P., Suzuki, J., Morales, Y., Goodman‐Meza, D., et al. (2025) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofaf370
Yi, K. S., & Choi, C. (2025) · Institute of Brain-based Education Korea National University of Education · doi:10.31216/bdl.2025.15.2.6
Development of a GI Cancer Diagnostic Chatbot Based on RAG Model
Vijayvargiya, A., Kaushish, A., Rawat, K., Kumar, M., Sr., & Singh, G. D. (2025) · OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry (OTCON) · doi:10.1109/otcon65728.2025.11070521
Shao, Y., Xia, Chen, Y., Shen, Y., Zhou, Q., Yang, Y., & Tong, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0036-9_13
Bridging Knowledge Gaps: Fine-Tuned RAG Frameworks for Biomedical Evidence-Based Question Answering
Wang, X., You, Z.-H., Li, Y., Yan, Y.-H., & Li, Y.-C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_7
Vurgun, U., Kaviyarasu, A., Hwang, S., Batugo, A., Thomas, S., Tang, B., Acevedo, A., Mitchell, O. J. L., & Mowery, D. L. (2025) · medRxiv · doi:10.1101/2025.08.04.25331524
Enhancing Abstractive Summarization with T5SAM: A Transformer-Based Approach
Chaudhary, H., & Solanki, A. (2025) · International Conference on Networks and Cryptology (NETCRYPT) · doi:10.1109/netcrypt65877.2025.11102242
Efficient Application of Transformer Structure in Text Generation Tasks
Pan, Q., & Yang, B. (2025) · International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise66443.2025.11100195
Exposing the Molecular Reaction Blind Spots of LLMs with PathwayQAPreprint
Nayar, G., Carpenter, K. A., Smith, D. A., Xiong, B., & Altman, R. B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.12.669911
Autonomous LLMs for Healthcare Diagnoses
Ahmad, H. F., & Shafqat, S. (2025) · International Symposium on Autonomous Decentralized Systems · doi:10.1109/isads66912.2025.00015
Bogadi, C. R. (2025) · International Journal of Computer Trends and Technology · doi:10.14445/22312803/ijctt-v73i7p101 · Cited by 1
Caufield, J. H., Hegde, H., Emonet, V., Harris, N. L., Joachimiak, M. P., Matentzoglu, N., Kim, H., Moxon, S. A. T., Reese, J. T., Haendel, M. A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.02711
Metadata
- Authors
- Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y.
- Affiliations
- Microsoft Research Asia (China) (CN)
- Peking University (CN)
- Microsoft (United States) (US)
- Corresponding author
- Xia, Yingce
- Qin, Tao
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-09-24
- Volume
- 23
- Issue
- 6
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- OpenAlex
- W4297253404
- Cited by
- 1,150
- References
- 65
- Citations by year
- 2026: 329 · 2025: 375 · 2024: 287 · 2023: 159
- FWCI
- 67.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Briefings in Bioinformatics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- BioGPT, generative pre-trained transformer, biomedical natural language processing, relation extraction, drug-drug interactions, PubMedQA, biomedical literature, domain-specific language models
- MeSH terms
- Natural Language Processing, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.