CoChat
Sign Up
Cited by 333Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

A generalist vision-language foundation model for diverse biomedical tasks

Open access
Zhang, Kai, Zhou, Rong, Adhikarla, Eashan, Yan, Zhiling, Liu, Yixin, Yu, Jun, Liu, Zhengliang, Chen, Xun +15 more·2024·Nature Medicine
DOI 10.1038/s41591-024-03185-2PMC PMC12581140
Open this paper in CoChat — free accountOpen PDF

Abstract

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners and patients. Here, we describe BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03185-2

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • BiomedGPT integrated different modalities of clinical images and corresponding clinical reports using unified tokenization methods.

    “BiomedGPT integrated different modalities of clinical images and corresponding clinical reports using unified tokenization methods [197].”

    From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [197]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 65 · Cited by 333

Research works printed but not linked: 61

  1. [1]Thirunavukarasu AJ Large language models in medicine. Nat. Med. 29, 1930– 1940 (2023). 37460753
  2. [3]Moody L The person-centred care guideline: from principle to practice. J. Patient Exp. 5, 282– 288 (2018). 30574549 PMC6295806
  3. [4]Langberg EM, Dyhr L & Davidsen AS Development of the concept of patient-centredness–a systematic review. Patient Educ. Couns. 102, 1228– 1236 (2019). 30846206
  4. [5]Bates DW Reducing the frequency of errors in medicine using information technology. J. Am. Med. Inform. Assoc. 8, 299– 308 (2001). 11418536 PMC130074
  5. [7]Reed S A generalist agent. Transact. Mach. Learn. Res. https://openreview.net/pdf?id=1ikK0kHjvj (2022).
  6. [9]Vaswani A Attention is all you need. In Advances in Neural Information Processing Systems 30 (Neural Information Processing Systems Foundation, 2017).
  7. [10]Brown T Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877– 1901 (2020).
  8. [11]Touvron H Llama: open and efficient foundation language models. Preprint at https://arxiv.org/abs/2302.13971 (2023). · arXiv
  9. [13]Wu C, Zhang X, Zhang Y, Wang Y, & Xie W Towards generalist foundation model for radiology. Preprint at https://arxiv.org/abs/2308.02463 (2023). PMC12375113 40849424 · arXiv
  10. [14]Luo R BioGPT: generative pretrained transformer for biomedical text generation and mining. Brief. Bioinform. 23, bbac409 (2022). 36156661
  11. [15]Zhang S Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. Preprint at https://arxiv.org/abs/2303.00915 (2023). · arXiv
  12. [16]Phan LN Scifive: a text-to-text transformer model for biomedical literature. Preprint at https://arxiv.org/abs/2106.03598 (2021). · arXiv
  13. [17]Lau J A dataset of clinically generated visual questions and answers about radiology images. Sci. Data 5, 180251 (2018). 30457565 PMC6244189
  14. [18]Liu B Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In Proc. IEEE International Symposium on Biomedical Imaging (ISBI) 1650– 1654 (Institute of Electrical and Electronics Engineers, 2021).
  15. [19]He X Towards visual question answering on pathology images. In Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 708– 718 (Association for Computational Linguistics. 2021).
  16. [20]Demner-Fushman D Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. 23, 304– 310 (2016). 26133894 PMC5009925
  17. [22]Pavlopoulos J, Kougia V, & Androutsopoulos I A survey on biomedical image captioning. In Proc. Second Workshop on Shortcomings in Vision and Language 26– 36 (Association for Computational Linguistics, 2019).
  18. [23]Li P Self-supervised vision-language pretraining for medial visual question answering. In Proc. IEEE 20th International Symposium on Biomedical Imaging (ISBI) 1– 5 (Institute of Electrical and Electronics Engineers, 2023).
  19. [25]Van Sonsbeek T Open-ended medical visual question answering through prefix tuning of language models. In International Conference on Medical Image Computing and Computer-Assisted Intervention 726– 736 (MICCAI, 2023).
  20. [26]Lin CY Rouge: a package for automatic evaluation of summaries. In Text Summarization Branches Out 74– 81 (Association for Computational Linguistics, 2004).
  21. [27]Banerjee S & Lavie A Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (eds. Goldstein J, Lavie A, Lin C-Y & Voss C) 65– 72 (Association for Computational Linguistics, 2005).
  22. [28]Vedantam R, Zitnick CL & Parikh D Cider: Consensus-based image description evaluation. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR) 4566– 4575 (Institute of Electrical and Electronics Engineers, 2015).
  23. [29]Jing B, Xie P & Xing E On the automatic generation of medical imaging reports. In Proc. 56th Annual Meeting of the Association for Computational Linguistics (eds. Gurevych I & Miyao Y) 2577– 2586 (Association for Computational Linguistics, 2017).
  24. [31]Liu F Exploring and distilling posterior and prior knowledge for radiology report generation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 13753– 13762 (Institute of Electrical and Electronics Engineers/Computer Vision Foundation, 2021).
  25. [32]Yuan H Biobart: pretraining and evaluation of a biomedical generative language model. In Proc. 21st Workshop on Biomedical Language Processing (eds. Demner-Fushman D, Cohen KB, Ananiadou S & Tsujii J) 97– 109 (Association for Computational Linguistics, 2022).
  26. [33]Van Veen D Radadapt: radiology report summarization via lightweight domain adaptation of large language models. In 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks (eds. Demner-fushman, Ananiadou & Cohen) 449– 460 (Association for Computational Linguistics, 2023).
  27. [38]Jaeger S Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant. Imaging Med. Surg. 4, 475– 477 (2014). 25525580 PMC4256233
  28. [39]Capellán-Martín D A lightweight, rapid and efficient deep convolutional network for chest x-ray tuberculosis detection. In Proc. 2023 IEEE 20th Int. Symp. Biomed. Imaging (ISBI) 1– 5 (IEEE, 2023).
  29. [40]Manzari ON Medvit: a robust vision transformer for generalized medical image classification. Comput. Biol. Med. 157, 106791 (2023). 36958234
  30. [41]Lee RS A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 1– 9 (2017). PMC5735920 29257132
  31. [42]Romanov A & Shivade C Lessons from natural language inference in the clinical domain. In Proc. 2018 Conference on Empirical Methods in Natural Language Processing 1586– 1596 (Association for Computational Linguistics, 2018).
  32. [43]Gloeckler Ries LA Cancer survival and incidence from the surveillance, epidemiology, and end results (SEER) program. Oncologist 8, 541– 552 (2003). 14657533
  33. [44]Abacha AB, & Demner-Fushman D On the summarization of consumer health questions. In Proc. 57th Annual Meeting of the Association for Computational Linguistics 2228– 2234 (2019).
  34. [45]Zeng G Meddialog: large-scale medical dialogue datasets. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 9241– 9250 (Association for Computational Linguistics, 2020).
  35. [46]Johnson AE MIMIC-III a freely accessible critical care database. Sci. Data 3, 1– 9 (2019). PMC4878278 27219127
  36. [47]Dubey S Using machine learning for healthcare treatment planning. Front. Artif. Intell. 6, 1124182 (2023). 37181733 PMC10167842
  37. [48]Roberts K Overview of the TREC 2021 clinical trials track. In Proc. Thirtieth Text Retrieval Conference (TREC, 2021).
  38. [49]Van Aken B Clinical outcome prediction from admission notes using self-supervised knowledge integration. In Proc. 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume 881– 893 (Association for Computational Linguistics, 2021).
  39. [51]Wang P OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. Proc. Int. Conf. Mach. Learn. PMLR 162, 23318– 23340 (2022).
  40. [52]Hu X Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering. In Proc. 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 4156– 4165 (Association for Computing Machinery, 2023).
  41. [53]Jeong J Multimodal image-text matching improves retrieval-based chest x-ray report generation. In Proc. Medical Imaging with Deep Learning 227 978– 990 (Proceedings of Machine Learning Research, 2024).
  42. [54]Fu S Assessment of data quality variability across two EHR systems through a case study of post-surgical complications. In Proc. AMIA Joint Summits on Translational Science 196– 205 (American Medical Informatics Association, 2022). PMC9285181 35854735
  43. [56]Yang H, Lin J, Yang A, Wang P & Zhou C Prompt tuning for unified multimodal pretrained models. In Findings of the Association for Computational Linguistics: ACL 2023 (eds. Rogers A, Boyd-Graber J & Okazaki N) 402– 416 (Association for Computational Linguistics, 2023).
  44. [57]Chen Z Towards understanding the mixture-of-experts layer in deep learning. Adv. Neural Inf. Process. Syst. 35, 23049– 23062 (2022).
  45. [58]Dosovitskiy A An image is worth 16×16 words: transformers for image recognition at scale. In International Conference on Learning Representations. (2021).
  46. [59]Devlin J, Chang MW, Lee K & Toutanova K BERT: pretraining of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds. Burstein J, Doran C & Solorio T) 4171– 4186 (Association for Computational Linguistics, 2019).
  47. [60]Ke G He D & Liu TY Rethinking positional encoding in language pretraining. In International Conference on Learning Representations (ICLR, 2019).
  48. [61]Ba JL, Kiros JR & Hinton GE Layer normalization. Preprint at https://arxiv.org/abs/1607.06450 (2016) · arXiv
  49. [62]Shleifer S, Weston J & Ott M, Normformer: Improved transformer pretraining with extra normalization. Preprint at https://arxiv.org/abs/2110.09456 (2021). · arXiv
  50. [63]Dai Z, Liu H, Le QV & Tan M Coatnet: marrying convolution and attention for all data sizes. In Proc. Advances in Neural Information Processing Systems 34 (NeurIPS 2021) 3965– 3977 (Neural Information Processing Systems, 2021).
  51. [64]Wang Z SimVLM: simple visual language model pretraining with weak supervision. In International Conference on Learning Representations. (International Conference on Learning Representations, 2022).
  52. [65]Esser P, Rombach R & Ommer B Taming transformers for high-resolution image synthesis. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 12873– 12883 (Institute of Electrical and Electronics Engineers/Computer Vision Foundation, 2021).
  53. [66]Chen T Pix2seq: a language modeling framework for object detection. In International Conference on Learning Representations (International Conference on Learning Representations, 2022).
  54. [69]Wei J Finetuned language models are zero-shot learners. In International Conference on Learning Representations (International Conference on Learning Representations, 2022).
  55. [70]Schick T & Schütze H It’s not just size that matters: small language models are also few-shot learners. In Proc. 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds. Toutanova K) 2339– 2352 (Association for Computational Linguistics, 2021).
  56. [71]Bao H BEiT: BERT pretraining of image transformers. In International Conference on Learning Representations (International Conference on Learning Representations, 2022).
  57. [73]Sutskever I, Vinyals O & Le QV Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 (Conference on Neural Information Processing Systems, 2014).
  58. [74]Loshchilov I & Hutter F Decoupled weight decay regularization. In International Conference on Learning Representations (International Conference on Learning Representations, 2019).
  59. [75]Micikevicius P Mixed precision training. In International Conference on Learning Representations (International Conference on Learning Representations, 2018).
  60. [76]Raghu M Transfusion: understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems 32 (Conference on Neural Information Processing Systems, 2019).
  61. [77]Zhou C A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT. Preprint at https://arxiv.org/abs/2302.09419 (2023). · arXiv

Authors
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., Huang, J., Chen, C., Zhou, Y., Fu, S., Liu, W., Liu, T., Li, X., Chen, Y., He, L., Zou, J., Li, Q., Liu, H., & Sun, L.
Affiliations
  • Lehigh University (US)
  • University of Georgia (US)
  • Harvard University (US)
  • Children's Hospital of Philadelphia (US)
  • University of Central Florida (US)
  • University of California, Santa Cruz (US)
  • The University of Texas Health Science Center at Houston (US)
  • Mayo Clinic Hospital (US)
  • Leonard Davis Institute of Health Economics (US)
  • Stanford Medicine (US)
Corresponding author
  • Li, Xiang
  • He, Lifang
  • Sun, Lichao
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-08-07
Volume
30
Issue
11
Pages
3129-3141
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
DOI
10.1038/s41591-024-03185-2
PMC
PMC12581140
OpenAlex
W4401397240
Cited by
333
References
65
Citations by year
2026: 130 · 2025: 151 · 2024: 38 · 2023: 14
FWCI
44.66
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (CRII-2246067, POSE: Phase II-2346158, NCS-2319451, MRI-2215789, IIS-1909879, NCS-2319451, MRI-2215789 and IIS-1909879)
  • Lehigh University (FRGS00011497, S00010293, S001250)
  • U.S. Department of Health & Human Services | National Institutes of Health (R21EY034179, R01EB19403, R01LM11934, R03AG078625, R01HL159183, RF1AG057892, R01EB19403 and R01LM11934)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
vision-language foundation models, open-source models, lightweight models, radiology report generation, medical report summarization
MeSH terms
Artificial Intelligence, Humans, Radiology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review