Cited by 483Artificial Intelligence in Healthcare and EducationMultimodal Machine Learning ApplicationsMachine Learning in Healthcare

Towards Generalist Biomedical AI

Open access
Tu, Tao, Azizi, Shekoofeh, Driess, Danny, Schaekermann, Mike, Amin, Mohamed, Chang, Pi-Chuan, Carroll, Andrew, Lau, Charles +25 more·2024·NEJM AI

Abstract

BackgroundMedicine is inherently multimodal, requiring the simultaneous interpretation and integration of insights between many data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence systems that flexibly encode, integrate, and interpret these data might better enable impactful applications ranging from scientific discovery to care delivery.MethodsTo catalyze development of these models, we curated MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks, such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduced Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. To further probe the capabilities and limitations of Med-PaLM M, we conducted a radiologist evaluation of model-generated (and human) chest x-ray reports.ResultsWe observed encouraging performance across model scales. Med-PaLM M reached performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. In a side-by-side ranking on 246 retrospective chest x-rays, clinicians expressed a pairwise preference for Med-PaLM Multimodal reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility.ConclusionsAlthough considerable work is needed to validate these models in real-world cases and understand if cross-modality generalization is possible, our results represent a milestone toward the development of generalist biomedical artificial intelligence systems. (Funded by Alphabet Inc. and/or a subsidiary thereof.)

Integrity signals

Registry

Crossref · 10.1056/aioa2300138

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generative vision language models such as Med-PaLM M, LLaVA-Med, or PathChat can output text, but only at the patch level for small image regions up to 1024 × 1024 pixels.

    “Generative vision language models such as Med-PaLM M14, LLaVA-Med15, or PathChat16 can output text, but only at the patch level for small image regions up to 1024 × 1024 pixels.”
  • Despite advances with models such as Med-PaLM M or Med-Gemini, the complexities to develop a generalist foundation LLM that truly performs on par with precision medicine tools remain a substantial challenge.

    “Despite advances with models such as Med-PaLM M11 or Med-Gemini12, the complexities to develop a generalist foundation LLM that truly performs on par with precision medicine tools remain a substantial challenge.”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., Palepu, A., Mustafa, B., Chowdhery, A., Liu, Y., Kornblith, S., Fleet, D., Mansfield, P., Prakash, S., Wong, R., Virmani, S., Semturs, C., Mahdavi, S. S., Green, B., Dominowska, E., Arcas, B. A. Y., Barral, J., Webster, D., Corrado, G. S., Matias, Y., Singhal, K., Florence, P., Karthikesalingam, A., & Natarajan, V.
Affiliations
  • Google (United States) (US)
  • Google DeepMind (United Kingdom) (GB)
  • Google DeepMind
  • Google Research
Venue
NEJM AI
Published
2024-02-22
Volume
1
Issue
3
Pages
AIoa2300138-AIoa2300138
ISSN
2836-9386 (ISSN-L)
OpenAlex
W4392044798
Cited by
483
References
27
Citations by year
2026: 152 · 2025: 193 · 2024: 125 · 2023: 12
FWCI
44.03
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • NEJM AI (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
DeepMind
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
medical question answering, radiology report generation, chest X-ray reports, mammography interpretation, variant calling, cross-modality generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review