Cited by 74Artificial Intelligence in Healthcare and EducationMultimodal Machine Learning ApplicationsRadiomics and Machine Learning in Medical Imaging

The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: A narrative review

Open access
Simon, Benjamin D., Ozyoruk, Kutsev Bengisu, Gelikman, David G., Harmon, Stephanie A., Türkbey, Barış·2024·Diagnostic and Interventional Radiology

Abstract

With the ongoing revolution of artificial intelligence (AI) in medicine, the impact of AI in radiology is more pronounced than ever. An increasing number of technical and clinical AI-focused studies are published each day. As these tools inevitably affect patient care and physician practices, it is crucial that radiologists become more familiar with the leading strategies and underlying principles of AI. Multimodal AI models can combine both imaging and clinical metadata and are quickly becoming a popular approach that is being integrated into the medical ecosystem. This narrative review covers major concepts of multimodal AI through the lens of recent literature. We discuss emerging frameworks, including graph neural networks, which allow for explicit learning from non-Euclidean relationships, and transformers, which allow for parallel computation that scales, highlighting existing literature and advocating for a focus on emerging architectures. We also identify key pitfalls in current studies, including issues with taxonomy, data scarcity, and bias. By informing radiologists and biomedical AI experts about existing practices and challenges, we hope to guide the next wave of imaging-based multimodal AI research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.4274/dir.2024.242631

Cites

93 of 97 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • No review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.

    “However, to the best of our knowledge, no review has systematically summarized the methodologies and processes specific to the application of multimodal AI in critical care.”

    From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [25]

    The Main points section says existing reviews have generally focused on unimodal imaging AI or general multimodal deep learning, but it discusses reviews in the context of radiology. Our copy does not say whether a review has systematically summarized multimodal AI methodologies and processes specific to critical care; searches for critical care reviews and those methodologies did not establish that claim. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [11]Lyu W Dong X Wong R A Multimodal transformer: fusing clinical notes with structured EHR data for interpretable in-hospital mortality prediction. AMIA Annu Symp Proc 2022 2022 719 728 37128451 PMC10148371
  2. [25]Devlin J Chang MW Lee K Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, Minnesota Association for Computational Linguistics 2019 1 4171 4186
  3. [84]Oquab M Darcet T Moutakanni T DINOv2: learning robust visual features without supervision. 2023 arXiv
  4. [85]Zhou J Wei C Wang H iBOT: image BERT pre-training with Online Tokenizer. 2021 arXiv

Cited by

Sorted by citations

Authors
Simon, B. D., Ozyoruk, K. B., Gelikman, D. G., Harmon, S. A., & Türkbey, B.
Affiliations
National Institutes of Health (US)
Corresponding author
Türkbey, Barış
Venue
Diagnostic and Interventional Radiology
Published
2024-10-02
Volume
31
Issue
4
Pages
303-312
ISSN
  • 1300-4360 (ISSN-L)
  • 1305-3825
  • 1305-3612
OpenAlex
W4403076429
Cited by
73
References
93
Citations by year
2026: 52 · 2025: 21
FWCI
1.67
Type
journal article
Language
en
Open access
Licence
other-oa
Locations
  • Diagnostic and Interventional Radiology (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
multimodal AI, radiology, graph neural networks, transformer models, non-Euclidean data, data scarcity, algorithmic bias, AI taxonomy
MeSH terms
Metadata, Artificial Intelligence, Humans, Radiology, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review