CoChat

Beyond Medical Imaging - A Review of Multimodal Deep Learning in Radiology

Open access

The DOI matches its Crossref record; 84 of the 94 references it cites were found in a registry or an index.

Abstract

Healthcare data are inherently multimodal. Almost all data generated and acquired during a patient’s life can be hypothesized to contain information relevant to providing optimal personalized healthcare. Data sources such as ECGs, doctor’s notes, histopathological and radiological images all contribute to inform a physician’s treatment decision. However, most machine learning methods in healthcare focus on single-modality data. This becomes particularly apparent within the field of radiology, which, due to its information density, accessibility, and computational interpretability, constitutes a central pillar in the healthcare data landscape and traditionally has been one of the key target areas of medically-focused machine learning. Computer-assisted diagnostic systems of the future should be capable of simultaneously processing multimodal data, thereby mimicking physicians, who also consider a multitude of resources when treating patients. Before this background, this review offers a comprehensive assessment of multimodal machine learning methods that combine data from radiology and other medical disciplines. It establishes a modality-based taxonomy, discusses common architectures and design principles, evaluation approaches, challenges, and future directions. This work will enable researchers and clinicians to understand the topography of the domain, describe the state-of-the-art, and detect research gaps for future research in multimodal medical machine learning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.36227/techrxiv.19103432

Cites

84 of 84 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Medical data concerning a patient. Clinical information including imaging and non-imaging data is acquired. Alongside this, non-clinical data could also be obtained based on genome sequencing, user surveys etc.
Fig. 1 Medical data concerning a patient. Clinical information including imaging and non-imaging data is acquired. Alongside this, non-clinical data could also be obtained based on genome sequencing, user surveys etc.CC-BY · From the PDF
Fig. 2 Taxonomy for the methods with a modality combination. Each method is reviewed based on how it uses each modality in a modality combination, (a) fusion of the modality features, (b) learning enriched representations from multiple modalities, or (c) translating one modality to the other.
Fig. 2 Taxonomy for the methods with a modality combination. Each method is reviewed based on how it uses each modality in a modality combination, (a) fusion of the modality features, (b) learning enriched representations from multiple modalities, or (c) translating one modality to the other.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 4

  1. [2]2. D. Shen, G. Wu, H.-I. Suk, Deep learning in medical image analysis, Annual review of biomedical engineering 19 (2017) 221-248.
  2. [20]20. G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely con- nected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
  3. [23]23. M. Endo, R. Krishnan, V. Krishna, A. Y. Ng, P. Rajpurkar, Retrieval- based chest x-ray report generation using a pre-trained contrastive language-image model, in: Machine Learning for Health, PMLR, 2021, pp. 209-219.
  4. [69]69. A. B. Qasim, I. Ezhov, S. Shit, O. Schoppe, J. C. Paetzold, A. Sekuboy- ina, F. Kofler, J. Lipkova, H. Li, B. Menze, Red-gan: Attacking class imbalance via conditioned generation. yet another medical imaging perspective., in: Medical Imaging with Deep Learning, PMLR, 2020, pp. 655-668.

Cited by

See all 13

Metadata

Authors
Heiliger, L., Sekuboyina, A., Menze, B., Egger, J., & Kleesiek, J.
Affiliations
University of Zurich (CH)
Published
2022-02-03
OpenAlex
W4210486480
Cited by
13
References
94
Citations by year
2026: 4 · 2025: 5 · 2024: 4
Type
posted content
Language
en
Open access
Locations
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Universität Zürich
  • Helmut Horten Stiftung
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
multimodal deep learning, radiology, multimodal medical data, computer-aided diagnosis, multimodal architectures, evaluation approaches, multimodal fusion, personalized healthcare, ECG, clinical notes, histopathological images

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.