Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 211 of the 211 references it cites were found in a registry or an index.

Multimodal data integration for oncology in the era of deep neural networks: a review

Open access
Waqas, Asim, Tripathi, Aakash, Ramachandran, Ravi P., Stewart, Paul A., Rasool, Ghulam·2024·Frontiers in Artificial Intelligence

Abstract

Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/frai.2024.1408843

Cites

211 of 211 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Number of publications involving DL, GNNs, GNNs in the medical domain, overall multimodal and multimodal in biomedical and clinical sciences in the period 2015–2024 (Hook et al., 2018).
Figure 1 Number of publications involving DL, GNNs, GNNs in the medical domain, overall multimodal and multimodal in biomedical and clinical sciences in the period 2015–2024 (Hook et al., 2018).CC BY · PubMed Central
Figure 2 We present various data modalities that capture specific aspects of cancer at different scales. For example, radiological images capture organ or sub-organ level abnormalities, while tissue analysis may provide changes in the cellular structure and morphology. On the other hand, various molecular data types may provide insights into genetic mutations and epigenetic changes. (A) An overview of data collected from population to a tissue. (B) Detailed look into data modalities acquired for cancer care.
Figure 2 We present various data modalities that capture specific aspects of cancer at different scales. For example, radiological images capture organ or sub-organ level abnormalities, while tissue analysis may provide changes in the cellular structure and morphology. On the other hand, various molecular data types may provide insights into genetic mutations and epigenetic changes. (A) An overview of data collected from population to a tissue. (B) Detailed look into data modalities acquired for cancer care.CC BY · PubMed Central
Figure 3 Taxonomy, stages, and techniques of multimodal data fusion are presented. Early, late, cross-modality fusion methods integrate individual data modalities (or extracted features) before, after, or at the primary learning step, respectively.
Figure 3 Taxonomy, stages, and techniques of multimodal data fusion are presented. Early, late, cross-modality fusion methods integrate individual data modalities (or extracted features) before, after, or at the primary learning step, respectively.CC BY · PubMed Central
Figure 4 (A) The commonly occurring graph types are presented, including (1) undirected and directed, (2) homogeneous and heterogeneous, (3) dynamic and static, (4) attributed (edges) and unattributed. (B) Three different types of tasks performed using the graph data are presented and include (1) node-level, (2) link-level, and (3) graph-level analyses. (C) Various categories of representation learning for graphs are presented.
Figure 4 (A) The commonly occurring graph types are presented, including (1) undirected and directed, (2) homogeneous and heterogeneous, (3) dynamic and static, (4) attributed (edges) and unattributed. (B) Three different types of tasks performed using the graph data are presented and include (1) node-level, (2) link-level, and (3) graph-level analyses. (C) Various categories of representation learning for graphs are presented.CC BY · PubMed Central
Figure 5 Convolution operation for graphs vs. image data. The canonical order of the input is important in CNNs, whereas in GNNs, the order of the input nodes is not important. From the convolution operation perspective, CNNs can be considered a subset of GNNs (Hamilton, 2020).
Figure 5 Convolution operation for graphs vs. image data. The canonical order of the input is important in CNNs, whereas in GNNs, the order of the input nodes is not important. From the convolution operation perspective, CNNs can be considered a subset of GNNs (Hamilton, 2020).CC BY · PubMed Central
Figure 6 (A) Data processing pipeline for histopathology images using GNNs (Chen et al., 2020). (B) Graph processing pipeline on radiology data. Adapted from Singh et al. (2021).
Figure 6 (A) Data processing pipeline for histopathology images using GNNs (Chen et al., 2020). (B) Graph processing pipeline on radiology data. Adapted from Singh et al. (2021).CC BY · PubMed Central
Figure 7 Graph data processing pipeline for non-imagery data, including molecular and textual data. Adapted from Wang T. et al. (2021). GNN, graph neural network, FC, Fully-Connected; MLP, Multi-Layer Perception.
Figure 7 Graph data processing pipeline for non-imagery data, including molecular and textual data. Adapted from Wang T. et al. (2021). GNN, graph neural network, FC, Fully-Connected; MLP, Multi-Layer Perception.CC BY · PubMed Central
Figure 8 The original Transformer architecture is presented (Vaswani et al., 2017). A Transformer can have multiple encoder and decoder blocks, as well as some additional layers.
Figure 8 The original Transformer architecture is presented (Vaswani et al., 2017). A Transformer can have multiple encoder and decoder blocks, as well as some additional layers.CC BY · PubMed Central
Figure 9 Four different strategies of fusing information from various data modalities in multimodal Transformers are presented. (A) Early Fusion. (B) late Fusion. (C) Hierarchial Attention. (D) Cross Attention.
Figure 9 Four different strategies of fusing information from various data modalities in multimodal Transformers are presented. (A) Early Fusion. (B) late Fusion. (C) Hierarchial Attention. (D) Cross Attention.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 206 references

Research works printed but not linked: 5

  1. Flamingo: a visual language model for few-shot learning 23716 23736 Alayrac J.-B. Donahue J. Luc P. Miech A. Barr I. Hasson Y. Adv. Neural Inf. Process. Syst 35 2022
  2. Convolutional neural networks on graphs with fast localized spectral filtering 3844 3852 Defferrard M. Bresson X. Vandergheynst P. Adv. Neural Inf. Process. Syst 29 2016
  3. Jaegle, 2021, Perceiver IO: A General Architecture for Structured Inputs
  4. Ma, 2019, “NeuGraph: parallel deep neural network computation on large graphs,”, 443
  5. Remmer, 2022, Explainability Methods for Transformer-based Artificial Neural Networks: a Comparative Analysis

Cited by

See all 165

Metadata

Authors
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G.
Affiliations
  • Moffitt Cancer Center (US)
  • Rowan University (US)
Corresponding author
Waqas, Asim
Venue
Frontiers in Artificial Intelligence
Published
2024-07-25
Volume
7
Pages
1408843-1408843
ISSN
2624-8212 (ISSN-L)
OpenAlex
W4400981456
Cited by
165
References
206
Citations by year
2026: 81 · 2025: 70 · 2024: 12
FWCI
5.72
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Artificial Intelligence (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Rowan Digitals Works (Rowan University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1903466, 2008690, 2234468, 2234836)
  • Moffitt Cancer Center (P30CA076292, CA076292)
  • National Cancer Institute (CA076292, P30CA076292)
  • Division of Electrical, Communications and Cyber Systems (ECCS-1903466)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
multimodal data integration, oncology, graph neural networks, transformer models, multimodal fusion, precision oncology, cancer diagnosis, cancer screening, histopathology, molecular data, data heterogeneity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal data integration for oncology in the era of deep neural networks: a review. https://app.cochat.ai/papers/10.3389/frai.2024.1408843
The paper
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024). Multimodal data integration for oncology in the era of deep neural networks: a review. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2024.1408843

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.