The DOI matches its Crossref record; 211 of the 211 references it cites were found in a registry or an index.
Multimodal data integration for oncology in the era of deep neural networks: a review
Open accessAbstract
Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Siegel, R. L., Miller, K. D., Wagle, N. S., & Jemal, A. (2023) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21763 · Cited by 16,303
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Research works printed but not linked: 5
- Flamingo: a visual language model for few-shot learning 23716 23736 Alayrac J.-B. Donahue J. Luc P. Miech A. Barr I. Hasson Y. Adv. Neural Inf. Process. Syst 35 2022
- Convolutional neural networks on graphs with fast localized spectral filtering 3844 3852 Defferrard M. Bresson X. Vandergheynst P. Adv. Neural Inf. Process. Syst 29 2016
- Jaegle, 2021, Perceiver IO: A General Architecture for Structured Inputs
- Ma, 2019, “NeuGraph: parallel deep neural network computation on large graphs,”, 443
- Remmer, 2022, Explainability Methods for Transformer-based Artificial Neural Networks: a Comparative Analysis
Cited by
Vision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 229
Self-citationDeep learning–driven multi-omics analysis: enhancing cancer diagnostics and therapeutics
Zhang, J., Che, Y., Liu, R., Wang, Z., & Liu, W. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf440 · Cited by 100
Artificial Intelligence in Head and Neck Cancer: Innovations, Applications, and Future Directions
Pham, T. D., Teh, M., Chatzopoulou, D., Holmes, S., & Coulthard, P. (2024) · Current Oncology · doi:10.3390/curroncol31090389 · Cited by 76
A Comprehensive Review of Multimodal Deep Learning for Enhanced Medical Diagnostics
Al‐Zoghby, A. M., Ebada, A. I., Saleh, A. S., Abdelhay, M., & Awad, W. A. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.065571 · Cited by 60
Building Flexible, Scalable, and Machine Learning-Ready Multimodal Oncology Datasets
Tripathi, A. G., Waqas, A., Venkatesan, K., Yılmaz, Y., & Rasool, G. (2024) · Sensors · doi:10.3390/s24051634 · Cited by 48
Self-citation
Metadata
- Authors
- Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G.
- Affiliations
- Moffitt Cancer Center (US)
- Rowan University (US)
- Corresponding author
- Waqas, Asim
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2024-07-25
- Volume
- 7
- Pages
- 1408843-1408843
- ISSN
- 2624-8212 (ISSN-L)
- PMC
- PMC11308435
- OpenAlex
- W4400981456
- Cited by
- 165
- References
- 206
- Citations by year
- 2026: 81 · 2025: 70 · 2024: 12
- FWCI
- 5.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Artificial Intelligence (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Rowan Digitals Works (Rowan University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1903466, 2008690, 2234468, 2234836)
- Moffitt Cancer Center (P30CA076292, CA076292)
- National Cancer Institute (CA076292, P30CA076292)
- Division of Electrical, Communications and Cyber Systems (ECCS-1903466)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- multimodal data integration, oncology, graph neural networks, transformer models, multimodal fusion, precision oncology, cancer diagnosis, cancer screening, histopathology, molecular data, data heterogeneity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal data integration for oncology in the era of deep neural networks: a review. https://app.cochat.ai/papers/10.3389/frai.2024.1408843
- The paper
- Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024). Multimodal data integration for oncology in the era of deep neural networks: a review. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2024.1408843
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.