Multimodal learning with graphs
Open accessAbstract
Artificial intelligence for graphs has achieved remarkable success in modeling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases-the set of assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets presents fundamental challenges because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, multimodal graph AI methods combine different modalities while leveraging cross-modal dependencies using graphs. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The all-in-one multimodal graph learning (MGL) architecture introduced a unified framework that included entity identification, topology construction, information propagation, and representation mixing for joint learning of modalities such as images, language sequences, or biological systems.
“The “all-in-one” multimodal graph learning (MGL) architecture [67] introduced a unified framework that included entity identification, topology construction, information propagation, and representation mixing for joint learning of modalities such as images, language sequences, or biological systems.”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [67]
For adapting GNNs into multimodal usage, data points from different modalities are all well-organized in defined graphs, and GNNs can directly learn interactions of intra- and inter-modalities simultaneously.
“For adapting GNNs into a multimodal usage [67], data points from different modalities are all well-organized in defined graphs, and in contrast to single-modal fusion architecture, GNNs can directly learn the interactions of intra- and inter-modalities simultaneously.”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
MoleculeNet: a benchmark for molecular machine learning
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018) · Chemical Science · doi:10.1039/c7sc02664a · Cited by 3,107 · see citing works
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · see citing works
A graph-convolutional neural network model for the prediction of chemical reactivity
Coley, C. W., Jin, W., Rogers, L., Jamison, T. F., Jaakkola, T. S., Green, W. H., Barzilay, R., & Jensen, K. F. (2019) · Chemical Science · doi:10.1039/c8sc04228d · Cited by 679 · see citing works
Graph Convolutional Neural Networks for Predicting Drug-Target Interactions
Torng, W., & Altman, R. B. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00628 · Cited by 371 · see citing works
Multi-Modal Graph Learning for Disease Prediction
Zheng, S., Zhu, Z., Liu, Z., Guo, Z., Liu, Y., Yang, Y., & Zhao, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3159264 · Cited by 196 · see citing works
Visible Machine Learning for Biomedicine
Yu, M. K., Ma, J., Fisher, J., Kreisberg, J. F., Raphael, B. J., & Ideker, T. (2018) · Cell · doi:10.1016/j.cell.2018.05.056 · Cited by 162 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- [1][1] Kim, J., Kim, T., Kim, S. & Yoo, C. D. Edge-labeling graph neural network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019). -
- [2][2] Zhang, M. & Chen, Y. Link prediction based on graph neural networks. In Bengio, S. et al. (eds.) Advances in Neural Information Processing Systems , vol. 31 (Curran Associates, Inc., 2018). -
- [3][3] Beani, D. et al. Directional graph networks. In Meila, M. & Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning , vol. 139 of Proceedings of Machine Learning Research , 748-758 (PMLR, 2021). -
- [4][4] Bodnar, C. et al. Weisfeiler and lehman go cellular: CW networks. In Thirty-Fifth Conference on Neural Information Processing Systems (2021). -
- [5][5] Alsentzer, E., Finlayson, S. G., Li, M. M. & Zitnik, M. Subgraph neural networks. Advances in Neural Information Processing Systems (2020). -
- [6][6] Vinyals, O., Bengio, S. & Kudlur, M. Order matters: Sequence to sequence for sets. In International Conference on Learning Representations (ICLR) (2016). -
- [7][7] Corso, G., Cavalleri, L., Beaini, D., Liò, P. & Veličković, P. Principal neighbourhood aggregation for graph nets. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H. (eds.) Advances in Neural Information Processing Systems , vol. 33, 13260-13271 (Curran Associates, Inc., 2020). -
- [8][8] Defferrard, M., Bresson, X. & Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I. & Garnett, R. (eds.) Advances in Neural Information Processing Systems , vol. 29 (Curran Associates, Inc., 2016). -
- [9][9] Wu, F. et al. Simplifying graph convolutional networks. In Chaudhuri, K. & Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning , vol. 97 of Proceedings of Machine Learning Research , 6861-6871 (PMLR, 2019). -
- [10][10] Levie, R., Monti, F., Bresson, X. & Bronstein, M. M. Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing 67 , 97-109 (2019). -
- [11][11] Veličković, P. et al. Graph attention networks. In International Conference on Learning Representations (2018). -
- [12][12] Kreuzer, D., Beaini, D., Hamilton, W. L., Létourneau, V. & Tossou, P. Rethinking graph transformers with spectral attention. In Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W. (eds.) Advances in Neural Information Processing Systems (2021). -
- [13][13] Ying, C. et al. Do transformers really perform badly for graph representation? In Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W. (eds.) Advances in Neural Information Processing Systems (2021). -
- [14][14] Rampasek, L. et al. Recipe for a general, powerful, scalable graph transformer. In Oh, A. H., Agarwal, A., Belgrave, D. & Cho, K. (eds.) Advances in Neural Information Processing Systems (2022). -
- [15][15] Hong, D. et al. Graph convolutional networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 59 , 5966-5978 (2021). -
- [16][16] Lu, Y., Chen, Y., Zhao, D. & Chen, J. Graph-fcn for image semantic segmentation. CoRR (2020). 2001.00335 . -
- [17][17] Yao, L., Mao, C. & Luo, Y. Graph convolutional networks for text classification. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'19/IAAI'19/EAAI'19 (AAAI Press, 2019).
Cited by
Sorted by citations
Multimodal data integration for oncology in the era of deep neural networks: a review
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1408843 · Cited by 165 · see citing works
“Molecular data modalities provide information about the underlying genetic changes and alterations in the cancer cells (Liu et al., 2021).”
BackgroundBayoudh, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 173 · see citing works
Attention-based graph neural networks: a survey
Sun, C., Li, C., Lin, X., Zheng, T., Meng, F., Rui, X., & Wang, Z. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10577-2 · Cited by 146 · see citing works
“Once complex relations between modalities can be built into a network structure, GNNs provide a powerful and flexible strategy to leverage interdependencies in multimodal datasets (Ektefaie et al. 2023).”
BackgroundGraph machine learning for integrated multi-omics analysis
Valous, N. A., Popp, F., Zörnig, I., Jäger, D., & Charoentong, P. (2024) · British Journal of Cancer · doi:10.1038/s41416-024-02706-7 · Cited by 144 · see citing works
“In contrast, graph machine learning can model such datasets by connecting different modalities in optimally de fi ned (but more realistically in context-de fi ned) graphs, and by building learning systems for a wide range of tasks [10].”
BackgroundCurrent and future directions in network biology
Žitnik, M., Li, M. M., Wells, A., Glass, K., Gysi, D. M., Krishnan, A., Murali, T. M., Predrag Radivojac, A., Roy, S., Baudot, A., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae099 · Cited by 118 · see citing works
“Network methods can integrate multimodal data to identify patient groups with coherent genetic, genomic, physiological, and clinical profiles [92, 128, 326], even when the underlying data are incomplete and noisy [311].”
BackgroundSelf-citationChatGPT, GPT-4, and Other Large Language Models: The Next Revolution for Clinical Microbiology?
Egli, A. (2023) · Clinical Infectious Diseases · doi:10.1093/cid/ciad407 · Cited by 109 · see citing works
“This holistic view of a dataset allows more robust predictions and decision-making [19].”
BackgroundCao, D., Chen, M., Zhang, R., Wang, Z., Huang, M., Yu, J., Jiang, X., Fan, Z., Zhang, W., Zhou, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02516-y · Cited by 106 · see citing works
Digital twins as global learning health and disease models for preventive and personalized medicine
Li, X., Loscalzo, J., Mahmud, A. K. M. F., Aly, D. M., Rzhetsky, A., Žitnik, M., & Benson, M. (2025) · Genome Medicine · doi:10.1186/s13073-025-01435-7 · Cited by 124 · see citing works
Self-citationNetworking Systems for Video Anomaly Detection: A Tutorial and Survey
Liu, J., Liu, Y., Lin, J., Li, J., Cao, L., Sun, P., Hu, B., Song, L., Boukerche, A., & Leung, V. C. M. (2025) · ACM Computing Surveys · doi:10.1145/3729222 · Cited by 90 · see citing works
Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 76 · see citing works
“SDPRL is build and tested using the benchmark dataset MIMIC-III.”
MethodContextual AI models for single-cell protein biology
Li, M. M., Huang, Y., Sumathipala, M. G., Liang, M. Q., Valdeolivas, A., Ananthakrishnan, A. N., Liao, K. P., Marbach, D., & Žitnik, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02341-3 · Cited by 77 · see citing works
“Through attention-based deep learning 28,29 , which specifies models that can pay attention to large inputs and learn the most important elements to focus on in each context, single-cell atlases can be leveraged to boost the mapping of gene regulatory networks that drive disease progression and…”
Self-citationMultimodal graph learning based on 3D Haar semi-tight framelet for student engagement prediction
Li, M., Zhuang, X., Bai, L., & Ding, W.-P. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102224 · Cited by 77 · see citing works
UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs
He, Y.-F., Sui, Y., He, X.-X., Liu, Y., Sun, Y., & Hooi, B. (2025) · The Web Conference · doi:10.1145/3696410.3714818 · Cited by 72 · see citing works
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75 · see citing works
“The “all-in-one” multimodal graph learning (MGL) architecture [67] introduced a unified framework that included entity identification, topology construction, information propagation, and representation mixing for joint learning of modalities such as images, language sequences, or biological systems.”
Graph Neural Networks in Histopathology: Emerging Trends and Future Directions
Brussee, S., Buzzanca, G., Schrader, A., & Kers, J. (2024) · Medical Image Anal. · doi:10.48550/arxiv.2406.12808 · Cited by 65
“This approach is efficient, as no additional model modules are required and allows quick injection or removal of information from different modalities [40] [41].”
Zhang, C., Tao, M., Chen, W., & Fan, J. (2023) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13098 · Cited by 51 · see citing works
“In recent years, graphs have proven to be advantageous in modeling unstructured data (Ektefaie et al., 2023), and deep learning on graphs using graph neural networks (GNNs) has contributed to significant advancements in various fields such as biology (Greener et al., 2022), chemistry (Wu et al.,…”
BackgroundGraph Neural Networks for building and civil infrastructure operation and maintenance enhancement
Wettewa, S., Hou, L., & Zhang, G. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102868 · Cited by 56 · see citing works
Exploring Causal Learning Through Graph Neural Networks: An In‐Depth Review
Job, S., Tao, X., Cai, T., Xie, H., Li, L., Li, Q., & Yong, J. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70024 · Cited by 25 · see citing works
MuSe-GNN: Learning Unified Gene Representation From Multimodal Biological Graph DataPreprint
Liu, T.-Y., Wang, Y., Ying, R., & Zhao, H. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.02275 · Cited by 45 · see citing works
“To the best of our knowledge, this is the first paper in gene representation learning that combines the Multimodal Machine Learning (MMML) concept [9, 27] with deep GNNs [49, 81] design.”
BackgroundPerspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies.
Liu, W., Chen, J.-W., Wang, H.-B., Fu, Z.-Q., Peijnenburg, W., & Hong, H.-X. (2024) · Environmental Science and Technology · doi:10.1021/acs.est.4c03088 · Cited by 44 · see citing works
Multimodal Artificial Synapses for Neuromorphic Application
Li, R., Yue, Z., Luan, H., Dong, Y., Chen, X., & Gu, M. (2024) · Research · doi:10.34133/research.0427 · Cited by 44 · see citing works
Building Flexible, Scalable, and Machine Learning-Ready Multimodal Oncology Datasets
Tripathi, A., Waqas, A., Venkatesan, K., Yilmaz, Y., & Rasool, G. (2023) · Italian National Conference on Sensors · doi:10.3390/s24051634 · Cited by 42 · see citing works
Learning on Multimodal Graphs: A SurveyPreprint
Peng, C.-Y., He, J.-Y., & Xia, F. (2024) · arXiv.org · doi:10.48550/arxiv.2402.05322 · Cited by 41 · see citing works
“To the best of our knowledge, there is only one survey paper on multimodal graph learning [ Ektefaie et al. , 2023 ] , which mainly discusses the potential of leveraging the characteristics of graphs to achieve multimodal learning.”
BackgroundTowards verifiable cancer digital twins: tissue level modeling protocol for precision medicine
Kemkar, S., Tao, M., Ghosh, A., Stamatakos, G., Graf, N., Poorey, K., Balakrishnan, U., Trask, N., & Radhakrishnan, R. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1473125 · Cited by 40 · see citing works
“The recent advancements in multi-modal (Baltrusaitis et al., 2019; Stahlschmidt et al., 2022; Ektefaie et al., 2023) and physics-informed (Cuomo et al., 2022; Jabbari Zideh et al., 2023) machine learning have significantly enhanced the ability to analyze complex, multimodal scientific datasets…”
BackgroundWu, M., Zheng, X., Zhang, Q., Shen, X., Luo, X., Zhu, X., & Pan, S. (2024) · arXiv.org · doi:10.48550/arxiv.2402.16374 · Cited by 39
“Data types in the real world exhibit various modalities and scales, e.g., images, texts, tabular data, audios, and graphs, which are crucial for the advancement of artificial intelligence and machine learning communities [114].”
BackgroundChen, L.-T., Wang, B., Zhang, W.-T., Zheng, S., Chen, Z.-F., Zhang, M.-Z., Dong, C., Pan, F., & Li, S.-N. (2024) · Journal of the American Chemical Society · doi:10.1021/jacs.3c11852 · Cited by 38 · see citing works
Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning
Zhu, J., Zhou, Y., Qian, S., He, Z., Zhao, T., Shah, N., & Koutra, D. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01326 · Cited by 12 · see citing works
“Multimodal graph learning (MMGL) aims to address this challenge by leveraging graph structure to learn from multimodal data with complicated relations among multiple multimodal neighbors [7, 8, 14, 48].”
MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Prediction
Cai, L., Zeng, W.-M., Chen, H., Zhang, H., Li, Y.-Y., Feng, Y., Yan, H.-J., Bian, L., Siok, W., & Wang, N.-Z. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3556420 · Cited by 31 · see citing works
“1) consists of three stages: • Modality-Rewarding Representation Learning.”
Lv, B., Wu, H., Chen, W.-B., Chen, C., Miao, Y.-S., & Zhao, C. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109398 · Cited by 26 · see citing works
HKANLP: Link Prediction With Hyperspherical Embeddings and Kolmogorov–Arnold Networks
Zhang, W., Fan, W., Su, W., & Bouguila, N. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3614341 · Cited by 24 · see citing works
“LP, which involves predicting the existence or likelihood of connections between node pairs, is essential in many applications, including recommendation systems [9], drug discovery [10], anomaly detection [11], and knowledge graph construction [12].”
Rethinking industrial artificial intelligence: a unified foundation framework
Lee, J., & Su, H.-Q. (2025) · International Journal of AI for Materials and Design · doi:10.36922/ijamd025080006 · Cited by 25
“Additionally, multi-modal learning approaches, which combine information from multiple modalities — such as images, text, audio, tabular data, and sensor measurements — enable the development of more comprehensive models suitable for complex industrial scenarios [13, 53].”
Zhou, C., Wang, M.-X., Ma, Y.-B., Wu, C., Chen, W., Qian, Z., Liu, X., Zhang, Y.-W., Wang, J.-H., Xu, H. E., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.25373 · Cited by 23
“This change compels models to reason exclusively from visual elements, such as bar heights and line slopes, in a manner akin to human interpretation [288].”
Qin, Y.-Y., Chang, J.-M., Li, L., & Wu, M.-H. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1583514 · Cited by 21 · see citing works
“This personalized approach not only enhances patient comprehension but also increases overall satisfaction with medical procedures (25).”
Fusion Modeling; Combining Clinical and Imaging Data to Advance Cardiac Care
van Assen, M., Tariq, A., Razavi, A. C., Yang, C., Banerjee, I., & Cecco, C. D. D. (2023) · Circulation Cardiovascular Imaging · doi:10.1161/circimaging.122.014533 · Cited by 21 · see citing works
Target identification and assessment in the era of AI
Pun, F. W., Podolskiy, D. I., Izumchenko, E., Mortlock, A. A., Oprea, T. I., Scheibye‐Knudsen, M., Fortney, K., Morgen, E. K., Ren, F., & Zhavoronkov, A. A. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01412-8 · Cited by 28 · see citing works
“Convolutional neural networks trained on large annotated datasets, such as that from the JUMP-Cell Painting Consortium 54 , can automatically learn features and patterns that distinguish various cellular components, facilitating object detection and recognition in complex biological images 55,56 .”
BioBLP: a modular framework for learning on multimodal biomedical knowledge graphs
Daza, D., Alivanistos, D., Mitra, P., Pijnenburg, T., Cochez, M., & Groth, P. (2023) · Journal of Biomedical Semantics · doi:10.1186/s13326-023-00301-y · Cited by 18 · see citing works
“Instead of directly mapping entities in a KG to an embedding, some methods have proposed embedding functions that take as input the attributes of an entity [21, 23, 25–31].”
Disentangling Homophily and Heterophily in Multimodal Graph Clustering
Guo, Z., Shen, Z.-X., Xie, X., Wen, L.-J., & Kang, Z. (2025) · ACM Multimedia · doi:10.1145/3746027.3755631 · Cited by 19
“Multimodal graphs—integrating unstructured heterogeneous modalities such as text, images, and audio with structured topological connections—have found widespread applications across diverse domains, including social network analysis [43], recommendation systems [38], and bioinformatics [20] [5, 32].”
Graph World ModelPreprint
Feng, T., Wu, Y., Lin, G., & You, J.-X. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2507.10539 · Cited by 19 · see citing works
“However, existing world models do not directly generalize to structured data, primarily graphs, that are ubiquitous in science (Jin et al., 2018; You et al., 2018) and industry (Ying et al., 2018; You et al., 2022) and can be further enriched with multi-modal information (Ektefaie et al., 2023).”
Sensing in Smart Cities: A Multimodal Machine Learning Perspective
Sadiq, T., & Omlin, C. W. P. (2025) · Smart Cities · doi:10.3390/smartcities9010003 · Cited by 16 · see citing works
“Graph neural networks [27] help show organized links like traffic patterns or sensor networks from different methods.”
InfluentialRuan, L., & Jiang, H. (2025) · Mathematics · doi:10.3390/math13172747 · Cited by 18 · see citing works
Enhancing foundation models for scientific discovery via multimodal knowledge graph representations
López, V., Hoang, L. T., Galindo, M. M., Fernández-Díaz, R., Sbodio, M. L., Ordóñez-Hurtado, R., Zayats, M., Mulligan, N., & Bettencourt‐Silva, J. H. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100845 · Cited by 22 · see citing works
Integrated visual transformer and flash attention for lip-to-speech generation GAN
Yang, Q., Bai, Y., Liu, F., & Zhang, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-55248-6 · Cited by 18 · see citing works
“Introduced the flash attention mechanism to reduce the computational burden from multiple interactions of the hierarchical iterative generator, thereby enhancing the model’s performance 7–10 .”
BackgroundTowards deployment-centric multimodal AI beyond vision and language
Liu, X., Zhang, J., Zhou, S., van der Plas, T. L., Vijayaraghavan, A., Grishina, A., Zhuang, M., Schofield, D., Tomlinson, C., Wang, Y., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01116-5 · Cited by 13 · see citing works
Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning
Vishwakarma, R., & Rezaei, A. (2024) · Design, Automation and Test in Europe · doi:10.48550/arxiv.2401.09479 · Cited by 17
“In our work, we target the fusion of graph [23], [24] and Euclidean data as the modalities of interest along with uncertainty estimation.”
MethodCross-Contrastive Clustering for Multimodal Attributed Graphs with Dual Graph Filtering
Zheng, H., Yang, R., Wang, H., & Xu, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3780236
Towards Modality Generalization: A Benchmark and Prospective Analysis
Liu, X., Xia, X., Huang, Z., Ng, S.-K., & Chua, T. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758174 · Cited by 3 · see citing works
Niu, Q., Chen, K.-Y., Li, M., Feng, P.-H., Bi, Z.-Q., Yan, L. K., Zhang, Y.-C., Yin, C. H., Fei, C., Liu, J.-Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.01812 · Cited by 16
Wang, Y.-F., Li, Y., Liu, L., Hong, P., & Xu, H. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00430 · Cited by 16 · see citing works
Structure-Based Drug Design with Geometric Deep Learning: A Comprehensive Survey
Zhang, Z.-X., Yan, J.-X., Huang, Y., Liu, Q., Chen, E.-H., Wang, M.-D., & Zitnik, M. (2025) · ACM Computing Surveys · doi:10.1145/3769677 · Cited by 15 · see citing works
Self-citationHerbal ingredient-target interaction prediction via multi-modal learning
Liang, X.-D., Lai, G.-C., Yu, J., Lin, T., Wang, C.-C., & Wang, W. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122115 · Cited by 15 · see citing works
Multi-Modal Generative AI: Multi-Modal LLMs, Diffusions, and the Unification
Wang, X., Zhou, Y., Huang, B., Chen, H., & Zhu, W.-W. (2024) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2025.3635224 · Cited by 15
“Ektefaie et al. [204] explore the combination of multiple data modalities via cross-modal dependencies and geometric relationships to develop multi-modal architectures, e.g., image-intensive, knowledge-grounded, and language-intensive models, in order to process diverse datasets.”
Deep Learning of radiology-genomics integration for computational oncology: A mini review
Wang, F., Li, Y., & Zeng, T. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.06.019 · Cited by 21 · see citing works
“They capture complex dependencies and relationships within this data using layers that iteratively update the representation of each node based on its neighbors [29].”
BackgroundResearch Progress in Artificial Intelligence-Assisted Preparation of High-Quality Biomaterials
Wei, D., Wang, Z., Lin, H., Yin, X.-P., & Wang, Y. (2026) · ACS Omega · doi:10.1021/acsomega.5c09392 · Cited by 14 · see citing works
MaRAG: a knowledge graph and retrieval-augmented framework for maritime accident analysis
Zou, Y., Wang, S., Li, G., & Ning, B. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.123622 · Cited by 13 · see citing works
Zhou, Z.-Q., & Wang, J.-W. (2025) · Rambam Maimonides Medical Journal · doi:10.5041/rmmj.10558 · Cited by 12
Engineering artificial intelligence: framework, challenges, and future direction
Lee, J., Su, H., Ji, D.-Y., & Minami, T. (2025) · Machine Learning Engineering · doi:10.1088/3049-4761/adce0d · Cited by 18 · see citing works
“Multi-Modal Machine Learning : aims to combine information from multiple modalities, such as images, text, audio, tabular data, or sensor signals, to create more robust and comprehensive models [18, 57, 54].”
A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis
Peng, J., He, Y., Chang, Y., Lu, Y., Zhang, P., Ou, Z., & Yu, Q. (2025) · Applied Sciences · doi:10.3390/app15020636 · Cited by 11 · see citing works
Towards multimodal graph neural networks for surgical instrument anticipation
Wagner, L., Schneider, D. N., Mayer, L. D., Jell, A., Müller, C., Lenz, A., Knoll, A., & Wilhelm, D. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03226-8 · Cited by 12 · see citing works
On knowing a gene: A distributional hypothesis of gene function
Kwon, J. J., Pan, J., Gonzalez, G., Hahn, W. C., & Žitnik, M. (2024) · Cell Systems · doi:10.1016/j.cels.2024.04.008 · Cited by 12 · see citing works
“86–90 This multimodal approach addresses the limitations of singular data type analyses and highlights how additional datatypes offer a greater comprehensive understanding of gene function across various biological contexts.”
BackgroundSelf-citationYe, T., Cao, H., Liu, J., Pang, B., Xu, Q.-Y., Song, Y., & Yuan, X.-F. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3541337 · Cited by 11
“Different types of nodes represent different sensor modalities, and this heterogeneity invalidates existing approaches that assume homogeneity of node characteristics [28].”
Zhao, Y., Li, L., Yu, X., Han, K., Duan, J., Liang, D., Chai, N., & Li, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107607 · Cited by 12 · see citing works
Carbone, M. R., Maffettone, P. M., Qu, X., Yoo, S., & Lu, D. (2024) · The Journal of Physical Chemistry A · doi:10.1021/acs.jpca.3c06910 · Cited by 12 · see citing works
van Rijswijk, R. V., Bogdanovic, M., Roy, J., Yeung, K., Zeebregts, C., Geelkerken, R. H., Jebbink, E. G., Wolterink, J., & Reijnen, M. (2025) · Journal of Endovascular Therapy · doi:10.1177/15266028251314359 · Cited by 10 · see citing works
FDFR-Net: A fruit ripeness classification method using multimodal learning technique
Wang, Y., Jin, X., Huang, W., Wang, X., & Yu, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130126 · Cited by 9 · see citing works
Shen, H., Cao, H., Wang, L., Ai, Z., & Dai, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114170 · Cited by 8 · see citing works
Xie, X.-D., Wu, J., Xiang, M., Tang, J.-T., & Sheng, Y.-X. (2025) · Journal of King Saud University: Computer and Information Sciences · doi:10.1007/s44443-025-00185-1 · Cited by 9 · see citing works
“Graph neural networks (GNNs) introduce relational inductive biases that are well suited to multimodal data with explicit or latent structures (Ektefaie et al. 2023; Li et al. 2023a; Zhang et al. 2024).”
Wang, Y., Lei, Y.-G., Li, N.-P., Feng, K., Wang, Z.-D., Tan, Y., & Li, H. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3573703 · Cited by 9 · see citing works
“Unimodal data provide only a limited and potentially biased perspective [8], making it highly likely that critical aspects of a machine’s health state are overlooked and that the machinery degradation process is inadequately represented.”
A Comprehensive Data-centric Overview of Federated Graph Learning
Wu, Z.-Y., Li, X.-K., Zhu, Y.-L., Chen, Z.-K., Yan, G.-C., Yan, Y., Zhang, H., Ai, Y., Jin, X., Li, R.-H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.16541 · Cited by 9
“[215] Within the context of FGL, integrating MGL is increasingly demanded by real-world scenarios.”
A System to Ensure Information Trustworthiness in Artificial Intelligence Enhanced Higher Education
Khan, U. A., Kauttonen, J., Aunimo, L., & Alamäki, A. V. (2024) · Journal of Information Technology Education Research · doi:10.28945/5295 · Cited by 12 · see citing works
“Recent breakthroughs in the construction and utilization of multimodal knowledge graphs (Chen et al., 2023; Ektefaie et al., 2023) lay the foundation for the EMIAS repository, which acts as a comprehensive system for evaluating information, bridging the connections between various information…”
BackgroundIntegrative multimodal hybrid data fusion for mortality prediction
Abuhamad, H., Zainudin, S., & Bakar, A. A. (2026) · Scientific Reports · doi:10.1038/s41598-026-36296-6 · Cited by 8 · see citing works
“In the work by Ektefaie et al. 16 , the study investigates the integration of different data modalities with graph-based techniques for learning and inference over complex structures.”
Li, X.-Y., Sun, C.-C., Huang, W.-R., Wang, Y.-J., & Ma, B.-Y. (2025) · Briefings Bioinform. · doi:10.1093/bib/bbaf496 · Cited by 8 · see citing works
“Second, we introduced a bi-level GNN framework for more detailed representation of TCR-pMHC interface 3D structures [26].”
Yang, J., Chen, C., Jia, F.-L., Xie, X., Fang, L., Wang, G., & Meng, L. (2025) · Cartography and Geographic Information Science · doi:10.1080/15230406.2025.2533316 · Cited by 8
DICCR: Double-gated intervention and confounder causal reasoning for vision-language navigation
Zhou, D., Deng, J., Pang, Z., & Li, W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107078 · Cited by 8 · see citing works
Chen, Y.-L., Liang, H., Li, H., Gou, W.-B., & Zhu, L.-F. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3593583 · Cited by 8
“The multimodal graph learning framework uses geometric dependencies between modalities to enhance the modeling of complex, heterogeneous graph datasets [25].”
Hypergraph-based Zero-shot Multi-modal Product Attribute Value Extraction
Hu, J., Gong, J., Shen, H.-D., & Eldardiry, H. (2025) · The Web Conference · doi:10.1145/3696410.3714714 · Cited by 7
Bojesomo, A., Liatsis, P., & Marzouqi, H. A. (2024) · IEEE Signal Processing Magazine · doi:10.1109/msp.2024.3381808 · Cited by 7 · see citing works
“An overview and a guide for constructing MM-GNNs are presented in [7].”
MethodHITS-based Propagation Paradigm for Graph Neural Networks
Khan, M., Mello, G. B. M., Habib, L., Engelstad, P., & Yazidi, A. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3638779 · Cited by 7 · see citing works
Zhou, Z.-Y., Fu, C., Feng, Y., Touya, G., Sester, M., & Weibel, R. (2026) · International Journal of Geographical Information Science · doi:10.1080/13658816.2026.2626935 · Cited by 6
AI-enabled energy baselines for verified building decarbonization
Li, J.-Y., Hao, Y., & Li, Y. (2026) · Scientific Reports · doi:10.1038/s41598-026-36284-w · Cited by 6 · see citing works
Leveraging multimodal learning for enhanced drug-target interaction prediction
Chen, G., & Sun, K.-X. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1639979 · Cited by 6 · see citing works
VQIT-GNN: A collaborative knowledge transfer for node-level structure imbalance
Guan, W., Yang, X.-B., Li, M., Guo, Q.-H., Liu, K.-Y., & Sun, Q.-G. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112632 · Cited by 6
Fan, Q.-Q., Geng, S., Rusli, O., Mazur, F., Han, Z., Rijs, N. J., & Chandrawati, R. (2025) · Nitric oxide · doi:10.1016/j.niox.2025.05.006 · Cited by 6 · see citing works
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 3 · see citing works
Modeling of spatially embedded networks via regional spatial graph convolutional networks
Fan, X., & Hackl, J. (2024) · Comput. Aided Civ. Infrastructure Eng. · doi:10.1111/mice.13286 · Cited by 6
“On the other hand, studies about multimodal GCN have been emerging, current studies are focusing on image processing, knowledge graphs, molecular graphs, and physical/chemistry networks (Ektefaie et al., 2023).”
BackgroundZhang, H., Ma, W.-H., Zhou, X.-S., Zhao, Z., Zhang, R., Bai, H., Huang, C., & Wang, Y.-J. (2025) · Vascular Health and Risk Management · doi:10.2147/vhrm.s555592 · Cited by 5
Multimodal learning for enhanced SPECT/CT imaging in sports injury diagnosis
Jiang, Z., & Shen, Y. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1605426 · Cited by 5 · see citing works
“Forinstance,astudyonautomated detection of synovial fluid in the human knee using MRI and transfer learning demonstrated the feasibility of identifying subtle biomechanical abnormalities through deep models (Iqbal et al., 10.3389/fphys.2025.1605426 2020).”
InfluentialTowards multimodal graph large language model
Wang, X., Zhang, Z., Xiao, L., Chen, H., Ge, C.-D., & Zhu, W.-W. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4627-3 · Cited by 5
“Furthermore, many existing multi-modal graph tasks are discriminative, and future efforts might be devoted to more generative multi-modal graph tasks to advance the evaluation and design of MG-LLM.”
Explaining Hypergraph Neural Networks: From Local Explanations to Global Concepts
Su, S.-Y., Duta, I., Magister, L., & Liò, P. (2024) · arXiv.org · doi:10.48550/arxiv.2410.07764 · Cited by 5
“The neural approach (Kipf & Welling, 2016) has enjoyed exciting successes, setting new state-of-the-art and expanding the reach of machine learning to new modalities (Ektefaie et al., 2023; Battaglia et al., 2018).”
BackgroundHeterogeneous graph attention network improves cancer multiomics integration
Tabakhi, S., Vandermeulen, C., Sudbery, I., & Lu, H. (2024) · arXiv.org · doi:10.48550/arxiv.2408.02845 · Cited by 5
“When performing baseline feature selection methods, we remove five features at each iteration in RFE, employ the recommended hyperparameter values for mRMR-mv as specified in its original paper, and use the values listed in Table S3 for MAgentOmics, ensuring a uniform comparison with…”
BackgroundMethodIdentifying pathological groups from MRI in prostate cancer using graph representation learning
Liu, F., Zhao, Y., Yan, C.-Z., Duan, J.-X., Tang, L., Gao, B., & Wang, R. (2024) · Displays (Guildford) · doi:10.1016/j.displa.2024.102699 · Cited by 5 · see citing works
DRAGON: Drug Repurposing via Graph Neural Networks with Drug and Protein Embeddings as Features
Artiñano-Muñoz, R., Santamaría, L. P., Pérez, A., & Rodríguez‐González, A. (2024) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms61543.2024.00036 · Cited by 4 · see citing works
Discrete Flow-Based Generative Models for Measurement Optimization in Quantum Computing
Huidobro-Meezs, I. L., Dai, J., & Vargas-Hernández, R. A. (2025) · Digital Discovery · doi:10.1039/d6dd00430j · Cited by 4
The Tsetlin Machine Goes Deep: Logical Learning and Reasoning With Graphs
Granmo, O.-C., Abdelwahab, Y., Andersen, P.-A., Borgersen, K. A. K., Clarke, P. F. A., Dumbre, K., Gronningsaeter, Y., Halenka, V., Helin, R., Jiao, L., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.14874 · Cited by 4
“While graphs have been extensively used to represent complex structures involving multimodal data in the context of Graph Neural Networks (GNNs) (Ektefaie et al., 2023), no such mechanism has been available for TMs prior to this work.”
Use of brain-computer interface in educational paradigm
Zaib-un-Nisa, Mazhar, T., Shahzad, T., Ahmad, W., & Hamam, H. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00394-8 · Cited by 4 · see citing works
“It involves visual, cognitive, perceptual, and experimental learning [44].”
Global Feature Enhancing and Fusion Framework for Strain Gauge Status Recognition
Zhang, X., Wang, P., Wang, C., Xu, Z., Nie, X.-H., & Wang, W. (2025) · The Web Conference · doi:10.1145/3701716.3715229 · Cited by 4 · see citing works
Uncertainty-Aware Unimodal and Multimodal Learning for Evolving Hardware Trojan Detection
Vishwakarma, R., & Rezaei, A. (2025) · Journal of Hardware and Systems Security · doi:10.1007/s41635-025-00160-2 · Cited by 4 · see citing works
Multi-Modal Graph-Based Sentiment Analysis via Hybrid Contrastive Learning
Zhang, J. (2024) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece61636.2024.10729439 · Cited by 3 · see citing works
“The task of multi-modal analysis is gaining popularity, including models that integrate attention mechanisms [2][3] and graph structures [4][5].”
BackgroundExplaining Graph Neural Networks for Node Similarity on Graphs
Daza, D., Chu, C. X., Tran, T.-K., Stepanova, D., Cochez, M., & Groth, P. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2407.07639 · Cited by 4
“In the following sections, we discuss the problems of learning representations of nodes for the similarity task, and our proposals on how similarity scores can be explained.”
Chen, J.-L., Seng, K., Ang, L.-M., Smith, J. S., & Xu, H. (2024) · Future Internet · doi:10.3390/fi16050161 · Cited by 4 · see citing works
“The introduction of graph learning for modality data [11] can be divided into reconstructing modalities into graph structure [12–14] and identifying graph nodes from modalities [15–17].”
BackgroundJohnson, R., Li, M. M., Noori, A., Queen, O., & Zitnik, M. (2023) · arXiv.org · doi:10.48550/arxiv.2310.13767 · Cited by 4
Self-citation
- Authors
- Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M., & Zitnik, M.
- Affiliations
- Harvard University (US)
- Venue
- Nature Machine Intelligence
- Published
- 2023-04-03
- Volume
- 5
- Issue
- 4
- Pages
- 340-350
- ISSN
- 2522-5839 (ISSN-L)
- PMC
- PMC10704992
- OpenAlex
- W4362579348
- Cited by
- 166
- References
- 97
- Citations by year
- 2026: 44 · 2025: 72 · 2024: 37 · 2023: 13
- FWCI
- 15.65
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- AstraZeneca
- U.S. Air Force (FA8702‐15‐D‐0001)
- Harvard Data Science Initiative, Harvard University
- National Human Genome Research Institute (T32HG002295)
- National Institute of Allergy and Infectious Diseases (R01AI155765)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Graph Theory and Algorithms
- Keywords
- graph representation learning, inductive bias, geometric relationships
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
