The DOI matches its Crossref record; 88 of the 93 references it cites were found in a registry or an index.
A survey of multimodal deep generative models
Open accessAbstract
Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e. generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on the position of the layer for the multimodal interaction, the data fusion architectures can be classified into early, intermediate, and later fusion [50].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [50]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Goyal et al. [20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 3×
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2×
“In addition to these, cycleconsistency loss is used in image-to-image translation, which is the task of transforming an image belonging to one domain to another domain [57,58].”
BackgroundPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Srivastava and Salakhutdinov [37] propose a multimodal generative model based on deep belief networks [38,39], which has since been used in various multimodal applications [40,41].”
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2×
“In general, a “good representation” is one that can be used for various tasks while retaining the characteristics of the original input [6].”
BackgroundAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 36
- Stein BE, 1993, The merging of the senses
- Kingma DP, 2014, International Conference on Learning Representations
- van den Oord A, Dieleman S, Zen H, et al. Wavenet: a generative model for raw audio; 2016. Preprint arXiv:1609.03499. · arXiv
- Van Oord A, 2016, International Conference on Machine Learning
- Rezende D, 2015, International Conference on Machine Learning
- Kingma DP, 2018, Advances in Neural Information Processing Systems
- Xu C, Tao D, Xu C. A survey on multi-view learning; 2013. Preprint arXiv:1304.5634. · arXiv
- Tsai Y-HH, 2021, International Conference on Learning Representations
- Sohn K, 2015, Adv Neural Inf Process Syst, 28, 3483
- Tschannen M, Bachem O, Lucic M. Recent advances in autoencoder-based representation learning; 2018. Preprint arXiv:1812.05069. · arXiv
- Srivastava N, Salakhutdinov R. Learning representations for multimodal data with deep belief nets. In: International Conference on Machine Learning Workshop, Vol. 79; 2012. p. 3.
- Sohn K, 2014, Adv Neural Inf Process Syst, 27, 2141
- 10.1162/jmlr.2003.3.4-5.993 · DOI
- Higgins I, Matthey L, Pal A, et al. beta-vae: learning basic visual concepts with a constrained variational framework. ICLR; 2016.
- Vincent P, 2010, J Machine Learning Res, 11, 3371
- Liu M-Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks; 2017. Preprint arXiv:1703.00848. · arXiv
- Wu M, Goodman N. Multimodal generative models for compositional representation learning; 2019. Preprint arXiv:1912.05075. · arXiv
- Suzuki M, Nakayama K, Matsuo Y. Improving bi-directional generation between different modalities with variational autoencoders; 2018. Preprint arXiv:1801.08702. · arXiv
- Daunhawer I, 2021, Pattern Recognit, 12544, 459
- Wu M, 2018, Advances in Neural Information Processing Systems
- Suzuki M, Nakayama K, Matsuo Y. Joint multimodal learning with deep generative models; 2016. Preprint arXiv:1611.01891. · arXiv
- Tian Y, Engel J. Latent translation: crossing modalities by bridging generative models; 2019. Preprint arXiv:1902.08261. · arXiv
- Hsu W-N, Glass J. Disentangling by partitioning: a representation learning framework for multimodal sensory data; 2018. Preprint arXiv:1805.11264. · arXiv
- Shi Y, 2021, International Conference on Learning Representations
- 10.1609/aaai.v34i07.6778 · DOI
- Wang D, 2015, International Joint Conference on Artificial Intelligence
- Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders; 2015. Preprint arXiv:1509.00519. · arXiv
- Bouchacourt D, 2018, AAAI Conference on Artificial Intelligence
- Maddison CJ, 2017, International Conference on Learning Representations
- Jang E, 2017, International Conference on Learning Representations
- van den Oord A, Li Y, Vinyals O. Representation learning with contrastive predictive coding; 2018. Preprint arXiv:1807.03748. · arXiv
- Xiao H, Rasul K, Vollgraf R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017. Preprint arXiv:1708.07747 · arXiv
- Ha D, Schmidhuber J. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems; 2018; Montreal, Canada.
- Hafner D, 2019, International Conference on Machine Learning
- Hafner D, 2019, International Conference on Learning Representations
- Suzuki M, Kaneko T, Matsuo Y. Pixyz: a library for developing deep generative models; 2021. Preprint arXiv: 2107.13109. · arXiv
Cited by
Sorted by citations
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 590
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
Hagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 227
“For example, text-based models may lack the ability to incorporate visual or emotional context into their understanding, while image-based models might lack textual or semantic understanding [65].”
BackgroundInfluentialVersatile Diffusion: Text, Images and Variations All in One Diffusion Model
Xu, X., Wang, Z., Zhang, E., Wang, K., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00713 · Cited by 212
“Multimodal generative tasks involve simultaneous representation learning and generation/synthesis [91], in which representation networks [99, 45, 25, 96, 64, 95] with contrastive loss [66, 15, 1, 93, 94] played an essential role.”
BackgroundLifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 136
Human Motion Generation: A Survey
Zhu, W., Ma, X., Ro, D., Ci, H., Zhang, J., Shi, J., Gao, F., Tian, Q. C., & Wang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3330935 · Cited by 141
“This survey serves as a complement to existing survey papers that focus on human pose estimation [53], [54], motion capture [55], [56], and deep generative models [57], [58], [59].”
BackgroundAn Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 124
“GenAI is an AI technology that can generate various multimedia content [46], [88], [115].”
BackgroundChallenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Survey
Vu, T.-H., Jagatheesaperumal, S. K., Nguyen, M.-D., Huynh, N. V., Kim, S., & Pham, Q. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3487627 · Cited by 78
“This is achieved by integrating diverse modalities like visual, auditory, and textual information [76].”
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S. M., Özkan, E., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103042 · Cited by 60
“Both research directions have experienced renewed interest in the light of contrastive and self-supervised learning [33, 34] and generative modeling [35].”
BackgroundLi, X., Yuan, Y., Liu, G., Han, Z. Y., & Stouffs, R. M. F. A. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2023.113876 · Cited by 54
Creativity and Machine Learning: A Survey
Franceschelli, G., & Musolesi, M. (2024) · ACM Computing Surveys · doi:10.1145/3664595 · Cited by 83
Multimedia Research Toward the Metaverse
Chen, S. (2022) · IEEE Multimedia · doi:10.1109/mmul.2022.3156185 · Cited by 38
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
“Image-text multimodal models [108, 169, 170, 171, 172, 173, 174, 175]are a product of artificial intelligence that can process and understand various types of information, such as text and images, simultaneously.”
BackgroundA systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 41
“These modalities have significantly different data characteristics and forms of representation (J. Chen & Zhang, 2020; Suzuki & Matsuo, 2022; Z. Zhang et al., 2022).”
Emergent communication through Metropolis-Hastings naming game with deep generative models
Taniguchi, T., Yoshida, Y., Matsui, Y., Le Hoang, N., Taniguchi, A., & Hagiwara, Y. (2023) · Advanced Robotics · doi:10.1080/01691864.2023.2260856 · Cited by 33
“Notably, inter-GMM + VAE can be regarded as a variant of multimodal VAEs [47].”
BackgroundSilva, D. D., Kaynak, O., El-Ayoubi, M., Mills, N., Alahakoon, D., & Manic, M. (2024) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2024.3382962 · Cited by 31
Bidirectional visual-tactile cross-modal generation using latent feature space flow model
Fang, Y., Zhang, X., Xu, W., Liu, G., & Zhao, J. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.12.042 · Cited by 25
Granda, B. A. S., Inzhivotkina, Y. V., Apolo, M. F. I., & Fajardo, J. G. U. (2024) · Edutec Revista Electrónica de Tecnología Educativa · doi:10.21556/edutec.2024.89.3251 · Cited by 31
Bayesian structural model updating with multimodal variational autoencoder
Itoi, T., Amishiki, K., Lee, S., & Yaoyama, T. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117148 · Cited by 20
“In this study, a multimodal VAE [25], [26], [27] was employed to develop a versatile framework for Bayesian structural model updating.”
BackgroundMethodA multimodal dynamical variational autoencoder for audiovisual speech representation learning
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106120 · Cited by 20
“VAEs are equipped with encoder and decoder models, resulting in a more stable and faster training process than GANs, which makes them well-suited for multimodal generative modeling (Suzuki & Matsuo, 2022).”
Viswan, V., Shaffi, N., Malathy, E., Selvi, G. C., Kavitha, B. R., Abdesselam, A., Wang, S., Suganthan, P. N., Shezawi, I. A., & Mahmud, M. (2026) · Brain Informatics · doi:10.1186/s40708-025-00291-w · Cited by 17
“Advanced DL architectures—such as Multimodal CNNs [92], RNNs [93], graph neural networks (GNNs) [94], Transformers [95], and generative models [96]— offer a solution by learning hierarchical and cross-modal representations that capture complex interdependencies.”
Duan, W., McNeese, N. J., & Li, L. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701197 · Cited by 22
Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows
Yamada, Y., Colan, J., Davila, A., & Hasegawa, Y. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03101-6 · Cited by 15
A Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17
“In a multimodal setting, an encoder would take one modality as input, and the decoder would generate another modality 33 (Fig.”
Informed machine learning for complex data
Oneto, L., Navarin, N., Micheli, A., Pasa, L., Gallicchio, C., Bacciu, D., & Anguita, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132505 · Cited by 16
Informed Machine Learning: Excess risk and generalization
Oneto, L., Ridella, S., & Anguita, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130521 · Cited by 21
Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation GuidelinesPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2024) · medRxiv · doi:10.1101/2024.10.23.24316003 · Cited by 15
Yang, W., Lü, Y., Yeom, S., & Herbert, D. (2025) · IEEE Access · doi:10.1109/access.2025.3622002 · Cited by 13
“…innovations through various models [1], [2]; others have explored the specific types of GenAI models, such as reviewing Generative adversarial networks (GANs) [3], video GANs [4], multimodal deep GenAI models [5], generative diffusion models [6], and GPT (Generative Pre-trained Transformer) [7].”
Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game
Hoang, N. L., Taniguchi, T., Hagiwara, Y., & Taniguchi, A. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1290604 · Cited by 12
“This development is notably relevant in the context of multimodal deep learning, which combines different data modalities to improve the modeling of diverse data (Suzuki and Matsuo, 2022).”
InfluentialZou, S., Wang, X., Tao, Y., Zeng, K., Li, G., & Xie, X. (2025) · The Journal of the Acoustical Society of America · doi:10.1121/10.0034831 · Cited by 10
MDLDroid: Multimodal Deep Learning Based Android Malware Detection
Singh, N., & Tripathy, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49099-6_10 · Cited by 9
SMOTE-enhanced XGBoost for rapid seismic damage assessment of bridge portfolios
Di, S., Wang, Y.-Y., Yang, D. Y., Liu, Y., Zhang, J., & Zheng, W. (2025) · Soil Dynamics and Earthquake Engineering · doi:10.1016/j.soildyn.2025.109712 · Cited by 8
A method for generating personalized learning content based on AIGC
Hu, Z. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101331 · Cited by 8
Zhang, C., Li, X., Bai, W., Sangaiah, A. K., & Li, D. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3524248 · Cited by 8
“The second challenge pertains to the establishment of a stable MG model, which may involve constructs such as multigranulation rough set models [6], multi-scale data analysis models [7], multi-modal models [8], and so forth.”
Multi-view-AE: A Python package for multi-view autoencoder models
Aguila, A. L., Jayme, A., Montaña-Brown, N., Heuveline, V., & Altmann, A. (2023) · The Journal of Open Source Software · doi:10.21105/joss.05093 · Cited by 8
Xiong, Y., Chai, C., Gan, Y. S., & Chong, H. (2025) · Innovative Infrastructure Solutions · doi:10.1007/s41062-025-01975-2 · Cited by 7
“Therefore, several studies are focused on adapting the GAN framework to overcome the limitation of accepting only a single modality [40].”
Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis
Lüpke, S., Yeganeh, Y., Adeli, E., Navab, N., & Farshad, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-84525-3_17 · Cited by 7
Zeus: zero-shot LLM instruction for union segmentation in multimodal medical imaging
Dai, S., Ye, K., Liu, G., Tang, H., & Zhan, L. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02742-6 · Cited by 10
“Besides, multimodal learning shows great importance in autonomous driving [55], generative model [44], healthcare [36, 63, 64].”
SwarmDiff: Swarm Robotic Trajectory Planning in Cluttered Environments via Diffusion Transformer
Ding, K., Jiao, C., Hu, Y., Zhou, K., Wu, P., Mu, Y., & Liu, C. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00400 · Cited by 9
Nadeem, M., Sohail, S. S., Madsen, D. Ø., Alzahrani, A. I., Ser, J. D., & Muhammad, K. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536937 · Cited by 7
“Interested readers refer to [45], [46], [47] for the same.”
Bimodal PET/MRI generative reconstruction based on VAE architectures
Gautier, V., Bousse, A., Sureau, F. C., Comtat, C., Maxim, V., & Sixou, B. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad9133 · Cited by 6
Bouatmane, A., Daaif, A., Bousselham, A., Bouihi, B., & Bouattane, O. (2025) · IEEE Access · doi:10.1109/access.2025.3556700 · Cited by 6
“Recent advancements in deep learning and multimodal [2], [3], [6], [9], [10], [19], [47] data fusion offer new opportunities to address these challenges.”
ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model
Wang, Q., Fan, C., Jia, T., Han, Y., & Wu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29486 · Cited by 6
Concept formation through multimodal integration using multimodal BERT and VQ-VAE
Miyazawa, K., & Nagai, T. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2141583 · Cited by 6
“Furthermore, various multimodal deep generative models have been proposed to handle multimodal information [22].”
BackgroundFrom imaging to omics: deep learning is bridging MRI and liquid biopsy in bone tumor diagnosis
Wang, F., Chen, J., Zheng, L., & Huang, X. (2026) · Journal of bone oncology · doi:10.1016/j.jbo.2026.100753 · Cited by 6
“One of the most pressing issues in multimodal modeling is the heterogeneity of the input data [222,230,231].”
Empowering multimodal analysis with visualization: A survey
Wang, J., Deng, Z., Deng, D., Wang, X., Sheng, R., Cai, Y., & Qu, H. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100748 · Cited by 4
Hierarchical multi-modal fusion architecture search for stock market forecasting
Lee, K., Kim, Y. J., Kim, M., & Kim, H. Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113581 · Cited by 6
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Ge, J., Subramanian, S., Darrell, T., & Li, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.75 · Cited by 4
Ikegawa, Y., Fukuma, R., Sugano, H., Oshino, S., Tani, N., Tamura, K., Iimura, Y., Suzuki, H., Yamamoto, S., Fujita, Y., et al. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad417a · Cited by 4
Taniguchi, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35382-6_2 · Cited by 4
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 5
Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
Zheng, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768625 · Cited by 3
Object-Aware Image Augmentation for Audio-Visual Zero-Shot Learning
Dong, Y., Chen, S., Duan, B., Ding, W., Wang, Y., & You, X. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3485624 · Cited by 3
“If we have a method to embed all modalities into a shared representation and generate modalities from it, we can achieve generation between any modalities through this space [63].”
Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
Šejnová, G., Vavrečka, M., & Štěpánová, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802160 · Cited by 3
“These models have emerged as a powerful extension of the unimodal Variational Autoencoder and are suitable for mapping multiple modalities into a unified joint space, allowing for the generation of one modality from another [9].”
A Multimodal Hierarchical Variational Autoencoder for Saliency Detection
Yu, Z., Zhang, J., & Barnes, N. M. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191461 · Cited by 3
“…properties such as its flexibility in conditioned generation with meaningful encoding, its efficient sampling strategy using amortized variational inference [59] as well as its explicitness in reflection of heterogeneity of multimodal data via differences in probabilistic distributions [60].”
A Multi-Modal Retrieval Model for Mathematical Expressions Based on ConvNeXt and Hesitant Fuzzy Set
Li, R., Wang, J., & Tian, X. (2023) · Electronics · doi:10.3390/electronics12204363 · Cited by 4
“Current research in multimodal studies is expanding rapidly, exploring the integration of various data types and modes to gain a more comprehensive understanding of complex phenomena [4,5].”
BackgroundSu, B., Wang, Y., Schiavazzi, D. E., & Liu, F. (2023) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v1i1.27495 · Cited by 3
Towards Generated Image Provenance Analysis via Conceptual-Similar-Guided-SLIP Retrieval
Xia, X., Wang, L., Sun, J., & Nakagawa, A. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3388958 · Cited by 3
“R ECENTLY, generative AI has become a prominent research area [1], [2]. synthesize Image generation models can synthesize realistic images that are indistinguishable from natural ones.”
BackgroundAn Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v1 · Cited by 3
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v2 · Cited by 2
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v3 · Cited by 2
Noda, K., Soda, T., & Yamashita, Y. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1330512 · Cited by 2
“Additionally, the concept of multimodal learning has been applied in the fields of machine learning and deep neural networks (Baltrušaitis et al., 2019; Suzuki and Matsuo, 2022).”
BackgroundWang, Q., Fan, C., Jia, T., Han, Y., & Wu, X. (2023) · International Conference on Neuromorphic Computing (ICNC) · doi:10.1109/icnc59488.2023.10462792 · Cited by 1
“Besides, deep generative models, including variational autoencoders (VAEs) [14], [15], [29], generative adversarial networks (GAN) [16]–[18], and so on, also show their excellent performance because of their superiority in feature representation [19].”
BackgroundMeasuring Similarity of Dual-Modal Academic Data Based on Multi-Fusion Representation Learning
Zhang, L., Gao, Q., Liu, M., Gu, Z., & Lang, B. (2024) · IEEE Access · doi:10.1109/access.2024.3427731 · Cited by 1
“Retrieving relevant texts via a single image, or retrieving relevant images through text or a combination of text and images has become an interesting and challenging problem [12], [13], [14], [15], [16], [17], [18].”
BackgroundLiang, B., Guo, J., Liu, S. Q., Zong, C., Cheng, Y., & Chen, J. (2025) · Optik · doi:10.1016/j.ijleo.2025.172261 · Cited by 1
Multimodal Romanian language resources and tools: challenges and perspectives
Mitrofan, M., Irimia, E., & Păiș, V. (2025) · Discover Data · doi:10.1007/s44248-025-00060-4 · Cited by 1
Virtual Trainer for Learning Mexican Sign Language Using Video Similarity Analysis
Rivera-Cervantes, F. D. J., Esparza, D. M. C., Terven, J. R., Romero-González, J.-A., González-Rodríguez, J.-R., Ibarra-Corona, M. A., & Ramírez-Pedraza, A. (2025) · Technologies · doi:10.3390/technologies13120540 · Cited by 1
Effectively Obtaining Acoustic, Visual, and Textual Data from Videos
León, J. E., & Carrasco, M. (2025) · Applied Sciences · doi:10.3390/app152312654 · Cited by 2
Gao, Z., Han, Z., Fu, Y., Lv, H., Li, M., Zhao, X., & Zhu, J. (2026) · Polymers · doi:10.3390/polym18020203 · Cited by 1
“Common fusion strategies include early fusion, mid-level fusion, and late fusion [34].”
Artificial Intelligence-Based Digital Transformation
Volpentesta, T. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80592-096-020261002 · Cited by 1
Shen, C., & Namiki, A. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.118384 · Cited by 1
Symbol Emergence as a Decentralized Bayesian Inference
Taniguchi, T. (2022) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.40.883 · Cited by 1
Eder, M., & Steinbauer-Wagner, G. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10609988 · Cited by 1
“Using deep multimodal variational autoencoders (DMVAEs) [8], [9], our framework combines robot-specific properties to represent a specific robot and environmental information from aerial imagery to improve traversability predictions, with the goal of increasing the applicability of the framework to…”
Instruction-Driven 3D Facial Expression Generation and Transition
Vo, A. H., Kim, T., Jin, H., Choi, S.-M., & Kim, Y.-G. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3565929 · Cited by 1
“Furthermore, integrating feature representations from different modalities minimizes ambiguity in redundant resources and effectively handles missing modalities, particularly in generative models, by improving inter-modality relations [27], [28].”
Ayub, S., Gorman, J. L., Chen, E., Jackson, H. W., Selega, A., & Campbell, K. R. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag010 · Cited by 1
“The latent space of a VAE captures the salient features that explain the variation in high dimensional data (Kingma and Welling 2014, Suzuki and Matsuo 2022).”
Robot Control based on World Models and Predictive Learning
Taniguchi, T., Okada, M., Maeyama, K., Okumura, R., & Kinose, A. (2022) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.40.790 · Cited by 1
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Özkan, E., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S., et al. (2023) · arXiv (Cornell University) · doi:10.3929/ethz-b-000601130 · Cited by 1
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271 · Cited by 1
MultiVae: A Python package for Multimodal Variational Autoencoders on Partial Datasets.
Senellart, A., Chadebec, C., & Allassonnière, S. (2025) · The Journal of Open Source Software · doi:10.21105/joss.07996 · Cited by 1
“See Kingma & Welling (2014) to learn more about the VAE framework and Suzuki & Matsuo (2022) for a survey on multimodal VAEs.”
Audiovisual speech representation learning applied to emotion recognition
Sadok, S. (2024) · theses.fr (ABES) · doi:10.70675/0f15a36fzecf8z4cffz94e7z120e7d8fd39d · Cited by 1
A multimodal dynamical variational autoencoder for audiovisual speech representation learningPreprint
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03582 · Cited by 1
“VAEs are equipped with encoder and decoder models, resulting in a more stable and faster training process than GANs, which makes them well-suited for multimodal generative modeling (Suzuki & Matsuo, 2022).”
集合的予測符号化に基づく言語と認知のダイナミクス ～記号創発ロボティクスの新展開に向けて～Preprint
Taniguchi, T. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4a6j7 · Cited by 1
Multimodal Outputs for the Workplace From Generative AI
Hai-Jew, S. (2024) · Advances in multimedia and interactive technologies book series · doi:10.4018/979-8-3693-2927-6.ch008
VAD Emotion Control in Visual Art Captioning via Disentangled Multimodal Representation
Ueda, R., Narimatsu, H., Miyao, Y., & Kumano, S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00020
“In the field of machine learning and representation learning, various MMRL methods have been proposed [16].”
Ahmed, B., Daaif, A., Bousselham, A., & Bouattane, O. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-88304-0_11
From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning
Xie, H., Maharjan, R. S., Tavella, F., & Cangelosi, A. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7033-8_6 · Cited by 1
Mahaseni, B., & Khan, N. (2025) · Empathic Computing · doi:10.70401/ec.2025.0010 · Cited by 2
Gao, Y. F., Shao, M., Zhao, C., Du, R., Wu, J., & Wu, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5388880
VAE-based Multivariate Nowcasting of the Japanese Economy
Ueda, T., Izumi, K., & Murayama, Y. (2025) · Computational Economics · doi:10.1007/s10614-025-11140-7
“For example, in supply chain analysis, it is considered crucial to handle micro-level, diverse distributions by product or process, making the explicit introduction of multimodality (Suzuki & Matsuo, 2022) potentially necessary.”
Langer, C., Georgie, Y. K., Porohovoj, I., Hafner, V. V., & Ay, N. (2025) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl63968.2025.11204413
“A more exhaustive overview over multimodal deep generative networks can be found in [12], where the authors discuss 22 different models and their properties in detail.”
Generative Künstliche Intelligenz im Marketing
Decker, R. (2025) · Springer eBooks · doi:10.1007/978-3-658-46709-8_69-1
Implementation Paths of Generative AI in Multimodal Learning
Tan, B. (2025) · Journal of Computer Science and Artificial Intelligence · doi:10.54097/rwr0vn28
AI-Driven Motion Capture Data Recovery: A Comprehensive Review and Future Outlook
Almaleh, A., Ushaw, G., & Davison, R. (2025) · Sensors · doi:10.3390/s25247525
¿Gerentes y administradores exclusivamente humanos o ciberadministradores? Virtus in medium est
Dinis, M. (2025) · Revista e-mercatoria · doi:10.18601/16923960.v25n1.06
Gao, Z., Han, Z., Fu, Y., Zhao, X., Lv, H., Li, M., & Zhu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5394703
Generative AI: Principles, Potentials, and Ethical Dimensions
Cheshmehzangi, A. (2025) · Urban sustainability · doi:10.1007/978-981-95-4766-1_3
Journey Through Language: Models and Prompt Engineering
Akerkar, R. A. (2026) · Springer eBooks · doi:10.1007/978-3-031-91084-5_7
Pioneer: A Method for Enhancing GPU Utilization in Workloads Through Potential-Aware Prioritization
Guo, Q., Zhao, W., Xie, Y., & Zhao, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5640-3_15
Bad Designs by Good Talkers: Chatbots Failing to Architect Audio Encoders for Image Synthesis
León, J. E., Carrasco, M., & Peters, A. A. (2026) · IEEE Access · doi:10.1109/access.2026.3668130
“Audio-to-image generation remains relatively underexplored, despite evidence that audio signals carry rich semantic information that could guide visual content creation [21], [22], [44].”
Influential
Metadata
- Authors
- Suzuki, M., & Matsuo, Y.
- Affiliations
- The University of Tokyo (JP)
- Corresponding author
- Suzuki, Masahiro
- Venue
- Advanced Robotics
- Publisher
- Informa UK Limited
- Published
- 2022-02-21
- Volume
- 36
- Issue
- 5-6
- Pages
- 261-278
- ISSN
- 0169-1864 (ISSN-L)
- 1568-5535
- OpenAlex
- W4213299273
- Cited by
- 109
- References
- 93
- Citations by year
- 2026: 18 · 2025: 38 · 2024: 27 · 2023: 22 · 2022: 4
- FWCI
- 7.45
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Advanced Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Energy and Industrial Technology Development Organization (JPNP16007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal learning, deep generative models, variational autoencoder, cross-modal generation, shared representations, multimodal generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
