A survey of multimodal deep generative models
Open accessAbstract
Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e. generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Based on the position of the layer for multimodal interaction, data fusion architectures can be classified into early, intermediate, and later fusion.
“Based on the position of the layer for the multimodal interaction, the data fusion architectures can be classified into early, intermediate, and later fusion [50].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [50]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2× · see citing works
“Goyal et al. [20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 3× · see citing works
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2× · see citing works
“In addition to these, cycleconsistency loss is used in image-to-image translation, which is the task of transforming an image belonging to one domain to another domain [57,58].”
BackgroundPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1× · see citing works
“Srivastava and Salakhutdinov [37] propose a multimodal generative model based on deep belief networks [38,39], which has since been used in various multimodal applications [40,41].”
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2× · see citing works
“In general, a “good representation” is one that can be used for various tasks while retaining the characteristics of the original input [6].”
BackgroundAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
A synaptic model of memory: long-term potentiation in the hippocampus
Bliss, T. V. P., & Collingridge, G. L. (1993) · Nature · doi:10.1038/361031a0 · Cited by 11,566 · see citing works
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1× · see citing works
“A common multimodal dataset used in many multimodal deep generative models [33, 88, 88, 89, 91, 104, 118] is the CelebA dataset [52], which contains face images and their attributes.”
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260 · cited 4× · see citing works
“To consider this, suppose that we are given two datasets, In transfer learning, the domain of X 1 is defined by a pairing of the data feature space X 1 and the marginal distribution p ( X 1 ) [67, 110].”
InfluentialRELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,894 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · cited 7× · see citing works
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
BackgroundMethodInfluentialTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1× · see citing works
“Wu and Goodman [65] consider unimodal inference as ‘expert’ and propose approximating joint inference using the following product-of-experts (PoE) [89] form:”
BackgroundReading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works
(1988) · Cited by 3,656 · see citing works
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 2× · see citing works
“However, most of these are conditional models [35, 123], i.”
2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,846 · see citing works
(2011) · Cited by 2,752 · see citing works
2008 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2008) · doi:10.1109/iros13647.2008 · Cited by 2,203 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · cited 1× · see citing works
“The first purpose is sometimes referred to as multimodal representation learning [50], and besides generative approaches, there are studies based on discriminative approaches such as contrastive learning [1, 97, 100].”
Background2014 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2014) · doi:10.1109/iros20755.2014 · Cited by 1,309 · see citing works
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 3× · see citing works
“In a multimodal learning framework, the term modality originally referred to each scheme and situation in which data are perceived [5, 45], i.”
Diverse Image-to-Image Translation via Disentangled Representations
Lee, H.-Y., Tseng, H.-Y., Huang, J., Singh, M., & Yang, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_3 · Cited by 1,285 · cited 2× · see citing works
“Although some of these models consider shared latent variables in different domains [59,60], they assume that the input is two domain images of the same size, not heterogeneous modalities; therefore, we do not include them in this survey.”
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder
Park, D., Hoshi, Y., & Kemp, C. C. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2801475 · Cited by 1,131 · cited 1× · see citing works
“[111] introduced an LSTM-based JVAE to detect anomalous feeding executions given 17 sensory signals from five types of sensors on a PR2 robot.”
Blei, D. M., & Jordan, M. I. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860460 · Cited by 1,078 · cited 2× · see citing works
“Other research involves extending topic models such as latent Dirichlet allocation (LDA) [9] to multimodal input [8, 58].”
BackgroundMethodHierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · see citing works
Hinton, G. E. (2009) · Scholarpedia · doi:10.4249/scholarpedia.5947 · Cited by 907 · cited 1× · see citing works
“Xing et al. [116] build a joint model of images and texts using a linear RBM. Srivastava et al. [82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891 · cited 1× · see citing works
“[76] propose CADA-VAE for generalized zero-shot learning [70, 114], which is used to embed different modalities, namely image features and class attributes, into a shared representation.”
A survey of multi-view machine learning
Sun, S. (2013) · Neural Computing and Applications · doi:10.1007/s00521-013-1362-6 · Cited by 878 · see citing works
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · cited 2× · see citing works
“[99] conducted experiments on real-world time-series multimodal datasets such as CMU-MOSI [120], which includes three modalities: language, acoustic, and visual.”
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · cited 1× · see citing works
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
Pyro: Deep Universal Probabilistic ProgrammingPreprint
Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F. H., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., & Goodman, N. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09538 · Cited by 767 · cited 2× · see citing works
“Probabilistic modeling languages, such as Pyro [7], and deep generative modeling libraries, such as Pixyz [90], might help to address this difficulty.”
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 763 · cited 1× · see citing works
“[39] worked on hand pose estimation using RGB images and keypoints of each hand in the Rendered Hand pose dataset [126] as multimodal data.”
MethodGeneralized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00844 · Cited by 747 · cited 6× · see citing works
“In CADA-VAE [76], the authors used the Caltech-UCSD Birds (CUB)-200 dataset [111], which contains bird images and their attributes, as a benchmark for zero-shot learning.”
InfluentialA Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 637 · cited 1× · see citing works
“The first purpose is sometimes referred to as multimodal representation learning [50], and besides generative approaches, there are studies based on discriminative approaches such as contrastive learning [1, 97, 100].”
Tolstikhin, I., Bousquet, O., Gelly, S., & Schölkopf, B. (2018) · MPG.PuRe (Max Planck Society) · Cited by 602 · see citing works
Giorgio Metta, B., Sandini, G., Vernon, D., Natale, L., & Nori, F. (2008) · Workshop on Performance Metrics for Intelligent Systems (PerMIS) · doi:10.1145/1774674.1774683 · Cited by 548 · see citing works
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 531 · cited 1× · see citing works
“To deal with this difficulty, it might be important to use techniques such as transfer learning and continuous learning [123,124], given that humans have the ability to act in unknown environments based on their memories of other environments and the ability to continuously learn new things from the past.”
BackgroundThrun, S., & Mitchell, T. M. (1995) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(95)00004-y · Cited by 523 · cited 1× · see citing works
“To deal with this difficulty, it might be important to use techniques such as transfer learning and continuous learning [49, 95], given that humans have the ability to act in unknown environments based on their memories of other environments and the ability to continuously learn new things from the past.”
BackgroundMultimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · cited 1× · see citing works
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
BackgroundA Review of Generalized Zero-Shot Learning Methods
Pourpanah, F., Abdar, M., Luo, Y., Zhou, X., Wang, R., Lim, C. P., Wang, X., & Wu, Q. M. J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3191696 · Cited by 493 · cited 1× · see citing works
“[72] propose CADA-VAE for generalized zero-shot learning [84,85], which is used to embed differentmodalities, namely image features and class attributes, into a shared representation.”
Sliced and Radon Wasserstein Barycenters of Measures
Bonneel, N., Rabin, J., Gabriel Peyré, A., & Pfister, H. (2014) · Journal of Mathematical Imaging and Vision · doi:10.1007/s10851-014-0506-3 · Cited by 471 · cited 1× · see citing works
“They optimize the encoder and decoder in the VAE framework while simultaneously optimizing the slicedWasserstein distance [82] between the embeddings in each modality, z1 and z ′ 2, and the loss in class label prediction from each embedding.”
MethodAssociation for the Advancement of Artificial Intelligence
Crosby, M., & Petrick, R. P. A. (2014) · Cited by 462 · see citing works
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · cited 1× · see citing works
“[82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
BackgroundCodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM
Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., & Davison, A. J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00271 · Cited by 384 · cited 1× · see citing works
“, in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audio-visual speech enhancement [73, 74] and the acquisition of the joint representation of depth and RGB images [11].”
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · cited 10× · see citing works
“Multimodal Factorization Model (MFM) [99] includes a label y as well as a multimodal input X as observation variables, and assumes the following generative model: where F s = { f sm } M m =1 and f c are modality-specific and modality-invariant latent factors, respectively.”
InfluentialMulti-source Deep Learning for Human Pose Estimation
Ouyang, W., Chu, X., & Wang, X. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.299 · Cited by 269 · cited 1× · see citing works
“Furthermore, multimodal DBMs are used in a variety of multimodal applications [66, 68, 86].”
BackgroundSelf-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 247 · see citing works
A Comprehensive Survey on Cross-modal RetrievalPreprint
Wang, K., Yin, Q., Wang, W., Wu, S., & Wang, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06215 · Cited by 230 · see citing works
Topic regression multi-modal Latent Dirichlet Allocation for image annotation
Putthividhy, D., Attias, H. T., & Nagarajan, S. S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540000 · Cited by 195 · see citing works
Multimodal integration learning of robot behavior using deep neural networks
Noda, K., Arie, H., Suga, Y., & Ogata, T. (2014) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.03.003 · Cited by 190 · cited 1× · see citing works
“Multimodal learning is especially important for robots that need to operate properly in the real world, because they need to make sense of the world based on the various types of information they receive through their onboard sensors [3].”
BackgroundAudio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. E. D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 185 · cited 1× · see citing works
“[82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
Transfer Learning on Heterogenous Feature Spaces via Spectral Transformation
Shi, X., Liu, Q., Fan, W., Yu, P. S., & Zhu, R. (2010) · Proceedings · doi:10.1109/icdm.2010.65 · Cited by 185 · cited 1× · see citing works
“Various types of modalities in our world–such as vision, sound, and smell–are said to be heterogeneous [77, 110].”
BackgroundRecurrent Independent MechanismsPreprint
Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., & Schölkopf, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10893 · Cited by 164 · cited 1× · see citing works
“[20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
BackgroundDeep Multimodal Learning for Affective Analysis and Retrieval
Pang, L., Zhu, S., & Ngo, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482228 · Cited by 154 · cited 1× · see citing works
“Furthermore, multimodal DBMs are used in a variety of multimodal applications [66, 68, 86].”
BackgroundIvanovic, B., Leung, K., Schmerling, E., & Pavone, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3043163 · Cited by 146 · cited 1× · see citing works
“These models are also sometimes referred to as multimodal generative models [36], but we do not include them in multimodal generative models in this paper because they do not acquire a shared representation and can only generate crossmodally in one direction.”
BackgroundJaques, N., Taylor, S., Sano, A., & Picard, R. W. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273601 · Cited by 135 · cited 1× · see citing works
“Normal deep autoencoders [105] can also acquire a representation of the data by introducing bottleneck constraints and noise, which have been applied in multimodal settings [38, 62]; however, they cannot include various priors for a good representation, such as disentanglement and structural representation.”
Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation
Sugiyama, M., Suzuki, T., & Kanamori, T. (2011) · Annals of the Institute of Statistical Mathematics · doi:10.1007/s10463-011-0343-8 · Cited by 130 · see citing works
The Annual Meeting of the Cognitive Science Society
(2005) · Cognitive Systems Research · doi:10.1016/j.cogsys.2005.02.002 · Cited by 99 · see citing works
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
Shi, Y., Siddharth, N., Paige, B., & Torr, P. H. S. (2019) · arXiv · doi:10.48550/arxiv.1911.03393 · Cited by 87 · cited 13× · see citing works
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
InfluentialGenerative Models of Visually Grounded Imagination
Vedantam, R., Fischer, I., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1705.10762 · Cited by 81 · cited 10× · see citing works
“The partitioned variational autoencoder (PVAE) proposed by Hsu et al. [33] assumes that a joint inference distribution can be factorized as and each of unimodal inference distributions is factorized as The objective of PVAE consists of that of JVAE plus the KL divergence between joint inference and unimodal inference…”
InfluentialAudio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders
Sadeghi, M., Leglaive, S., Alameda-Pineda, X., Laurent Girin, B., & Horaud, R. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3000593 · Cited by 81 · cited 1× · see citing works
“in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audiovisual speech enhancement [106,107] and the acquisition of the joint representation of depth and RGB images [108].”
SCAN: Learning Hierarchical Compositional Visual ConceptsPreprint
Higgins, I., Sonnerat, N., Matthey, L., Pal, A., Burgess, C., Bošnjak, M., Shanahan, M., Botvinick, M., Hassabis, D., & Lerchner, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03389 · Cited by 79 · see citing works
Topic Modeling of Multimodal Data: An Autoregressive Approach
Zheng, Y., Zhang, Y., & Larochelle, H. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.178 · Cited by 60 · cited 1× · see citing works
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodCoordination among neural modules through a shared global workspace
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., & Bengio, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.01197 · Cited by 59 · see citing works
SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model
Nakamura, T., Nagai, T., & Taniguchi, T. (2018) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2018.00025 · Cited by 50 · cited 4× · see citing works
“Neuro-SERKET, an extension of the SERKET framework for including neural networks, can integrate modules of different modalities that perform different inference procedures, such as Gibbs sampling and variational inference.”
InfluentialPrivate-Shared Disentangled Multimodal VAE for Learning of Latent Representations
Lee, M., & Pavlović, V. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00185 · Cited by 48 · see citing works
Taniguchi, T., Nakamura, T., Suzuki, M., Kuniyasu, R., Hayashi, K., Taniguchi, A., Horii, T., & Nagai, T. (2020) · New Generation Computing · doi:10.1007/s00354-019-00084-w · Cited by 44 · cited 3× · see citing works
“Neuro-SERKET, an extension of the SERKET framework for including neural networks, can integrate modules of different modalities that perform different inference procedures, such as Gibbs sampling and variational inference.”
Self-citationAraki, T., Nakamura, T., Nagai, T., Nagasaka, S., Taniguchi, T., & Iwahashi, N. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385812 · Cited by 42 · cited 1× · see citing works
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodSutter, T. M., Daunhawer, I., & Vogt, J. E. (2021) · arXiv · doi:10.48550/arxiv.2105.02470 · Cited by 41 · cited 7× · see citing works
“Models Aggregated inference Missing modalities Modality-specific End-to-end Scalability SCAN [26] × X(single modality) × × [119] × X(single modality) × X [96] × X(single modality) X X bi-VCCA [109] × X(single modality) × X CADA-VAE [76] × X(single modality) × X AVAE [39] × X(single modality) × × X JVAE X × × X X JMVAE…”
InfluentialTaniguchi, T., Yamakawa, H., Nagai, T., Doya, K., Sakagami, M., Suzuki, M., Nakamura, T., & Taniguchi, A. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.02.026 · Cited by 38 · see citing works
Self-citationMultimodal Generative Learning Utilizing Jensen-Shannon Divergence
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000392472 · Cited by 35 · cited 7× · see citing works
“[64] factorize the joint posterior, based on multi-level variational autoencoder [91], as follows:”
InfluentialHolyoak, K. J. (1987) · Science · doi:10.1126/science.236.4804.992 · Cited by 34 · see citing works
Bag of multimodal LDA models for concept formation
Nakamura, T., Nagai, T., & Iwahashi, N. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980324 · Cited by 33 · cited 1× · see citing works
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodMultimodal representation models for prediction and control from partial information
Zambelli, M., Cully, A., & Demiris, Y. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.103312 · Cited by 33 · cited 1× · see citing works
“Zambelli et al. [121] trained a JVAE using five modalities from an iCub robot [55] interacting with a piano keyboard as input, including joint positions and vision, to reconstruct missing modalities and predict their own sensorimotor states and others’ visual trajectories.”
Mining Associated Text and Images with Dual-Wing HarmoniumsPreprint
Xing, E. P., Yan, R., & Hauptmann, A. G. (2012) · arXiv (Cornell University) · doi:10.1184/r1/6607313 · Cited by 31 · see citing works
Robust Unsupervised Audio-Visual Speech Enhancement Using a Mixture of Variational Autoencoders
Sadeghi, M., & Alameda-Pineda, X. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053730 · Cited by 23 · cited 1× · see citing works
“, in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audio-visual speech enhancement [73, 74] and the acquisition of the joint representation of depth and RGB images [11].”
Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
Tang, Q., Wang, W., & Livescu, K. (2017) · Interspeech · doi:10.21437/interspeech.2017-1581 · Cited by 20 · see citing works
Variational Inference via $χ$-Upper Bound MinimizationPreprint
Dieng, A. B., Tran, D., Ranganath, R., Paisley, J. W., & Blei, D. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00328 · Cited by 11 · see citing works
Associative Variational Auto-Encoder with Distributed Latent Spaces and Associators
Jo, D. U., Lee, B., Choi, J., Yoo, H., & Choi, J. Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6778 · Cited by 10 · cited 5× · see citing works
“[86] propose an associative variational autoencoder (AVAE), which has associators, pρ21(z2 | z1) and pρ12(z1 | z2), to map between latent variables of different modalities.”
InfluentialMulti-Modal Generative Models for Learning Epistemic Active Sensing
Korthals, T., Rudolph, D., Leitner, J., Hesse, M., & Rückert, U. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794458 · Cited by 9 · cited 5× · see citing works
“[44] argued that the representation of the multimodal deep generative model can be used as a state as well as a reward in deep reinforcement learning, and showed that the proposed model is effective in multi-agent reinforcement learning with multiple robots.”
InfluentialBridged Variational Autoencoders for Joint Modeling of Images and Attributes
Yadav, R., Sardana, A., Namboodiri, V. P., & Hegde, R. M. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093565 · Cited by 7 · see citing works
Multimodal VAE Active Inference Controller
Meo, C., & Lanillos, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636394 · Cited by 4 · cited 1× · see citing works
“[54] proposed considering the joint angles in a simulated robot arm and the images provided by the camera as different modalities, and using JVAE to learn them and control the robot based on the active inference.”
Method
Flagged first, then by citations
6 of the 93 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 522 · see citing works
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,099 · see citing works
Hagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 192 · see citing works
“For example, text-based models may lack the ability to incorporate visual or emotional context into their understanding, while image-based models might lack textual or semantic understanding [65].”
BackgroundInfluentialVersatile Diffusion: Text, Images and Variations All in One Diffusion Model
Xu, X., Wang, Z.-Y., Zhang, E. J., Wang, K., & Shi, H. (2022) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00713 · Cited by 299 · see citing works
“Multimodal generative tasks involve simultaneous representation learning and generation/synthesis [91], in which representation networks [99, 45, 25, 96, 64, 95] with contrastive loss [66, 15, 1, 93, 94] played an essential role.”
BackgroundLifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 125 · see citing works
Human Motion Generation: A Survey
Zhu, W., Ma, X., Ro, D., Ci, H., Zhang, J., Shi, J., Gao, F., Tian, Q. C., & Wang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3330935 · Cited by 118 · see citing works
“This survey serves as a complement to existing survey papers that focus on human pose estimation [53], [54], motion capture [55], [56], and deep generative models [57], [58], [59].”
BackgroundAn Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 117 · see citing works
“GenAI is an AI technology that can generate various multimedia content [46], [88], [115].”
BackgroundChallenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75 · see citing works
Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Survey
Vu, T.-H., Jagatheesaperumal, S. K., Nguyen, M.-D., Huynh, N. V., Kim, S., & Pham, Q. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3487627 · Cited by 70 · see citing works
“This is achieved by integrating diverse modalities like visual, auditory, and textual information [76].”
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S. M., Özkan, E., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103042 · Cited by 55 · see citing works
“Both research directions have experienced renewed interest in the light of contrastive and self-supervised learning [33, 34] and generative modeling [35].”
BackgroundLi, X., Yuan, Y., Liu, G., Han, Z. Y., & Stouffs, R. M. F. A. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2023.113876 · Cited by 54 · see citing works
Creativity and Machine Learning: A Survey
Franceschelli, G., & Musolesi, M. (2024) · ACM Computing Surveys · doi:10.1145/3664595 · Cited by 65 · see citing works
Multimedia Research Toward the Metaverse
Chen, S. (2022) · IEEE Multimedia · doi:10.1109/mmul.2022.3156185 · Cited by 38 · see citing works
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 50 · see citing works
“Image-text multimodal models [108, 169, 170, 171, 172, 173, 174, 175]are a product of artificial intelligence that can process and understand various types of information, such as text and images, simultaneously.”
BackgroundA systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 35 · see citing works
“These modalities have significantly different data characteristics and forms of representation (J. Chen & Zhang, 2020; Suzuki & Matsuo, 2022; Z. Zhang et al., 2022).”
Emergent communication through Metropolis-Hastings naming game with deep generative models
Taniguchi, T., Yoshida, Y., Matsui, Y., Le Hoang, N., Taniguchi, A., & Hagiwara, Y. (2023) · Advanced Robotics · doi:10.1080/01691864.2023.2260856 · Cited by 33 · see citing works
“Notably, inter-GMM + VAE can be regarded as a variant of multimodal VAEs [47].”
BackgroundSilva, D. D., Kaynak, O., El-Ayoubi, M., Mills, N., Alahakoon, D., & Manic, M. (2024) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2024.3382962 · Cited by 28 · see citing works
Bidirectional visual-tactile cross-modal generation using latent feature space flow model
Fang, Y., Zhang, X., Xu, W., Liu, G., & Zhao, J. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.12.042 · Cited by 21 · see citing works
Granda, B. A. S., Inzhivotkina, Y. V., Apolo, M. F. I., & Fajardo, J. G. U. (2024) · Edutec Revista Electrónica de Tecnología Educativa · doi:10.21556/edutec.2024.89.3251 · Cited by 21 · see citing works
Bayesian structural model updating with multimodal variational autoencoder
Itoi, T., Amishiki, K., Lee, S., & Yaoyama, T. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117148 · Cited by 20 · see citing works
“In this study, a multimodal VAE [25], [26], [27] was employed to develop a versatile framework for Bayesian structural model updating.”
BackgroundMethodA multimodal dynamical variational autoencoder for audiovisual speech representation learning
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106120 · Cited by 20 · see citing works
“VAEs are equipped with encoder and decoder models, resulting in a more stable and faster training process than GANs, which makes them well-suited for multimodal generative modeling (Suzuki & Matsuo, 2022).”
Viswan, V., Shaffi, N., Malathy, E., Selvi, G. C., Kavitha, B. R., Abdesselam, A., Wang, S., Suganthan, P. N., Shezawi, I. A., & Mahmud, M. (2026) · Brain Informatics · doi:10.1186/s40708-025-00291-w · Cited by 17 · see citing works
“Advanced DL architectures—such as Multimodal CNNs [92], RNNs [93], graph neural networks (GNNs) [94], Transformers [95], and generative models [96]— offer a solution by learning hierarchical and cross-modal representations that capture complex interdependencies.”
Duan, W., McNeese, N. J., & Li, L. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701197 · Cited by 17 · see citing works
Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows
Yamada, Y., Colan, J., Davila, A., & Hasegawa, Y. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03101-6 · Cited by 15 · see citing works
A Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17 · see citing works
“In a multimodal setting, an encoder would take one modality as input, and the decoder would generate another modality 33 (Fig.”
Informed machine learning for complex data
Oneto, L., Navarin, N., Micheli, A., Pasa, L., Gallicchio, C., Bacciu, D., & Anguita, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132505 · Cited by 16 · see citing works
Informed Machine Learning: Excess risk and generalization
Oneto, L., Ridella, S., & Anguita, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130521 · Cited by 14 · see citing works
Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation GuidelinesPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2024) · medRxiv · doi:10.1101/2024.10.23.24316003 · Cited by 15 · see citing works
Yang, W., Lü, Y., Yeom, S., & Herbert, D. (2025) · IEEE Access · doi:10.1109/access.2025.3622002 · Cited by 13 · see citing works
“…innovations through various models [1], [2]; others have explored the specific types of GenAI models, such as reviewing Generative adversarial networks (GANs) [3], video GANs [4], multimodal deep GenAI models [5], generative diffusion models [6], and GPT (Generative Pre-trained Transformer) [7].”
Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game
Hoang, N. L., Taniguchi, T., Hagiwara, Y., & Taniguchi, A. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1290604 · Cited by 12 · see citing works
“This development is notably relevant in the context of multimodal deep learning, which combines different data modalities to improve the modeling of diverse data (Suzuki and Matsuo, 2022).”
InfluentialZou, S., Wang, X., Tao, Y., Zeng, K., Li, G., & Xie, X. (2025) · The Journal of the Acoustical Society of America · doi:10.1121/10.0034831 · Cited by 10 · see citing works
MDLDroid: Multimodal Deep Learning Based Android Malware Detection
Singh, N., & Tripathy, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49099-6_10 · Cited by 9 · see citing works
SMOTE-enhanced XGBoost for rapid seismic damage assessment of bridge portfolios
Di, S., Wang, Y.-Y., Yang, D. Y., Liu, Y., Zhang, J., & Zheng, W. (2025) · Soil Dynamics and Earthquake Engineering · doi:10.1016/j.soildyn.2025.109712 · Cited by 8 · see citing works
A method for generating personalized learning content based on AIGC
Hu, Z. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101331 · Cited by 8 · see citing works
Zhang, C., Li, X., Bai, W., Sangaiah, A. K., & Li, D. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3524248 · Cited by 8 · see citing works
“The second challenge pertains to the establishment of a stable MG model, which may involve constructs such as multigranulation rough set models [6], multi-scale data analysis models [7], multi-modal models [8], and so forth.”
Multi-view-AE: A Python package for multi-view autoencoder models
Aguila, A. L., Jayme, A., Montaña-Brown, N., Heuveline, V., & Altmann, A. (2023) · The Journal of Open Source Software · doi:10.21105/joss.05093 · Cited by 8 · see citing works
Xiong, Y., Chai, C., Gan, Y. S., & Chong, H. (2025) · Innovative Infrastructure Solutions · doi:10.1007/s41062-025-01975-2 · Cited by 7 · see citing works
“Therefore, several studies are focused on adapting the GAN framework to overcome the limitation of accepting only a single modality [40].”
Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis
Lüpke, S., Yeganeh, Y., Adeli, E., Navab, N., & Farshad, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-84525-3_17 · Cited by 7 · see citing works
Zeus: zero-shot LLM instruction for union segmentation in multimodal medical imaging
Dai, S., Ye, K., Liu, G., Tang, H., & Zhan, L. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02742-6 · Cited by 7 · see citing works
“Besides, multimodal learning shows great importance in autonomous driving [55], generative model [44], healthcare [36, 63, 64].”
SwarmDiff: Swarm Robotic Trajectory Planning in Cluttered Environments via Diffusion Transformer
Ding, K., Jiao, C., Hu, Y., Zhou, K., Wu, P., Mu, Y., & Liu, C. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00400 · Cited by 7 · see citing works
Nadeem, M., Sohail, S. S., Madsen, D. Ø., Alzahrani, A. I., Ser, J. D., & Muhammad, K. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536937 · Cited by 6 · see citing works
“Interested readers refer to [45], [46], [47] for the same.”
Bimodal PET/MRI generative reconstruction based on VAE architectures
Gautier, V., Bousse, A., Sureau, F. C., Comtat, C., Maxim, V., & Sixou, B. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad9133 · Cited by 6 · see citing works
Bouatmane, A., Daaif, A., Bousselham, A., Bouihi, B., & Bouattane, O. (2025) · IEEE Access · doi:10.1109/access.2025.3556700 · Cited by 6 · see citing works
“Recent advancements in deep learning and multimodal [2], [3], [6], [9], [10], [19], [47] data fusion offer new opportunities to address these challenges.”
ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model
Wang, Q., Fan, C., Jia, T., Han, Y., & Wu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29486 · Cited by 6 · see citing works
Concept formation through multimodal integration using multimodal BERT and VQ-VAE
Miyazawa, K., & Nagai, T. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2141583 · Cited by 6 · see citing works
“Furthermore, various multimodal deep generative models have been proposed to handle multimodal information [22].”
BackgroundFrom imaging to omics: deep learning is bridging MRI and liquid biopsy in bone tumor diagnosis
Wang, F., Chen, J., Zheng, L., & Huang, X. (2026) · Journal of bone oncology · doi:10.1016/j.jbo.2026.100753 · Cited by 5 · see citing works
“One of the most pressing issues in multimodal modeling is the heterogeneity of the input data [222,230,231].”
Empowering multimodal analysis with visualization: A survey
Wang, J., Deng, Z., Deng, D., Wang, X., Sheng, R., Cai, Y., & Qu, H. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100748 · Cited by 4 · see citing works
Hierarchical multi-modal fusion architecture search for stock market forecasting
Lee, K., Kim, Y. J., Kim, M., & Kim, H. Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113581 · Cited by 5 · see citing works
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Ge, J., Subramanian, S., Darrell, T., & Li, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.75 · Cited by 4 · see citing works
Ikegawa, Y., Fukuma, R., Sugano, H., Oshino, S., Tani, N., Tamura, K., Iimura, Y., Suzuki, H., Yamamoto, S., Fujita, Y., et al. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad417a · Cited by 4 · see citing works
Taniguchi, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35382-6_2 · Cited by 4 · see citing works
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 3 · see citing works
Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
Zheng, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768625 · Cited by 3 · see citing works
Object-Aware Image Augmentation for Audio-Visual Zero-Shot Learning
Dong, Y., Chen, S., Duan, B., Ding, W., Wang, Y., & You, X. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3485624 · Cited by 3 · see citing works
“If we have a method to embed all modalities into a shared representation and generate modalities from it, we can achieve generation between any modalities through this space [63].”
Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
Šejnová, G., Vavrečka, M., & Štěpánová, K. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802160 · Cited by 3 · see citing works
“These models have emerged as a powerful extension of the unimodal Variational Autoencoder and are suitable for mapping multiple modalities into a unified joint space, allowing for the generation of one modality from another [9].”
A Multimodal Hierarchical Variational Autoencoder for Saliency Detection
Yu, Z., Zhang, J., & Barnes, N. M. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191461 · Cited by 3 · see citing works
“…properties such as its flexibility in conditioned generation with meaningful encoding, its efficient sampling strategy using amortized variational inference [59] as well as its explicitness in reflection of heterogeneity of multimodal data via differences in probabilistic distributions [60].”
A Multi-Modal Retrieval Model for Mathematical Expressions Based on ConvNeXt and Hesitant Fuzzy Set
Li, R., Wang, J., & Tian, X. (2023) · Electronics · doi:10.3390/electronics12204363 · Cited by 3 · see citing works
“Current research in multimodal studies is expanding rapidly, exploring the integration of various data types and modes to gain a more comprehensive understanding of complex phenomena [4,5].”
BackgroundSu, B., Wang, Y., Schiavazzi, D. E., & Liu, F. (2023) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v1i1.27495 · Cited by 3 · see citing works
Towards Generated Image Provenance Analysis via Conceptual-Similar-Guided-SLIP Retrieval
Xia, X., Wang, L., Sun, J., & Nakagawa, A. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3388958 · Cited by 3 · see citing works
“R ECENTLY, generative AI has become a prominent research area [1], [2]. synthesize Image generation models can synthesize realistic images that are indistinguishable from natural ones.”
BackgroundAn Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v1 · Cited by 3 · see citing works
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v2 · Cited by 2 · see citing works
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271.v3 · Cited by 2 · see citing works
Noda, K., Soda, T., & Yamashita, Y. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1330512 · Cited by 1 · see citing works
“Additionally, the concept of multimodal learning has been applied in the fields of machine learning and deep neural networks (Baltrušaitis et al., 2019; Suzuki and Matsuo, 2022).”
BackgroundWang, Q., Fan, C., Jia, T., Han, Y., & Wu, X. (2023) · International Conference on Neuromorphic Computing (ICNC) · doi:10.1109/icnc59488.2023.10462792 · Cited by 1 · see citing works
“Besides, deep generative models, including variational autoencoders (VAEs) [14], [15], [29], generative adversarial networks (GAN) [16]–[18], and so on, also show their excellent performance because of their superiority in feature representation [19].”
BackgroundMeasuring Similarity of Dual-Modal Academic Data Based on Multi-Fusion Representation Learning
Zhang, L., Gao, Q., Liu, M., Gu, Z., & Lang, B. (2024) · IEEE Access · doi:10.1109/access.2024.3427731 · Cited by 1 · see citing works
“Retrieving relevant texts via a single image, or retrieving relevant images through text or a combination of text and images has become an interesting and challenging problem [12], [13], [14], [15], [16], [17], [18].”
BackgroundLiang, B., Guo, J., Liu, S. Q., Zong, C., Cheng, Y., & Chen, J. (2025) · Optik · doi:10.1016/j.ijleo.2025.172261 · Cited by 1 · see citing works
Multimodal Romanian language resources and tools: challenges and perspectives
Mitrofan, M., Irimia, E., & Păiș, V. (2025) · Discover Data · doi:10.1007/s44248-025-00060-4 · Cited by 1 · see citing works
Virtual Trainer for Learning Mexican Sign Language Using Video Similarity Analysis
Rivera-Cervantes, F. D. J., Esparza, D. M. C., Terven, J. R., Romero-González, J.-A., González-Rodríguez, J.-R., Ibarra-Corona, M. A., & Ramírez-Pedraza, A. (2025) · Technologies · doi:10.3390/technologies13120540 · Cited by 1 · see citing works
Effectively Obtaining Acoustic, Visual, and Textual Data from Videos
León, J. E., & Carrasco, M. (2025) · Applied Sciences · doi:10.3390/app152312654 · Cited by 2 · see citing works
Gao, Z., Han, Z., Fu, Y., Lv, H., Li, M., Zhao, X., & Zhu, J. (2026) · Polymers · doi:10.3390/polym18020203 · Cited by 1 · see citing works
“Common fusion strategies include early fusion, mid-level fusion, and late fusion [34].”
Artificial Intelligence-Based Digital Transformation
Volpentesta, T. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80592-096-020261002 · Cited by 1 · see citing works
Shen, C., & Namiki, A. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.118384 · Cited by 1 · see citing works
Symbol Emergence as a Decentralized Bayesian Inference
Taniguchi, T. (2022) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.40.883 · Cited by 1 · see citing works
Eder, M., & Steinbauer-Wagner, G. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10609988 · Cited by 1 · see citing works
“Using deep multimodal variational autoencoders (DMVAEs) [8], [9], our framework combines robot-specific properties to represent a specific robot and environmental information from aerial imagery to improve traversability predictions, with the goal of increasing the applicability of the framework to…”
Instruction-Driven 3D Facial Expression Generation and Transition
Vo, A. H., Kim, T., Jin, H., Choi, S.-M., & Kim, Y.-G. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3565929 · Cited by 1 · see citing works
“Furthermore, integrating feature representations from different modalities minimizes ambiguity in redundant resources and effectively handles missing modalities, particularly in generative models, by improving inter-modality relations [27], [28].”
Ayub, S., Gorman, J. L., Chen, E., Jackson, H. W., Selega, A., & Campbell, K. R. (2026) · Bioinformatics Advances · doi:10.1093/bioadv/vbag010 · Cited by 1 · see citing works
“The latent space of a VAE captures the salient features that explain the variation in high dimensional data (Kingma and Welling 2014, Suzuki and Matsuo 2022).”
Robot Control based on World Models and Predictive Learning
Taniguchi, T., Okada, M., Maeyama, K., Okumura, R., & Kinose, A. (2022) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.40.790 · Cited by 1 · see citing works
Marcinkevičs, R., Wolfertstetter, P. R., Klimiene, U., Özkan, E., Chin-Cheong, K., Paschke, A., Zerres, J., Denzinger, M., Niederberger, D., Wellmann, S., et al. (2023) · arXiv (Cornell University) · doi:10.3929/ethz-b-000601130 · Cited by 1 · see citing works
An Overview on Generative AI at Scale with Edge-Cloud ComputingPreprint
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · doi:10.36227/techrxiv.23272271 · Cited by 1 · see citing works
MultiVae: A Python package for Multimodal Variational Autoencoders on Partial Datasets.
Senellart, A., Chadebec, C., & Allassonnière, S. (2025) · The Journal of Open Source Software · doi:10.21105/joss.07996 · Cited by 1 · see citing works
“See Kingma & Welling (2014) to learn more about the VAE framework and Suzuki & Matsuo (2022) for a survey on multimodal VAEs.”
Audiovisual speech representation learning applied to emotion recognition
Sadok, S. (2024) · theses.fr (ABES) · doi:10.70675/0f15a36fzecf8z4cffz94e7z120e7d8fd39d · Cited by 1 · see citing works
A multimodal dynamical variational autoencoder for audiovisual speech representation learningPreprint
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03582 · Cited by 1 · see citing works
“VAEs are equipped with encoder and decoder models, resulting in a more stable and faster training process than GANs, which makes them well-suited for multimodal generative modeling (Suzuki & Matsuo, 2022).”
集合的予測符号化に基づく言語と認知のダイナミクス ～記号創発ロボティクスの新展開に向けて～Preprint
Taniguchi, T. (2023) · doi:10.31234/osf.io/4a6j7 · Cited by 1 · see citing works
Multimodal Outputs for the Workplace From Generative AI
Hai-Jew, S. (2024) · Advances in multimedia and interactive technologies book series · doi:10.4018/979-8-3693-2927-6.ch008
VAD Emotion Control in Visual Art Captioning via Disentangled Multimodal Representation
Ueda, R., Narimatsu, H., Miyao, Y., & Kumano, S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00020
“In the field of machine learning and representation learning, various MMRL methods have been proposed [16].”
Ahmed, B., Daaif, A., Bousselham, A., & Bouattane, O. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-88304-0_11
From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning
Xie, H., Maharjan, R. S., Tavella, F., & Cangelosi, A. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7033-8_6
Mahaseni, B., & Khan, N. (2025) · Empathic Computing · doi:10.70401/ec.2025.0010
Gao, Y. F., Shao, M., Zhao, C., Du, R., Wu, J., & Wu, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5388880
VAE-based Multivariate Nowcasting of the Japanese Economy
Ueda, T., Izumi, K., & Murayama, Y. (2025) · Computational Economics · doi:10.1007/s10614-025-11140-7
“For example, in supply chain analysis, it is considered crucial to handle micro-level, diverse distributions by product or process, making the explicit introduction of multimodality (Suzuki & Matsuo, 2022) potentially necessary.”
Langer, C., Georgie, Y. K., Porohovoj, I., Hafner, V. V., & Ay, N. (2025) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl63968.2025.11204413
“A more exhaustive overview over multimodal deep generative networks can be found in [12], where the authors discuss 22 different models and their properties in detail.”
Generative Künstliche Intelligenz im Marketing
Decker, R. (2025) · Springer eBooks · doi:10.1007/978-3-658-46709-8_69-1
Implementation Paths of Generative AI in Multimodal Learning
Tan, B. (2025) · Journal of Computer Science and Artificial Intelligence · doi:10.54097/rwr0vn28
AI-Driven Motion Capture Data Recovery: A Comprehensive Review and Future Outlook
Almaleh, A., Ushaw, G., & Davison, R. (2025) · Sensors · doi:10.3390/s25247525
¿Gerentes y administradores exclusivamente humanos o ciberadministradores? Virtus in medium est
Dinis, M. (2025) · Revista e-mercatoria · doi:10.18601/16923960.v25n1.06
Gao, Z., Han, Z., Fu, Y., Zhao, X., Lv, H., Li, M., & Zhu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5394703
Generative AI: Principles, Potentials, and Ethical Dimensions
Cheshmehzangi, A. (2025) · Urban sustainability · doi:10.1007/978-981-95-4766-1_3
Journey Through Language: Models and Prompt Engineering
Akerkar, R. A. (2026) · Springer eBooks · doi:10.1007/978-3-031-91084-5_7
Pioneer: A Method for Enhancing GPU Utilization in Workloads Through Potential-Aware Prioritization
Guo, Q., Zhao, W., Xie, Y., & Zhao, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5640-3_15
Bad Designs by Good Talkers: Chatbots Failing to Architect Audio Encoders for Image Synthesis
León, J. E., Carrasco, M., & Peters, A. A. (2026) · IEEE Access · doi:10.1109/access.2026.3668130
“Audio-to-image generation remains relatively underexplored, despite evidence that audio signals carry rich semantic information that could guide visual content creation [21], [22], [44].”
Influential
- Authors
- Suzuki, M., & Matsuo, Y.
- Affiliations
- The University of Tokyo (JP)
- Corresponding author
- Suzuki, Masahiro
- Venue
- Advanced Robotics
- Publisher
- Informa UK Limited
- Published
- 2022-02-21
- Volume
- 36
- Issue
- 5-6
- Pages
- 261-278
- ISSN
- 0169-1864 (ISSN-L)
- 1568-5535
- OpenAlex
- W4213299273
- Cited by
- 109
- References
- 93
- Citations by year
- 2026: 18 · 2025: 38 · 2024: 27 · 2023: 22 · 2022: 4
- FWCI
- 7.45
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Advanced Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Energy and Industrial Technology Development Organization (JPNP16007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal learning, deep generative models, variational autoencoder, cross-modal generation, shared representations, multimodal generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
