The DOI matches its Crossref record; 88 of the 93 references it cites were found in a registry or an index.
A survey of multimodal deep generative models
Open accessAbstract
Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e. generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on the position of the layer for the multimodal interaction, the data fusion architectures can be classified into early, intermediate, and later fusion [50].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [50]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Goyal et al. [20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 3×
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2×
“In addition to these, cycleconsistency loss is used in image-to-image translation, which is the task of transforming an image belonging to one domain to another domain [57,58].”
BackgroundPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Srivastava and Salakhutdinov [37] propose a multimodal generative model based on deep belief networks [38,39], which has since been used in various multimodal applications [40,41].”
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2×
“In general, a “good representation” is one that can be used for various tasks while retaining the characteristics of the original input [6].”
BackgroundAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
A synaptic model of memory: long-term potentiation in the hippocampus
Bliss, T. V. P., & Collingridge, G. L. (1993) · Nature · doi:10.1038/361031a0 · Cited by 11,566
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 1×
“A common multimodal dataset used in many multimodal deep generative models [33, 88, 88, 89, 91, 104, 118] is the CelebA dataset [52], which contains face images and their attributes.”
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260 · cited 4×
“To consider this, suppose that we are given two datasets, In transfer learning, the domain of X 1 is defined by a pairing of the data feature space X 1 and the marginal distribution p ( X 1 ) [67, 110].”
InfluentialRELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 6,041
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · cited 7×
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
BackgroundMethodInfluentialTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“Wu and Goodman [65] consider unimodal inference as ‘expert’ and propose approximating joint inference using the following product-of-experts (PoE) [89] form:”
BackgroundReading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261
(1988) · Cited by 3,656
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 2×
“However, most of these are conditional models [35, 123], i.”
2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,846
(2011) · Cited by 2,752
2008 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2008) · doi:10.1109/iros13647.2008 · Cited by 2,203
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · cited 1×
“The first purpose is sometimes referred to as multimodal representation learning [50], and besides generative approaches, there are studies based on discriminative approaches such as contrastive learning [1, 97, 100].”
Background2014 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2014) · doi:10.1109/iros20755.2014 · Cited by 1,309
Diverse Image-to-Image Translation via Disentangled Representations
Lee, H.-Y., Tseng, H.-Y., Huang, J., Singh, M., & Yang, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_3 · Cited by 1,300 · cited 2×
“Although some of these models consider shared latent variables in different domains [59,60], they assume that the input is two domain images of the same size, not heterogeneous modalities; therefore, we do not include them in this survey.”
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 3×
“In a multimodal learning framework, the term modality originally referred to each scheme and situation in which data are perceived [5, 45], i.”
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder
Park, D., Hoshi, Y., & Kemp, C. C. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2801475 · Cited by 1,192 · cited 1×
“[111] introduced an LSTM-based JVAE to detect anomalous feeding executions given 17 sensory signals from five types of sensors on a PR2 robot.”
Blei, D. M., & Jordan, M. I. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860460 · Cited by 1,078 · cited 2×
“Other research involves extending topic models such as latent Dirichlet allocation (LDA) [9] to multimodal input [8, 58].”
BackgroundMethodHierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968
Hinton, G. E. (2009) · Scholarpedia · doi:10.4249/scholarpedia.5947 · Cited by 933 · cited 1×
“Xing et al. [116] build a joint model of images and texts using a linear RBM. Srivastava et al. [82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891 · cited 1×
“[76] propose CADA-VAE for generalized zero-shot learning [70, 114], which is used to embed different modalities, namely image features and class attributes, into a shared representation.”
A survey of multi-view machine learning
Sun, S. (2013) · Neural Computing and Applications · doi:10.1007/s00521-013-1362-6 · Cited by 878
Pyro: Deep Universal Probabilistic ProgrammingPreprint
Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F. H., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., & Goodman, N. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09538 · Cited by 853 · cited 2×
“Probabilistic modeling languages, such as Pyro [7], and deep generative modeling libraries, such as Pixyz [90], might help to address this difficulty.”
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · cited 2×
“[99] conducted experiments on real-world time-series multimodal datasets such as CMU-MOSI [120], which includes three modalities: language, acoustic, and visual.”
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · cited 1×
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
Learning to Estimate 3D Hand Pose from Single RGB Images
Zimmermann, C., & Brox, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.525 · Cited by 763 · cited 1×
“[39] worked on hand pose estimation using RGB images and keypoints of each hand in the Rendered Hand pose dataset [126] as multimodal data.”
MethodGeneralized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00844 · Cited by 747 · cited 6×
“In CADA-VAE [76], the authors used the Caltech-UCSD Birds (CUB)-200 dataset [111], which contains bird images and their attributes, as a benchmark for zero-shot learning.”
InfluentialA Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 657 · cited 1×
“The first purpose is sometimes referred to as multimodal representation learning [50], and besides generative approaches, there are studies based on discriminative approaches such as contrastive learning [1, 97, 100].”
Tolstikhin, I., Bousquet, O., Gelly, S., & Schölkopf, B. (2018) · MPG.PuRe (Max Planck Society) · Cited by 602
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., & Díaz-Rodríguez, N. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.12.004 · Cited by 593 · cited 1×
“To deal with this difficulty, it might be important to use techniques such as transfer learning and continuous learning [123,124], given that humans have the ability to act in unknown environments based on their memories of other environments and the ability to continuously learn new things from the past.”
BackgroundGiorgio Metta, B., Sandini, G., Vernon, D., Natale, L., & Nori, F. (2008) · Workshop on Performance Metrics for Intelligent Systems (PerMIS) · doi:10.1145/1774674.1774683 · Cited by 548
Thrun, S., & Mitchell, T. M. (1995) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(95)00004-y · Cited by 536 · cited 1×
“To deal with this difficulty, it might be important to use techniques such as transfer learning and continuous learning [49, 95], given that humans have the ability to act in unknown environments based on their memories of other environments and the ability to continuously learn new things from the past.”
BackgroundA Review of Generalized Zero-Shot Learning Methods
Pourpanah, F., Abdar, M., Luo, Y., Zhou, X., Wang, R., Lim, C. P., Wang, X., & Wu, Q. M. J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3191696 · Cited by 521 · cited 1×
“[72] propose CADA-VAE for generalized zero-shot learning [84,85], which is used to embed differentmodalities, namely image features and class attributes, into a shared representation.”
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · cited 1×
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
BackgroundSliced and Radon Wasserstein Barycenters of Measures
Bonneel, N., Rabin, J., Gabriel Peyré, A., & Pfister, H. (2014) · Journal of Mathematical Imaging and Vision · doi:10.1007/s10851-014-0506-3 · Cited by 471 · cited 1×
“They optimize the encoder and decoder in the VAE framework while simultaneously optimizing the slicedWasserstein distance [82] between the embeddings in each modality, z1 and z ′ 2, and the loss in class label prediction from each embedding.”
MethodAssociation for the Advancement of Artificial Intelligence
Crosby, M., & Petrick, R. P. A. (2014) · Cited by 462
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · cited 1×
“[82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
BackgroundCodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM
Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., & Davison, A. J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00271 · Cited by 401 · cited 1×
“, in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audio-visual speech enhancement [73, 74] and the acquisition of the joint representation of depth and RGB images [11].”
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · cited 10×
“Multimodal Factorization Model (MFM) [99] includes a label y as well as a multimodal input X as observation variables, and assumes the following generative model: where F s = { f sm } M m =1 and f c are modality-specific and modality-invariant latent factors, respectively.”
InfluentialMulti-source Deep Learning for Human Pose Estimation
Ouyang, W., Chu, X., & Wang, X. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.299 · Cited by 273 · cited 1×
“Furthermore, multimodal DBMs are used in a variety of multimodal applications [66, 68, 86].”
BackgroundSelf-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 258
Recurrent Independent MechanismsPreprint
Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., & Schölkopf, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10893 · Cited by 245 · cited 1×
“[20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
BackgroundA Comprehensive Survey on Cross-modal RetrievalPreprint
Wang, K., Yin, Q., Wang, W., Wu, S., & Wang, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06215 · Cited by 230
Multimodal integration learning of robot behavior using deep neural networks
Noda, K., Arie, H., Suga, Y., & Ogata, T. (2014) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.03.003 · Cited by 201 · cited 1×
“Multimodal learning is especially important for robots that need to operate properly in the real world, because they need to make sense of the world based on the various types of information they receive through their onboard sensors [3].”
BackgroundTopic regression multi-modal Latent Dirichlet Allocation for image annotation
Putthividhy, D., Attias, H. T., & Nagarajan, S. S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540000 · Cited by 195
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
Shi, Y., Siddharth, N., Paige, B., & Torr, P. H. S. (2019) · arXiv · doi:10.48550/arxiv.1911.03393 · Cited by 186 · cited 13×
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
InfluentialAudio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. E. D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 185 · cited 1×
“[82] propose a multimodal generative model based on deep belief networks [28, 29], which has since been used in various multimodal applications [34, 41].”
Transfer Learning on Heterogenous Feature Spaces via Spectral Transformation
Shi, X., Liu, Q., Fan, W., Yu, P. S., & Zhu, R. (2010) · Proceedings · doi:10.1109/icdm.2010.65 · Cited by 185 · cited 1×
“Various types of modalities in our world–such as vision, sound, and smell–are said to be heterogeneous [77, 110].”
BackgroundIvanovic, B., Leung, K., Schmerling, E., & Pavone, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3043163 · Cited by 154 · cited 1×
“These models are also sometimes referred to as multimodal generative models [36], but we do not include them in multimodal generative models in this paper because they do not acquire a shared representation and can only generate crossmodally in one direction.”
BackgroundDeep Multimodal Learning for Affective Analysis and Retrieval
Pang, L., Zhu, S., & Ngo, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482228 · Cited by 153 · cited 1×
“Furthermore, multimodal DBMs are used in a variety of multimodal applications [66, 68, 86].”
BackgroundJaques, N., Taylor, S., Sano, A., & Picard, R. W. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273601 · Cited by 135 · cited 1×
“Normal deep autoencoders [105] can also acquire a representation of the data by introducing bottleneck constraints and noise, which have been applied in multimodal settings [38, 62]; however, they cannot include various priors for a good representation, such as disentanglement and structural representation.”
Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation
Sugiyama, M., Suzuki, T., & Kanamori, T. (2011) · Annals of the Institute of Statistical Mathematics · doi:10.1007/s10463-011-0343-8 · Cited by 130
SCAN: Learning Hierarchical Compositional Visual ConceptsPreprint
Higgins, I., Sonnerat, N., Matthey, L., Pal, A., Burgess, C., Bošnjak, M., Shanahan, M., Botvinick, M., Hassabis, D., & Lerchner, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03389 · Cited by 122
The Annual Meeting of the Cognitive Science Society
(2005) · Cognitive Systems Research · doi:10.1016/j.cogsys.2005.02.002 · Cited by 99
Generative Models of Visually Grounded Imagination
Vedantam, R., Fischer, I., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1705.10762 · Cited by 81 · cited 10×
“The partitioned variational autoencoder (PVAE) proposed by Hsu et al. [33] assumes that a joint inference distribution can be factorized as and each of unimodal inference distributions is factorized as The objective of PVAE consists of that of JVAE plus the KL divergence between joint inference and unimodal inference…”
InfluentialAudio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders
Sadeghi, M., Leglaive, S., Alameda-Pineda, X., Laurent Girin, B., & Horaud, R. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3000593 · Cited by 81 · cited 1×
“in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audiovisual speech enhancement [106,107] and the acquisition of the joint representation of depth and RGB images [108].”
Topic Modeling of Multimodal Data: An Autoregressive Approach
Zheng, Y., Zhang, Y., & Larochelle, H. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.178 · Cited by 60 · cited 1×
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodPrivate-Shared Disentangled Multimodal VAE for Learning of Latent Representations
Lee, M., & Pavlovic, V. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00185 · Cited by 59
Coordination among neural modules through a shared global workspace
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., & Bengio, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.01197 · Cited by 59
SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model
Nakamura, T., Nagai, T., & Taniguchi, T. (2018) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2018.00025 · Cited by 50 · cited 4×
“Neuro-SERKET, an extension of the SERKET framework for including neural networks, can integrate modules of different modalities that perform different inference procedures, such as Gibbs sampling and variational inference.”
InfluentialTaniguchi, T., Nakamura, T., Suzuki, M., Kuniyasu, R., Hayashi, K., Taniguchi, A., Horii, T., & Nagai, T. (2020) · New Generation Computing · doi:10.1007/s00354-019-00084-w · Cited by 44 · cited 3×
“Neuro-SERKET, an extension of the SERKET framework for including neural networks, can integrate modules of different modalities that perform different inference procedures, such as Gibbs sampling and variational inference.”
Self-citationAraki, T., Nakamura, T., Nagai, T., Nagasaka, S., Taniguchi, T., & Iwahashi, N. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385812 · Cited by 42 · cited 1×
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodSutter, T. M., Daunhawer, I., & Vogt, J. E. (2021) · arXiv · doi:10.48550/arxiv.2105.02470 · Cited by 41 · cited 7×
“Models Aggregated inference Missing modalities Modality-specific End-to-end Scalability SCAN [26] × X(single modality) × × [119] × X(single modality) × X [96] × X(single modality) X X bi-VCCA [109] × X(single modality) × X CADA-VAE [76] × X(single modality) × X AVAE [39] × X(single modality) × × X JVAE X × × X X JMVAE…”
InfluentialTaniguchi, T., Yamakawa, H., Nagai, T., Doya, K., Sakagami, M., Suzuki, M., Nakamura, T., & Taniguchi, A. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.02.026 · Cited by 39
Self-citationMultimodal Generative Learning Utilizing Jensen-Shannon Divergence
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000392472 · Cited by 35 · cited 7×
“[64] factorize the joint posterior, based on multi-level variational autoencoder [91], as follows:”
InfluentialHolyoak, K. J. (1987) · Science · doi:10.1126/science.236.4804.992 · Cited by 34
Bag of multimodal LDA models for concept formation
Nakamura, T., Nagai, T., & Iwahashi, N. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980324 · Cited by 33 · cited 1×
“The multimodal LDA has similar advantages as in the case of DBMs and various studies have extended this model [2, 57, 59, 71, 124]; however, they are all based on topic models and cannot directly handle data with high dimensionality and complex structures such as images, thereby requiring preprocessing such as extract…”
MethodMultimodal representation models for prediction and control from partial information
Zambelli, M., Cully, A., & Demiris, Y. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.103312 · Cited by 33 · cited 1×
“Zambelli et al. [121] trained a JVAE using five modalities from an iCub robot [55] interacting with a piano keyboard as input, including joint positions and vision, to reconstruct missing modalities and predict their own sensorimotor states and others’ visual trajectories.”
Mining Associated Text and Images with Dual-Wing HarmoniumsPreprint
Xing, E. P., Yan, R., & Hauptmann, A. G. (2012) · arXiv (Cornell University) · doi:10.1184/r1/6607313 · Cited by 31
Multimodal VAE Active Inference Controller
Meo, C., & Lanillos, P. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636394 · Cited by 24 · cited 1×
“[54] proposed considering the joint angles in a simulated robot arm and the images provided by the camera as different modalities, and using JVAE to learn them and control the robot based on the active inference.”
MethodRobust Unsupervised Audio-Visual Speech Enhancement Using a Mixture of Variational Autoencoders
Sadeghi, M., & Alameda-Pineda, X. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053730 · Cited by 23 · cited 1×
“, in a setting where not all modalities are inferred and generated, various multimodal tasks are performed with VAEs, such as audio-visual speech enhancement [73, 74] and the acquisition of the joint representation of depth and RGB images [11].”
Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
Tang, Q., Wang, W., & Livescu, K. (2017) · Interspeech · doi:10.21437/interspeech.2017-1581 · Cited by 20
Associative Variational Auto-Encoder with Distributed Latent Spaces and Associators
Jo, D. U., Lee, B., Choi, J., Yoo, H., & Choi, J. Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6778 · Cited by 11 · cited 5×
“[86] propose an associative variational autoencoder (AVAE), which has associators, pρ21(z2 | z1) and pρ12(z1 | z2), to map between latent variables of different modalities.”
InfluentialVariational Inference via $χ$-Upper Bound MinimizationPreprint
Dieng, A. B., Tran, D., Ranganath, R., Paisley, J. W., & Blei, D. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00328 · Cited by 11
Multi-Modal Generative Models for Learning Epistemic Active Sensing
Korthals, T., Rudolph, D., Leitner, J., Hesse, M., & Rückert, U. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794458 · Cited by 9 · cited 5×
“[44] argued that the representation of the multimodal deep generative model can be used as a state as well as a reward in deep reinforcement learning, and showed that the proposed model is effective in multi-agent reinforcement learning with multiple robots.”
InfluentialBridged Variational Autoencoders for Joint Modeling of Images and Attributes
Yadav, R., Sardana, A., Namboodiri, V. P., & Hegde, R. M. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093565 · Cited by 7
Research works printed but not linked: 36
- Stein BE, 1993, The merging of the senses
- Kingma DP, 2014, International Conference on Learning Representations
- van den Oord A, Dieleman S, Zen H, et al. Wavenet: a generative model for raw audio; 2016. Preprint arXiv:1609.03499. · arXiv
- Van Oord A, 2016, International Conference on Machine Learning
- Rezende D, 2015, International Conference on Machine Learning
- Kingma DP, 2018, Advances in Neural Information Processing Systems
- Xu C, Tao D, Xu C. A survey on multi-view learning; 2013. Preprint arXiv:1304.5634. · arXiv
- Tsai Y-HH, 2021, International Conference on Learning Representations
- Sohn K, 2015, Adv Neural Inf Process Syst, 28, 3483
- Tschannen M, Bachem O, Lucic M. Recent advances in autoencoder-based representation learning; 2018. Preprint arXiv:1812.05069. · arXiv
- Srivastava N, Salakhutdinov R. Learning representations for multimodal data with deep belief nets. In: International Conference on Machine Learning Workshop, Vol. 79; 2012. p. 3.
- Sohn K, 2014, Adv Neural Inf Process Syst, 27, 2141
- 10.1162/jmlr.2003.3.4-5.993 · DOI
- Higgins I, Matthey L, Pal A, et al. beta-vae: learning basic visual concepts with a constrained variational framework. ICLR; 2016.
- Vincent P, 2010, J Machine Learning Res, 11, 3371
- Liu M-Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks; 2017. Preprint arXiv:1703.00848. · arXiv
- Wu M, Goodman N. Multimodal generative models for compositional representation learning; 2019. Preprint arXiv:1912.05075. · arXiv
- Suzuki M, Nakayama K, Matsuo Y. Improving bi-directional generation between different modalities with variational autoencoders; 2018. Preprint arXiv:1801.08702. · arXiv
- Daunhawer I, 2021, Pattern Recognit, 12544, 459
- Wu M, 2018, Advances in Neural Information Processing Systems
- Suzuki M, Nakayama K, Matsuo Y. Joint multimodal learning with deep generative models; 2016. Preprint arXiv:1611.01891. · arXiv
- Tian Y, Engel J. Latent translation: crossing modalities by bridging generative models; 2019. Preprint arXiv:1902.08261. · arXiv
- Hsu W-N, Glass J. Disentangling by partitioning: a representation learning framework for multimodal sensory data; 2018. Preprint arXiv:1805.11264. · arXiv
- Shi Y, 2021, International Conference on Learning Representations
- 10.1609/aaai.v34i07.6778 · DOI
- Wang D, 2015, International Joint Conference on Artificial Intelligence
- Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders; 2015. Preprint arXiv:1509.00519. · arXiv
- Bouchacourt D, 2018, AAAI Conference on Artificial Intelligence
- Maddison CJ, 2017, International Conference on Learning Representations
- Jang E, 2017, International Conference on Learning Representations
- van den Oord A, Li Y, Vinyals O. Representation learning with contrastive predictive coding; 2018. Preprint arXiv:1807.03748. · arXiv
- Xiao H, Rasul K, Vollgraf R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017. Preprint arXiv:1708.07747 · arXiv
- Ha D, Schmidhuber J. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems; 2018; Montreal, Canada.
- Hafner D, 2019, International Conference on Machine Learning
- Hafner D, 2019, International Conference on Learning Representations
- Suzuki M, Kaneko T, Matsuo Y. Pixyz: a library for developing deep generative models; 2021. Preprint arXiv: 2107.13109. · arXiv
Cited by
Sorted by citations
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 590
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
Hagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 227
“For example, text-based models may lack the ability to incorporate visual or emotional context into their understanding, while image-based models might lack textual or semantic understanding [65].”
BackgroundInfluentialVersatile Diffusion: Text, Images and Variations All in One Diffusion Model
Xu, X., Wang, Z., Zhang, E., Wang, K., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00713 · Cited by 212
“Multimodal generative tasks involve simultaneous representation learning and generation/synthesis [91], in which representation networks [99, 45, 25, 96, 64, 95] with contrastive loss [66, 15, 1, 93, 94] played an essential role.”
BackgroundLifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 136
Metadata
- Authors
- Suzuki, M., & Matsuo, Y.
- Affiliations
- The University of Tokyo (JP)
- Corresponding author
- Suzuki, Masahiro
- Venue
- Advanced Robotics
- Publisher
- Informa UK Limited
- Published
- 2022-02-21
- Volume
- 36
- Issue
- 5-6
- Pages
- 261-278
- ISSN
- 0169-1864 (ISSN-L)
- 1568-5535
- OpenAlex
- W4213299273
- Cited by
- 109
- References
- 93
- Citations by year
- 2026: 18 · 2025: 38 · 2024: 27 · 2023: 22 · 2022: 4
- FWCI
- 7.45
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Advanced Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Energy and Industrial Technology Development Organization (JPNP16007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal learning, deep generative models, variational autoencoder, cross-modal generation, shared representations, multimodal generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
