The DOI matches its Crossref record; 88 of the 93 references it cites were found in a registry or an index.
A survey of multimodal deep generative models
Open accessAbstract
Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e. generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on the position of the layer for the multimodal interaction, the data fusion architectures can be classified into early, intermediate, and later fusion [50].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [50]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Goyal et al. [20] proposed an attention-based GW architecture for communicating positions and modules in Transformer [103] and slot-based modular architectures [21].”
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 3×
“The most commonly used are image datasets such as MNIST [46], FashionM-NIST [115], and the street view house number (SVHN) [61], which are often used as toy problems for deep learning.”
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2×
“In addition to these, cycleconsistency loss is used in image-to-image translation, which is the task of transforming an image belonging to one domain to another domain [57,58].”
BackgroundPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Srivastava and Salakhutdinov [37] propose a multimodal generative model based on deep belief networks [38,39], which has since been used in various multimodal applications [40,41].”
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 2×
“In general, a “good representation” is one that can be used for various tasks while retaining the characteristics of the original input [6].”
BackgroundAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 36
- Stein BE, 1993, The merging of the senses
- Kingma DP, 2014, International Conference on Learning Representations
- van den Oord A, Dieleman S, Zen H, et al. Wavenet: a generative model for raw audio; 2016. Preprint arXiv:1609.03499. · arXiv
- Van Oord A, 2016, International Conference on Machine Learning
- Rezende D, 2015, International Conference on Machine Learning
- Kingma DP, 2018, Advances in Neural Information Processing Systems
- Xu C, Tao D, Xu C. A survey on multi-view learning; 2013. Preprint arXiv:1304.5634. · arXiv
- Tsai Y-HH, 2021, International Conference on Learning Representations
- Sohn K, 2015, Adv Neural Inf Process Syst, 28, 3483
- Tschannen M, Bachem O, Lucic M. Recent advances in autoencoder-based representation learning; 2018. Preprint arXiv:1812.05069. · arXiv
- Srivastava N, Salakhutdinov R. Learning representations for multimodal data with deep belief nets. In: International Conference on Machine Learning Workshop, Vol. 79; 2012. p. 3.
- Sohn K, 2014, Adv Neural Inf Process Syst, 27, 2141
- 10.1162/jmlr.2003.3.4-5.993 · DOI
- Higgins I, Matthey L, Pal A, et al. beta-vae: learning basic visual concepts with a constrained variational framework. ICLR; 2016.
- Vincent P, 2010, J Machine Learning Res, 11, 3371
- Liu M-Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks; 2017. Preprint arXiv:1703.00848. · arXiv
- Wu M, Goodman N. Multimodal generative models for compositional representation learning; 2019. Preprint arXiv:1912.05075. · arXiv
- Suzuki M, Nakayama K, Matsuo Y. Improving bi-directional generation between different modalities with variational autoencoders; 2018. Preprint arXiv:1801.08702. · arXiv
- Daunhawer I, 2021, Pattern Recognit, 12544, 459
- Wu M, 2018, Advances in Neural Information Processing Systems
- Suzuki M, Nakayama K, Matsuo Y. Joint multimodal learning with deep generative models; 2016. Preprint arXiv:1611.01891. · arXiv
- Tian Y, Engel J. Latent translation: crossing modalities by bridging generative models; 2019. Preprint arXiv:1902.08261. · arXiv
- Hsu W-N, Glass J. Disentangling by partitioning: a representation learning framework for multimodal sensory data; 2018. Preprint arXiv:1805.11264. · arXiv
- Shi Y, 2021, International Conference on Learning Representations
- 10.1609/aaai.v34i07.6778 · DOI
- Wang D, 2015, International Joint Conference on Artificial Intelligence
- Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders; 2015. Preprint arXiv:1509.00519. · arXiv
- Bouchacourt D, 2018, AAAI Conference on Artificial Intelligence
- Maddison CJ, 2017, International Conference on Learning Representations
- Jang E, 2017, International Conference on Learning Representations
- van den Oord A, Li Y, Vinyals O. Representation learning with contrastive predictive coding; 2018. Preprint arXiv:1807.03748. · arXiv
- Xiao H, Rasul K, Vollgraf R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017. Preprint arXiv:1708.07747 · arXiv
- Ha D, Schmidhuber J. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems; 2018; Montreal, Canada.
- Hafner D, 2019, International Conference on Machine Learning
- Hafner D, 2019, International Conference on Learning Representations
- Suzuki M, Kaneko T, Matsuo Y. Pixyz: a library for developing deep generative models; 2021. Preprint arXiv: 2107.13109. · arXiv
Cited by
Sorted by citations
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 590
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
Hagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 227
“For example, text-based models may lack the ability to incorporate visual or emotional context into their understanding, while image-based models might lack textual or semantic understanding [65].”
BackgroundInfluentialVersatile Diffusion: Text, Images and Variations All in One Diffusion Model
Xu, X., Wang, Z., Zhang, E., Wang, K., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00713 · Cited by 212
“Multimodal generative tasks involve simultaneous representation learning and generation/synthesis [91], in which representation networks [99, 45, 25, 96, 64, 95] with contrastive loss [66, 15, 1, 93, 94] played an essential role.”
BackgroundLifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 136
Metadata
- Authors
- Suzuki, M., & Matsuo, Y.
- Affiliations
- The University of Tokyo (JP)
- Corresponding author
- Suzuki, Masahiro
- Venue
- Advanced Robotics
- Publisher
- Informa UK Limited
- Published
- 2022-02-21
- Volume
- 36
- Issue
- 5-6
- Pages
- 261-278
- ISSN
- 0169-1864 (ISSN-L)
- 1568-5535
- OpenAlex
- W4213299273
- Cited by
- 109
- References
- 93
- Citations by year
- 2026: 18 · 2025: 38 · 2024: 27 · 2023: 22 · 2022: 4
- FWCI
- 7.45
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Advanced Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Energy and Industrial Technology Development Organization (JPNP16007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal learning, deep generative models, variational autoencoder, cross-modal generation, shared representations, multimodal generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
