The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Deep Multimodal Learning: A Survey on Recent Advances and Trends
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
“[10] provide an overview on deep multi-modal learning, and mention its applications in diverse research fields, such as robotic grasping and human action recognition.”
BackgroundSoybean yield prediction from UAV using multimodal data fusion and deep learning
Maimaitijiang, M., Sagan, V., Sidike, P., Hartling, S., Esposito, F., & Fritschi, F. B. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.111599 · Cited by 971
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
BackgroundInfluential
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Multimodal Learning: A Survey on Recent Advances and Trends. https://app.cochat.ai/papers/10.1109/msp.2017.2738401
- The paper
- Ramachandram, D., & Taylor, G. W. (2017). Deep Multimodal Learning: A Survey on Recent Advances and Trends. IEEE Signal Processing Magazine. https://doi.org/10.1109/msp.2017.2738401
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.