Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Deep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
Combining Pattern Classifiers: Methods and Algorithms
Kuncheva, L. I. (2004) · Wiley eBooks · doi:10.1002/0471660264 · Cited by 3,005
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“To capture temporal structures and relationships, deep multimodal learning approaches typically use temporal components such as LSTMs or hidden Markov models combined with visual representation learning layers like CNNs or 3-D-CNNs [42], [43].”
Method(2011) · Cited by 2,752
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Multisensor data fusion: A review of the state-of-the-art
Khaleghi, B., Khamis, A., Karray, F. O., & Razavi, S. N. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.001 · Cited by 2,074
Learning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 1×
“The basic idea proposed by Elman [73] is to start with a relatively small network and add hidden units or layers incrementally until the best performing architecture is found.”
BackgroundDeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
Reed, R. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.248452 · Cited by 1,708
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · Journal of Machine Learning Research · doi:10.5555/2627435.2697059 · Cited by 1,659
Deep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989385 · Cited by 1,424
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 1×
“For the purposes of our review, we adopt the definition provided by Lahat et al. [2], where we consider phenomena or systems that are observed using multiple sensors and each sensor output can be termed a modality associated with a single data set.”
MethodMultimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
Auto-Encoding Variational Bayes
Chen, Y., Liu, J., Peng, L., Wu, Y., Xu, Y., & Zhang, Z. (2024) · Cambridge Explorations in Arts and Sciences · doi:10.61603/ceas.v2i1.33 · Cited by 934
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
Chen, C., Jafari, R., & Kehtarnavaz, N. (2015) · 2015 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2015.7350781 · Cited by 765
Genetic algorithms and neural networks: optimizing connections and connectivity
Whitley, D., Starkweather, T., & Bogart, C. (1990) · Parallel Computing · doi:10.1016/0167-8191(90)90086-o · Cited by 747
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
A tutorial survey of architectures, algorithms, and applications for deep learning
Deng, L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2013.9 · Cited by 725
Applications of Deep Learning in Biomedicine
Mamoshina, P., Vieira, A., Lane, E., & Zhavoronkov, A. (2016) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.5b00982 · Cited by 701
Classification using discriminative restricted Boltzmann machines
Larochelle, H., & Bengio, Y. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390224 · Cited by 655
Net2Net: Accelerating Learning via Knowledge TransferPreprint
Chen, T., Goodfellow, I., & Shlens, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05641 · Cited by 649
Multimodal Neuroimaging Feature Learning for Multiclass Diagnosis of Alzheimer's Disease
Liu, S., Liu, S., Cai, W., Che, H., Pujol, S., Kikinis, R., Feng, D., Fulham, M. J., & ADNI (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2372011 · Cited by 637
Gated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 610
Poria, S., Cambria, E., & Gelbukh, A. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1303 · Cited by 533
Berkeley MHAD: A comprehensive Multimodal Human Action Database
Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., & Bajcsy, R. (2013) · 2013 IEEE Workshop on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2013.6474999 · Cited by 484
Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition
Wu, D., Pigou, L., Kindermans, P.-J., Le, N. D.-H., Shao, L., Dambre, J., & Odobez, J.-M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2537340 · Cited by 453
Robust Face Recognition via Multimodal Deep Face Representation
Ding, C., & Tao, D. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477042 · Cited by 449 · cited 2×
“…deep-learning methods have been applied to various problems involving human activity such as action recognition (an activity can be composed of two or more sequences of shorter actions), gesture recognition [8], gaze-direction estimation [9], face recognition [10], and emotion recognition [11].”
BackgroundMethodModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420
Self-citationEmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412
HMOG: New Behavioral Biometric Features for Continuous Authentication of Smartphone Users
Sitova, Z., Sedenka, J., Yang, Q., Peng, G., Zhou, G., Gasti, P., & Balagani, K. S. (2016) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2015.2506542 · Cited by 398
Combining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391
Jiang, Y.-G., Wu, Z., Wang, J., Xue, X., & Chang, S.-F. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2670560 · Cited by 381
Interpretable deep neural networks for single-trial EEG classification
Sturm, I., Lapuschkin, S., Samek, W., & Müller, K.-R. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.008 · Cited by 374
Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_48 · Cited by 373
Exploring Models and Data for Image Question Answering
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 359
Banos, O., Villalonga, C., Garcia, R., Saez, A., Damas, M., Holgado-Terriza, J. A., Lee, S., Pomares, H., & Rojas, I. (2015) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-14-s2-s6 · Cited by 321
KinectFaceDB: A Kinect Database for Face Recognition
Min, R., Kose, N., & Dugelay, J.-L. (2014) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2014.2331215 · Cited by 274
Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487478 · Cited by 271 · cited 1×
“To solve this problem, Jain et al. [35] incorporated a temporal term that grows exponentially in time into their cost function for a multimodal RNN with LSTM units.”
BackgroundIntegrative data analysis of multi-platform cancer data with a multimodal deep learning approach
Liang, M., Li, Z., Chen, T., & Zeng, J. (2015) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2014.2377729 · Cited by 231
ChaLearn Looking at People Challenge 2014: Dataset and Results
Escalera, S., Baró, X., Gonzàlez, J., Bautista, M. A., Madadi, M., Reyes, M., Ponce-López, V., Escalante, H. J., Shotton, J., & Guyon, I. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16178-5_32 · Cited by 211
Multimodal Residual Learning for Visual QA
Kim, J.-H., Lee, S., Kwak, D., Heo, M.-O., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01455 · Cited by 209
Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion
Valada, A., Oliveira, G. L., Brox, T., & Burgard, W. (2017) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-319-50115-4_41 · Cited by 199
Shaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
A Deep Metric for Multimodal Registration
Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., Navab, N., & Komodakis, N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46726-9_2 · Cited by 185
Deep Head Pose: Gaze-Direction Estimation in Multimodal Video
Mukherjee, S. S., & Robertson, N. M. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482819 · Cited by 182
RGBD Datasets: Past, Present and Future
Firman, M. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.88 · Cited by 182
Learning Representations for Multimodal Data with Deep Belief Nets
Srivastava, N., & Salakhutdinov, R. (2012) · Cited by 180
Multimodal Similarity-Preserving Hashing
Masci, J., Bronstein, M. M., Bronstein, A. M., & Schmidhuber, J. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.225 · Cited by 179
Online multimodal deep similarity learning with application to image retrieval
Wu, P., Hoi, S. C. H., Xia, H., Zhao, P., Wang, D., & Miao, C. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502112 · Cited by 172
Wu, Z., Jiang, Y., Wang, J., Pu, J., & Xue, X. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654931 · Cited by 156
Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition
Wang, A., Lu, J., Cai, J., Cham, T.-J., & Wang, G. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2476655 · Cited by 153
Efficient evolution of neural network topologies
Stanley, K. O., & Miikkulainen, R. P. (2003) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2002.1004508 · Cited by 151
Towards multimodal deep learning for activity recognition on mobile devices
Radu, V., Lane, N. D., Bhattacharya, S., Mascolo, C., Marina, M. K., & Kawsar, F. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct · doi:10.1145/2968219.2971461 · Cited by 147
Improved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 145
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Jain, A., Tompson, J., LeCun, Y., & Bregler, C. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16808-1_21 · Cited by 142
Shared representation learning for heterogenous face recognition
Yi, D., Lei, Z., & Li, S. Z. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163093 · Cited by 135
Joint Multimodal Learning with Deep Generative Models
Suzuki, M., Nakayama, K., & Matsuo, Y. (2016) · arXiv · doi:10.48550/arxiv.1611.01891 · Cited by 126
Structured Regularizers for High-Dimensional Problems: Statistical and Computational Issues
Wainwright, M. J. (2014) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-022513-115643 · Cited by 121
MMSS: Multi-modal Sharable and Specific Feature Learning for RGB-D Object Recognition
Wang, A., Cai, J., Lu, J., & Cham, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.134 · Cited by 108
Machine Learning in Computer Vision
Sebe, N., Cohen, I., Garg, A., & Huang, T. S. (2005) · Springer eBooks · doi:10.1007/1-4020-3275-7 · Cited by 101
Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep Structure
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2455415 · Cited by 97
Multimodal video classification with stacked contractive autoencoders
Liu, Y., Xiao-qing, F., & Zhou, Z. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.01.001 · Cited by 96
Medical Image Retrieval: A Multimodal Approach
Cao, Y., Steffey, S., He, J., Xiao, D., Tao, C., Chen, P., & Müller, H. (2014) · Cancer Informatics · doi:10.4137/cin.s14053 · Cited by 90
Learning the Structure of Deep Convolutional Networks
Feng, J., & Darrell, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.315 · Cited by 71
Blockout: Dynamic Model Selection for Hierarchical Deep Networks
Murdock, C., Li, Z., Zhou, H., & Duerig, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.283 · Cited by 67
Structure discovery of deep neural network based on evolutionary algorithms
Shinozaki, T., & Watanabe, S. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178918 · Cited by 62
Deep Multimodal Fusion: Combining Discrete Events and Continuous Signals
Martínez, H. P., & Yannakakis, G. N. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663236 · Cited by 62
(2017) · doi:10.1109/ijcnn.2017.7965870 · Cited by 57
Segmenting Hippocampus from Infant Brains by Sparse Patch Matching with Deep-Learned Features
Guo, Y., Guorong Wu, A., Commander, L. A., Szary, S., Jewells, V. L., Lin, W., & Shen, D. G. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10470-6_39 · Cited by 53
Unconstrained Multimodal Multi-Label Learning
Huang, Y., Wang, W., & Wang, L. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2476658 · Cited by 50
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated ImagesPreprint
Mao, J., Xu, J., Jing, Y., & Yuille, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08321 · Cited by 46
Kim, S. G., Theera-Ampornpunt, N., Fang, C.-H., Harwani, M., Grama, A. Y., & Chaterji, S. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0302-3 · Cited by 44
Deep Multimodal Fusion: A Hybrid Approach
Amer, M. R., Shields, T. J., Siddiquie, B., Tamrakar, A., Divakaran, A., & Chai, S. M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-0997-7 · Cited by 42
Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy UpdatesPreprint
Gu, S., Holly, E., Lillicrap, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00633 · Cited by 39
Structured feature selection and task relationship inference for multi-task learning
Fei, H., & Huan, J. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-012-0543-4 · Cited by 37
Multimodal fusion using dynamic hybrid models
Amer, M. R., Siddiquie, B., Khan, S. M., Divakaran, A., & Sawhney, H. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6836053 · Cited by 28
Maximum mutual information regularized classification
Wang, J. J.-Y., Wang, Y., Zhao, S., & Gao, X. (2014) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2014.08.009 · Cited by 26
Stacked Multiscale Feature Learning for Domain Independent Medical Image Segmentation
Kiros, R., Popuri, K., Cobzaş, D., & Jägersand, M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10581-9_4 · Cited by 23
Learning the Structure of Deep Architectures Using L1 Regularization
Kulkarni, P., Zepeda, J., Jurie, F., Pérez, P., & Chevallier, L. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.23 · Cited by 16
Modout: Learning Multi-Modal Architectures by Stochastic Regularization
Li, F., Neverova, N., Wolf, C., & Taylor, G. W. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.59 · Cited by 13
Self-citationA structure optimization algorithm of neural networks for large-scale data sets
Yang, J., Ma, J., Berryman, M. J., & Perez, P. (2014) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2014.6891662 · Cited by 10
Multimodal deep network learning‐based image annotation
Zhu, S., Li, X., & Shen, S. (2015) · Electronics Letters · doi:10.1049/el.2015.0258 · Cited by 8
Sports video classification from multimodal information using deep neural networks
Sachan, D. S., Tekwani, U., & Sethi, A. (2013) · National Conference on Artificial Intelligence · Cited by 8
A Multimodal Human-Robot Interaction Dataset
Azagra, P., Mollard, Y., Golemo, F., Murillo, A. C., Lopes, M. C., & Civera, J. (2016) · HAL · Cited by 4
Structure Optimization for Deep Multimodal Fusion Networks using Graph-Induced KernelsPreprint
Ramachandram, D., Lisicki, M., Shields, T. J., Amer, M. R., & Taylor, G. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00750 · Cited by 4
Self-citation
Cited by
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
“[10] provide an overview on deep multi-modal learning, and mention its applications in diverse research fields, such as robotic grasping and human action recognition.”
BackgroundSoybean yield prediction from UAV using multimodal data fusion and deep learning
Maimaitijiang, M., Sagan, V., Sidike, P., Hartling, S., Esposito, F., & Fritschi, F. B. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.111599 · Cited by 971
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
BackgroundInfluential
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.