A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
MethodRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“There are several well-known deep architectures: convolutional neural networks (CNN), recurrent neural networks (RNN), and generative adversarial networks (GAN) (Bengio, Courville, & Vincent, 2013; Chen & Lin, 2014).”
BackgroundSqueeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · cited 2×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
MethodBidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 2×
“For example, to model the bidirectional dependency of the sequential data, Schuster and Paliwal (1997) proposed the bidirectional RNN, where there are two independent computing processes that encode the forward dependency and the backward dependency.”
MethodShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239 · cited 1×
“The restricted Boltzmann machine (RBM) is the basic block of the deep belief net (Zhang, Ding, Zhang, & Xue, 2018; Bengio, 2009).”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1×
“It has been applied in various domains, such as speech analysis (Mulder, Bethard, & Moens, 2015), image caption (Xu et al., 2015), and language translation (Graves & Jaitly, 2014), achieving outperforming performance.”
MethodExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 2×
“DAE (Vincent et al., 2008) A denoising variant that reconstructs the clear data from the noising data.”
BackgroundRecent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · cited 2×
“Inception (Christian et al., 2017) A deeper and wider network is designed by using the uniform grid size for the blocks with auxiliary information.”
BackgroundMethodEfficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1×
“A convolutional neural network is a special kind of deep network that considers the local topology of data (Li, Xia, Du, Lin, & Samat, 2017; Sze, Chen, Yang, & Emer, 2017).”
BackgroundA Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852 · cited 4×
“Alexnet (Krizhevsky, Sutskever, & Hinton, 2012) The nonsaturating neurons and the dropout are adopted in the nonlinear computational layers, based on a GPU implementation, respectively.”
BackgroundMethodInfluential(2011) · Cited by 2,752
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Journal of Machine Learning Research · Cited by 2,631
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504 · cited 1×
“These deep learning methods have made great progress in both generative and discriminative tasks based on supervised and unsupervised training strategies (Guo et al., 2016).”
BackgroundConditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 2×
“To fit the sequential training object well, RNN uses the backpropagation-through-time algorithm to train its parameters (Hermans & Schrauwen, 2013; Zheng et al., 2015).”
MethodDeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond
Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00246 · Cited by 2,307 · cited 2×
“…channel interdependencies, some Squeeze-and-Excitation networks are introduced by using the global informational embedding and adaption recalibration operations, which are regarded as self-attention networks on local-and-global information (Jie, Li, & Sun, 2018; Cao, Xu, Lin, Wei, & Hu, 2019).”
BackgroundMethodMobileNetV2: Inverted Residuals and Linear BottlenecksPreprint
Sandler, M., Howard, A., Zhu, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04381 · Cited by 2,255
Multisensor data fusion: A review of the state-of-the-art
Khaleghi, B., Khamis, A., Karray, F. O., & Razavi, S. N. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.001 · Cited by 2,074 · cited 1×
“The reasonable fusion of these multimodal data can help us better understand the event of interest, especially when one modality is incomplete (Khaleghi, Khamis, Karray, & Razavi, 2013; Lahat, Adali, & Jutten, 2015).”
BackgroundMultimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · Journal of Machine Learning Research · doi:10.5555/2627435.2697059 · Cited by 1,807 · cited 1×
“Srivastava and Salakhutdinov (2012) proposed a multimodal generative model based on the deep Boltzmann learning model, learning multimodal representations by fitting the joint distributions of multimodal data over the various modalities, such as image, text, and audio.”
BackgroundGenerating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · International Conference on Machine Learning · Cited by 1,783
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Alzheimer’s Disease Neuroimaging Initiative
Mueller, S. G., Weiner, M. W., Thal, L. J., Petersen, R. C., Jack, C., Jagust, W., Trojanowski, J. Q., Toga, A. W., & Beckett, L. (2007) · Advances in Behavioral Biology · doi:10.1007/978-0-387-72076-0_18 · Cited by 1,468
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 1×
“The reasonable fusion of these multimodal data can help us better understand the event of interest, especially when one modality is incomplete (Khaleghi, Khamis, Karray, & Razavi, 2013; Lahat, Adali, & Jutten, 2015).”
BackgroundBig data deep learning: Challenges and perspectives
Chen, X.-W., & Lin, X. (2014) · IEEE Access · doi:10.1109/access.2014.2325029 · Cited by 1,255 · cited 1×
“There are several well-known deep architectures: convolutional neural networks (CNN), recurrent neural networks (RNN), and generative adversarial networks (GAN) (Bengio, Courville, & Vincent, 2013; Chen & Lin, 2014).”
BackgroundA survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226 · cited 1×
“Deep belief net RBM (Zhang et al., 2018) A generative graphic model that uses the energy to capture the probability distribution between visible units and hidden units.”
BackgroundSelf-citationConvolutional neural networks for large-scale remote-sensing image classification
Maggiori, E., Tarabalka, Y., Charpiat, G., & Alliez, P. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2612821 · Cited by 1,114 · cited 1×
“It has made much progress in medical image recognition (Maggiori, Tarabalka, Charpiat, & Alliez, 2017) and semantic analysis (Hu, Lu, Li, & Chen, 2014).”
BackgroundConvolutional Neural Network Architectures for Matching Natural Language Sentences
Hu, B., Lu, Z., Li, H., & Chen, Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03244 · Cited by 1,098 · cited 1×
“It has made much progress in medical image recognition (Maggiori, Tarabalka, Charpiat, & Alliez, 2017) and semantic analysis (Hu, Lu, Li, & Chen, 2014).”
BackgroundDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Auto-encoder based dimensionality reduction
Wang, Y., Yao, H., & Zhao, S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.08.104 · Cited by 976 · cited 1×
“The SAE has been widely used in many fields, including dimension reduction (Wang, Yao, & Zhao, 2016), image recognition (Jia, Shao, Li, Zhao, & Fu, 2018), and text classification (Chen & Zaki, 2017).”
MethodHierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968
Canonical analysis of several sets of variables
KETTENRING, J. R. (1971) · Biometrika · doi:10.1093/biomet/58.3.433 · Cited by 896 · cited 1×
“For example, Kettenring (1971) proposed the multimodal canonical correlation analysis for the linear intermodal-ity relationship as well as the cross-modality generalization information.”
BackgroundMVAE: Multimodal Variational Autoencoder for Fake News Detection
Khattar, D., Goud, J. S., Gupta, M., & Varma, V. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313552 · Cited by 799 · cited 1×
“Khattar, Goud, Gupta, and Varma (2019) designed a multimodal variational framework based on the encoder-decoder architecture.”
Backgroundk-Sparse AutoencodersPreprint
Makhzani, A., & Frey, B. J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5663 · Cited by 730 · cited 17×
“However, they neglect the spatial and temporal topologies of the multimodal data. by Ngiam et al. (2011) is the most representative deep learning model based on the stacked autoencoder (SAE) for multimodal data fusion.”
BackgroundInfluentialDeep Pyramidal Residual Networks
Han, D., Kim, J., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.668 · Cited by 715
Offline Arabic Handwriting Recognition with Multidimensional Recurrent Neural Networks
Graves, A. (2012) · Guide to OCR for Arabic Scripts · doi:10.1007/978-1-4471-4072-6_12 · Cited by 684 · cited 1×
“Unlike deep forward architectures (i.e., DBN, SAE, and CNN), it not only maps the input patterns to the output results but also transfers the hidden states to the outputs by employing the connections between the hidden units (Graves & Schmidhuber, 2008).”
BackgroundA Comprehensive Review of Stability Analysis of Continuous-Time Recurrent Neural Networks
Zhang, H., Wang, Z., & Liu, D. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2317880 · Cited by 654 · cited 1×
“Recurrentneuralnetwork RNN (Zhang et al., 2014) A fully connected network where the self-connection between hidden layers is used to model the time dependency.”
BackgroundDeep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 620 · cited 2×
“…that models the joint semantic distribution over the learned language and vision representation, with the following form: (3.3) Furthermore, to train the multimodal recurrent neural model, Mao et al. (2014) use an average log-likelihood loss function based on a standard language evaluation method.”
MethodInfluentialMultimodal Deep Autoencoder for Human Pose Recovery
Hong, C., Yu, J., Wan, J., Tao, D., & Wang, M. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2487860 · Cited by 568
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · International Conference on Machine Learning · Cited by 525
A Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 524 · cited 1×
“And each modality of different source, type, and distribution contains modality-specific information (Li, Yang, & Zhang, 2019; Gao, Li, & Li, 2016).”
BackgroundData fusion through cross-modality metric learning using similarity-sensitive hashing
Bronstein, M. M., Bronstein, A. M., Michel, F., & Paragios, N. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539928 · Cited by 455 · cited 1×
“…mining, aims to integrate the data of different distributions, sources, and types into a global space in which both intermodality and cross-modality can be represented in a uniform manner (Bramon et al., 2012; Bronstein, Bronstein, Michel, & Paragios, 2010; Poria, Cambria, Bajpai, & Hussain, 2017).”
BackgroundHow Can Heterogeneous Internet of Things Build Our Future: A Survey
Qiu, T., Chen, N., Li, K., Atiquzzaman, M., & Zhao, W. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2803740 · Cited by 444
A review of multivariate methods for multimodal fusion of brain imaging data
Sui, J., Adali, T., Yu, Q., Chen, J., & Calhoun, V. D. (2012) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.10.031 · Cited by 401 · cited 1×
“In the past, some multimodal data fusion methods were presented to explore the complementary and cross-modality information between modalities (Sui, Adali, Yu, Chen, & Calhoun, 2012).”
BackgroundNext-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics
Lv, Z., Song, H., Basanta-Val, P., Steed, A., & Jo, M. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2650204 · Cited by 365 · cited 1×
“These data, often referred to as big data, hold such characteristics as high volume, high variety, high velocity, and high veracity (Gao, Li, & Chen, 2019; Lv, Song, Val, Steed, & Jo, 2017).”
BackgroundMultimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
Recent Advances in Autoencoder-Based Representation Learning
Tschannen, M., Bachem, O., & Lučić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05069 · Cited by 329 · cited 3×
“More autoencoder variants can be found in Michael et al. (2018).”
BackgroundConcurrent EEG/fMRI analysis by multiway Partial Least Squares
Martı́nez-Montes, E., Valdés-Sosa, P. A., Miwakeichi, F., Goldman, R. I., & Cohen, M. S. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.03.038 · Cited by 326
Linked independent component analysis for multimodal data fusion
Groves, A. R., Beckmann, C. F., Smith, S. M., & Woolrich, M. W. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.073 · Cited by 319 · cited 1×
“Groves, Beckmann, Smith, and Woolrich (2011) presented a multimodal independent component analysis that is a probabilistic model using the Bayesian framework to combine the independent variables of each different modality.”
BackgroundLi, E., Xia, J., Du, P., Lin, C., & Samat, A. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2711275 · Cited by 296
Simple Recurrent Units for Highly Parallelizable Recurrence
Lei, T., Zhang, Y., Wang, S. I., Dai, H., & Artzi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1477 · Cited by 289 · cited 2×
“SRNN (Lei et al., 2018) A fast variant in which the light recurrence and highway network are proposed to improve the learning efficiency for a parallelized implementation.”
BackgroundA survey on the application of recurrent neural networks to statistical language modeling
De Mulder, W., Bethard, S., & Moens, M.-F. (2015) · Computer Speech & Language · doi:10.1016/j.csl.2014.09.005 · Cited by 272 · cited 1×
“It has been applied in various domains, such as speech analysis (Mulder, Bethard, & Moens, 2015), image caption (Xu et al., 2015), and language translation (Graves & Jaitly, 2014), achieving outperforming performance.”
MethodMulti-source Deep Learning for Human Pose Estimation
Ouyang, W., Chu, X., & Wang, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.299 · Cited by 262 · cited 1×
“To accurately estimate human poses, Ouyang, Chu, and Wang (2014) designed a multisource deep learning model that learns multimodal representation from mixture type, appearance score, and deformation modalities by extracting the joint distribution of the body pattern in high-order space.”
BackgroundAn overview on Restricted Boltzmann Machines
Zhang, N., Ding, S., Zhang, J., & Xue, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.09.065 · Cited by 245 · cited 1×
“The restricted Boltzmann machine (RBM) is the basic block of the deep belief net (Zhang, Ding, Zhang, & Xue, 2018; Bengio, 2009).”
BackgroundSelf-citationAudio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks
Hou, J.-C., Wang, S.-S., Lai, Y.-H., Tsao, Y., Chang, H.-W., & Wang, H.-M. (2018) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2017.2784878 · Cited by 239 · cited 1×
“For instance, Hou, Wang, Lai, Chang, and Wang (2018) proposed a multimodal speech enhancement framework.”
BackgroundGCNet: Non-local Networks Meet Squeeze-Excitation Networks and BeyondPreprint
Cao, Y., Xu, J., Lin, S. C.-F., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11492 · Cited by 212 · cited 2×
“…channel interdependencies, some Squeeze-and-Excitation networks are introduced by using the global informational embedding and adaption recalibration operations, which are regarded as self-attention networks on local-and-global information (Jie, Li, & Sun, 2018; Cao, Xu, Lin, Wei, & Hu, 2019).”
BackgroundMethodDeep Convolutional Computation Model for Feature Learning on Big Data in Internet of Things
Li, P., Chen, Z., Yang, L. T., Zhang, Q., & Deen, M. J. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2739340 · Cited by 199 · cited 1×
“…s t on 31 A ugu s t 2021 data of high volume, high velocity, high variety, and high veracity since they are based on the shallow feature that cannot capture intrinsic internal structures and external relationships in multimodal data (Li, Chen, Yang, Zhang, & Deen, 2018; Zhang, Yang, & Chen, 2016).”
BackgroundSelf-citationAnalysis of Multimodal Neuroimaging Data
Bießmann, F., Plis, S., Meinecke, F. C., Eichele, T., & Müller, K. (2011) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2011.2170675 · Cited by 177 · cited 1×
“It can provide richer information than a single modality by leveraging modality-specific information (Biessmann, Plis, Meinecke, Eichele, & Muller, 2011; Wagner, Andre, Lingenfelser, & Kim, 2011).”
BackgroundDeep Computation Model for Unsupervised Feature Learning on Big Data
Zhang, Q., Yang, L. T., & Chen, Z. (2015) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2015.2497705 · Cited by 158
Self-citationExploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156 · cited 1×
“It can provide richer information than a single modality by leveraging modality-specific information (Biessmann, Plis, Meinecke, Eichele, & Muller, 2011; Wagner, Andre, Lingenfelser, & Kim, 2011).”
BackgroundEnhancing Medical Smartphone Networks via Blockchain-Based Trust Management Against Insider Attacks
Meng, W., Li, W., & Zhu, L. (2019) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2019.2921736 · Cited by 154
DEVDAN: Deep evolving denoising autoencoder
Ashfahani, A., Pratama, M., Lughofer, E. D., & Ong, Y.-S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.106 · Cited by 134
Majumdar, A. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2838679 · Cited by 129
Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks
Sano, A., Chen, W. '., Lopez-Martinez, D., Taylor, S., & Picard, R. W. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2867619 · Cited by 84
A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips
Nguyen, T.-L., Kavuri, S., & Lee, M. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.06.010 · Cited by 84 · cited 1×
“Nguyen, Kavuri, and Lee (2019) introduced a multimodal CNN network to classify the emotion of movie clips.”
BackgroundTraining and analyzing deep recurrent neural networks
Hermans, M., & Schrauwen, B. (2013) · Cited by 84
Multimodal Convolutional Neural Networks for Matching Image and SentencePreprint
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06063 · Cited by 77 · cited 1×
“To model the semantic mapping distribution between images and sentences, Ma, Lu, Shang, and Li (2015) proposed a multimodal convolutional neural network.”
MethodRecurrent neural network-based semantic variational autoencoder for Sequence-to-sequence learning
Jang, M., Seo, S., & Kang, P. (2019) · Information Sciences · doi:10.1016/j.ins.2019.03.066 · Cited by 75
Multimodal Data Fusion Based on Mutual Information
Bramon, R., Boada, I., Bardera, A., Rodriguez, J., Feixas, M., Puig, J., & Sbert, M. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.280 · Cited by 69 · cited 2×
“…mining, aims to integrate the data of different distributions, sources, and types into a global space in which both intermodality and cross-modality can be represented in a uniform manner (Bramon et al., 2012; Bronstein, Bronstein, Michel, & Paragios, 2010; Poria, Cambria, Bajpai, & Hussain, 2017).”
BackgroundMethodUnseen Noise Estimation Using Separable Deep Auto Encoder for Speech Enhancement
Sun, M., Zhang, X., hamme, H. V., & Zheng, T. (2016) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2498101 · Cited by 67 · cited 1×
“As shown in Figure 2, the stacked autoencoder, the most typical fully connected neural network, consists of an input layer, an output layer, and several hidden layers (Sun, Zhang, Hamme, & Zheng, 2016).”
BackgroundA multimodal deep learning framework using local feature representations for face recognition
Al‐Waisy, A. S., Qahwaji, R., Ipson, S., & Al-Fahdawi, S. (2017) · Machine Vision and Applications · doi:10.1007/s00138-017-0870-2 · Cited by 64 · cited 1×
“In this method, a DBN-based model is used to model the multimodal distribution over the local handcrafted features captured by the Curvelet transform, which can merge the advantages of the local and deep features (Al-Waisy et al., 2018).”
MethodImproving variational autoencoder with deep feature consistent and generative adversarial training
Hou, X., Sun, K., Shen, L., & Qiu, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.03.013 · Cited by 62 · cited 1×
“GAE (Hou et al., 2019) An adversarial variant that the decoder subnetwork that is also regarded as the generator, adopting game theory to more consistent features with input data.”
BackgroundLearning Cascaded Deep Auto-Encoder Networks for Face Alignment
Weng, R., Lu, J., Tan, Y., & Zhou, J. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2591508 · Cited by 52 · cited 1×
“A stacked autoencoder (SAE) is a typical deep learning model of the encoder-decoder architecture (Michael, Olivier, & Mario, 2018; Weng, Lu, Tan, & Zhou, 2016).”
Deep Multimodal Fusion: A Hybrid Approach
Amer, M. R., Shields, T. J., Siddiquie, B., Tamrakar, A., Divakaran, A., & Chai, S. M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-0997-7 · Cited by 42
Wang, C.-Y., Wang, J., Santoso, A., Chiang, C.-C., & Wu, C. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2738443 · Cited by 38
Multimodal Recurrent Neural Networks With Information Transfer Layers for Indoor Scene Labeling
Abdulnabi, A. H., Shuai, B., Zuo, Z., Chau, L.-P., & Wang, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2774007 · Cited by 33 · cited 1×
“For example, Abdulnabi, Shuai, Zuo, Chau, and Wang (2018) designed a multimodal RNN to label indoor scenes in which the intermodality feature and cross-modality feature are learned by the RNN and transform layers.”
BackgroundGao, J., Li, P., & Chen, Z. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.04.048 · Cited by 31 · cited 1×
“These data, often referred to as big data, hold such characteristics as high volume, high variety, high velocity, and high veracity (Gao, Li, & Chen, 2019; Lv, Song, Val, Steed, & Jo, 2017).”
BackgroundSelf-citationApproximate event detection over multi-modal sensing data
Gao, J., Li, J., & Li, Y. (2015) · Journal of Combinatorial Optimization · doi:10.1007/s10878-015-9847-0 · Cited by 29
Self-citationAssociative Multichannel Autoencoder for Multimodal Word Representation
Wang, S., Zhang, J., & Zong, C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1011 · Cited by 27 · cited 1×
“For example, Wang, Zhang, and Zong (2018) designed a multimodal stacked autoencoder for feature learning of words, which the association and gating mechanisms are adopted to improve the word features.”
BackgroundSelf-citationStacked Denoising Tensor Auto-Encoder for Action Recognition With Spatiotemporal Corruptions
Jia, C., Shao, M., Li, S., Zhao, H., & Fu, Y. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2781299 · Cited by 23
LCD: A Fast Contrastive Divergence Based Algorithm for Restricted Boltzmann Machine
Ning, L., Pittman, R., & Shen, X. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.08.018 · Cited by 23
KATE: K-Competitive Autoencoder for TextPreprint
Chen, Y., & Zaki, M. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02033 · Cited by 20 · cited 1×
“The SAE has been widely used in many fields, including dimension reduction (Wang, Yao, & Zhao, 2016), image recognition (Jia, Shao, Li, Zhao, & Fu, 2018), and text classification (Chen & Zaki, 2017).”
MethodSparse Boltzmann Machines with Structure Learning as Applied to Text Analysis
Chen, Z., Zhang, N. L., Yeung, D., & Chen, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10773 · Cited by 15 · cited 5×
“Motivated by this, the concatenated vector of the midlevel representations is fed into SRBM to model the correlation of multiple modalities, which shows better performance.”
BackgroundMethodInfluentialSelf-citationTensorizing Restricted Boltzmann Machine
Ju, F., Sun, Y., Gao, J., Antolovich, M., Dong, J., & Yin, B. (2019) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3321517 · Cited by 5 · cited 2×
“TTRBM (Ju et al., 2019) A compact variant that the parameters between the visible layer and hidden layer are reduced by transforming into the tensor-train format.”
BackgroundSelf-citationTemporal Multimodal Fusion for Driver Behavior Prediction Tasks using Gated Recurrent Fusion Units.
Narayanan, A., Siravuru, A., & Dariush, B. (2019) · Cited by 4
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932
“CNNs, RNNs, deep belief networks and autoencoders (AE) [21]; score-level fusion [22]; or multimodal data fusion [23].”
BackgroundZhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480
“Compared to the existing surveys [2, 4, 45, 46, 121, 128, 227, 235, 243] that usually focus on one single task (such as multimodal object recognition) with one specific combination of two modalities (such as RGB+depth data), this survey owns a broader scope covering various modalities and their…”
BackgroundA Systematic Literature Review on Crop Yield Prediction with Deep Learning and Remote Sensing
Muruganantham, P., Wibowo, S., Grandhi, S., Samrat, N. H., & Islam, N. (2022) · Remote Sensing · doi:10.3390/rs14091990 · Cited by 339
“The ANN approach was not efficient to capture the variability in micro-topography, which affected the prediction and had large model errors for fields with high yield variability [75].”
BackgroundVideo Summarization Using Deep Neural Networks: A Survey
Apostolidis, E., Adamantidou, E., Metsai, A. I., Mezaris, V., & Patras, I. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3117472 · Cited by 350
Deep Gait Recognition: A Survey
Sepas-Moghaddam, A., & Etemad, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3151865 · Cited by 307
“ods [253], [254], [255], [256], [257], [258], while multi-modal deep learning methods [259], [260], notably based on fusion [261], joint learning [233], and attention [262] net-”
Background
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.