Deep Multimodal Representation Learning: A Survey
Open accessAbstract
Multimodal representation learning, which aims to narrow the heterogeneity gap among different modalities, plays an indispensable role in the utilization of ubiquitous multimodal data. Due to the powerful representation ability with multiple levels of abstraction, deep learning-based multimodal representation learning has attracted much attention in recent years. In this paper, we provided a comprehensive survey on deep multimodal representation learning which has never been concentrated entirely. To facilitate the discussion on how the heterogeneity gap is narrowed, according to the underlying structures in which different modalities are integrated, we category deep multimodal representation learning methods into three frameworks: joint representation, coordinated representation, and encoder-decoder. Additionally, we review some typical models in this area ranging from conventional models to newly developed technologies. This paper highlights on the key issues of newly developed technologies, such as encoder-decoder model, generative adversarial networks, and attention mechanism in a multimodal representation learning perspective, which, to the best of our knowledge, have never been reviewed previously, even though they have become the major focuses of much contemporary research. For each framework or model, we discuss its basic structure, learning objective, application scenes, key issues, advantages, and disadvantages, such that both novel and experienced researchers can benefit from this survey. Finally, we suggest some important directions for future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BERT-style cross-modal pretraining models produce excellent results on various downstream vision-language tasks but fail to be applied to generative tasks directly.
“while the BERT-style cross-modal pretraining models produce excellent results on various down-stream visionlanguage tasks, they fail to be applied to generative tasks directly.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
Masked language and region modelling as pre-training tasks have the main advantage that the Transformer encoder learned from these supervisions can encode both vision and language patterns based on bidirectional context and is naturally fit for semantic understanding tasks.
“As discussed in [208], the masked language and region modelling as pre-training task have a main advantage that the Transformer encoder learned from these supervisions can encode both vision and language patterns based on bidirectional context and it is naturally fit for the semantic understanding tasks, e.g., VQA, image-text retrieval.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
Both VideoBERT and CBT have to train a separate video-to-text decoder for video captioning.
“As discussed in [208], both VideoBERT [7] and CBT [107] have to train a separate video-to-text decoder for video captioning.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
XGPT is tailor-made for image captioning, to overcome the limitation that BERT-based cross-modal pretrained models fail to be applied to generative tasks directly.
“XGPT [208] is tailor-made for image captioning, to overcome the limitation that BERT-based cross-modal pretrained models fail to be applied to generative tasks directly.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
Down-stream task-agnostic pretraining is optional, not necessary, and down-stream task specific pretraining is widely studied.
“In practices of multimodal Transformers, the aforementioned down-stream task -agnostic pretraining is optional, not necessary, and down-stream task specific pretraining is also widely studied [150], [190], [208], [211].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
BERT-like encoder-only multimodal Transformers need separately to train decoders for generation tasks.
“As discussed in [208], the BERT-like encoder-only multimodal Transformers (e.g., VideoBERT [7], CBT [107]) need separately to train decoders for generation tasks.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
Cross-task gap is another major obstacle to transfer, due to different reasoning and input-output workflows.
“Cross-task gap is another major obstacle to transfer [208], [274], due to the different reasoning and input-output workflows”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [208]
No recent work directly covers the latest advances in deep multimodal learning specifically for the computer vision community.
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · see citing works
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · see citing works
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · see citing works
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Improved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · see citing works
RELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,429 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 4× · see citing works
“By adding a shared RBM hidden layer on top of modality-specific DBN, it learns a joint representation across modalities.”
BackgroundInfluentialResidual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073 · see citing works
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · see citing works
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,297 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · see citing works
(2011) · Cited by 2,752 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Multimodal Unsupervised Image-to-Image Translation
Huang, X., Liu, M.-Y., Belongie, S., & Kautz, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_11 · Cited by 2,597 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works
Deep Generative Image Models using a Laplacian Pyramid of Adversarial NetworksPreprint
Denton, E., Chintala, S., Szlam, A., & Fergus, R. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1506.05751 · Cited by 2,360 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Arjovsky, M., Chintala, S., & Bottou, L. (2017) · arXiv · doi:10.48550/arxiv.1701.07875 · Cited by 2,225 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,073 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
Salakhutdinov, R., & Hinton, G. E. (2009) · Cited by 1,890 · see citing works
OpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,771 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · see citing works
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · Cited by 1,603 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,546 · see citing works
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459 · see citing works
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · see citing works
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R. G., Lévy, R., & Vasconcelos, N. M. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1873987 · Cited by 1,358 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
(1993) · Cited by 1,173 · see citing works
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 3× · see citing works
“From the perspective of multimodal deep learning, the closest effort to ours is [20] which mainly reviews the models and applications relying on multimodal features fusion (categorized as joint representation in ours).”
BackgroundEnd-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472618 · Cited by 1,129 · see citing works
The Dynamic Representation of Scenes
Rensink, R. A. (2000) · Visual Cognition · doi:10.1080/135062800394667 · Cited by 1,079 · see citing works
Multi-view learning overview: Recent progress and new challenges
Zhao, J., Xie, X., Xu, X., & Sun, S. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.007 · Cited by 1,002 · see citing works
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 996 · see citing works
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · see citing works
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855 · see citing works
Image Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836 · see citing works
Learning Deep Representations of Fine-Grained Visual Descriptions
Reed, S., Akata, Z., Lee, H., & Schiele, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.13 · Cited by 802 · see citing works
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · see citing works
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · see citing works
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779 · see citing works
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_28 · Cited by 753 · see citing works
On Deep Multi-View Representation Learning
Wang, W., Livescu, K., & Bilmes, J. (2015) · Cited by 751 · see citing works
Dual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.232 · Cited by 749 · see citing works
Video Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z. Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637 · see citing works
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.497 · Cited by 573 · see citing works
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654902 · Cited by 572 · see citing works
Multimodal Deep Autoencoder for Human Pose Recovery
Hong, C., Yu, J., Wan, J., Tao, D., & Wang, M. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2487860 · Cited by 570 · see citing works
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.12024 · Cited by 535 · see citing works
Fusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534 · see citing works
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G. A., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · doi:10.1109/cvpr.2011.5995466 · Cited by 521 · see citing works
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R. J., Darrell, T., & Saenko, K. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.337 · Cited by 500 · see citing works
A kernel method for canonical correlation analysisPreprint
Akaho, S. (2006) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0609071 · Cited by 459 · see citing works
A Probabilistic Interpretation of Canonical Correlation Analysis
Bach, F., & Jordan, M. I. (2005) · Cited by 410 · see citing works
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00750 · Cited by 409 · see citing works
Deep correlation for matching images and text
Yan, F., & Mikolajczyk, K. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298966 · Cited by 407 · see citing works
Jiang, Y., Wu, Z., Wang, J., Xue, X., & Chang, S. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2670560 · Cited by 401 · see citing works
Self-citationDeep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · see citing works
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Efficient Learning of Deep Boltzmann Machines
Salakhutdinov, R., & Larochelle, H. (2010) · Cited by 352 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · see citing works
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward
Ren, Z., Wang, X., Zhang, N., Lv, X., & Li, L.-J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.128 · Cited by 338 · see citing works
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · see citing works
Self-citationATRank: An Attention-Based User Behavior Modeling Framework for Recommendation
Zhou, C., Bai, J., Song, J., Liu, X., Zhao, Z., Chen, X., & Gao, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11618 · Cited by 320 · see citing works
Fast and scalable polynomial kernels via explicit feature maps
Pham, N., & Pagh, R. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487591 · Cited by 319 · see citing works
Xu, R., Xiong, C., Chen, W. R., & Corso, J. J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9512 · Cited by 309 · see citing works
Combining Language and Vision with a Multimodal Skip-gram Model
Lazaridou, A., Pham, N. T., & Baroni, M. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1016 · Cited by 283 · see citing works
Multi-source Deep Learning for Human Pose Estimation
Ouyang, W., Chu, X., & Wang, X. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.299 · Cited by 269 · see citing works
Kernel Independent Component Analysis
Yu, X., Hu, D., & Xu, J. (2014) · doi:10.1002/9781118679852.ch9 · Cited by 232 · see citing works
Cristianini, N., Shawe-Taylor, J. S., & Lodhi, H. M. (2002) · Journal of Intelligent Information Systems · doi:10.1023/a:1013625426931 · Cited by 229 · see citing works
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228 · see citing works
Unsupervised Generative Adversarial Cross-Modal Hashing
Zhang, J., Peng, Y., & Yuan, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11263 · Cited by 224 · see citing works
Learning Grounded Meaning Representations with Autoencoders
Silberer, C., & Lapata, M. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1068 · Cited by 221 · see citing works
Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval
Wu, L. Y., Wang, Y., & Shao, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2878970 · Cited by 218 · see citing works
Video Captioning by Adversarial LSTM
Yang, Y., Zhou, J., Ai, J., Bin, Y., Hanjalić, A., Shen, H. T., & Ji, Y. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2855422 · Cited by 214 · see citing works
Deep adversarial metric learning for cross-modal retrieval
Xu, X., He, L., Lu, H., Gao, L., & Ji, Y. (2018) · World Wide Web · doi:10.1007/s11280-018-0541-x · Cited by 210 · see citing works
Recurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 209 · see citing works
Audio-visual deep learning for noise robust speech recognition
Huang, J., & Kingsbury, B. E. D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639140 · Cited by 185 · see citing works
Learning Representations for Multimodal Data with Deep Belief Nets
Srivastava, N., & Salakhutdinov, R. (2012) · Cited by 181 · see citing works
Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network
Peng, Y., Qi, J., & Yuan, Y. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2852503 · Cited by 157 · see citing works
Effective multi-modal retrieval based on stacked auto-encoders
Wang, W., Ooi, B. C., Yang, X., et al. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732296.2732301 · Cited by 144 · see citing works
Unsupervised learning of acoustic features via deep canonical correlation analysis
Wang, W., Arora, R., Livescu, K., & Bilmes, J. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178840 · Cited by 140 · see citing works
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Y., Xiang, S., Kang, C., Wang, J., & Pan, C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2558463 · Cited by 135 · see citing works
Self-citationDeep Multilingual Correlation for Improved Word Embeddings
Lu, A., Wang, W., Bansal, M., Gimpel, K., & Livescu, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1028 · Cited by 134 · see citing works
Multimodal Keyless Attention Fusion for Video Classification
Long, X., Gan, C., de Melo, G., Liu, X., Li, Y., Li, F., & Wen, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12319 · Cited by 125 · see citing works
Aytar, Y., Castrejón, L., Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2753232 · Cited by 123 · see citing works
Structured Attentions for Visual Question Answering
Zhu, C., Zhao, Y., Huang, S., Tu, K., & Ma, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.145 · Cited by 121 · see citing works
Deep Coupled Metric Learning for Cross-Modal Matching
Liong, V. E., Lu, J., Tan, Y., et al. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2646180 · Cited by 120 · see citing works
Gan, C., Li, Y., Li, H., Sun, C., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.201 · Cited by 116 · see citing works
Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts
Wu, S., Bondugula, S., Florian Luisier, A., Zhuang, X., & Natarajan, P. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.341 · Cited by 113 · see citing works
Deep multimodal hashing with orthogonal regularization
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · International Conference on Artificial Intelligence · Cited by 111 · see citing works
Multimodal video classification with stacked contractive autoencoders
Liu, Y., Xiao-qing, F., & Zhou, Z. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.01.001 · Cited by 96 · see citing works
Precise Recovery of Latent Vectors from Generative Adversarial NetworksPreprint
Lipton, Z. C., & Tripathi, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04782 · Cited by 91 · see citing works
Deep Variational Canonical Correlation Analysis
Wang, W., Yan, X., Lee, H., & Livescu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.03454 · Cited by 89 · see citing works
Temporal Attention-Gated Model for Robust Sequence Classification
Pei, W., Baltrušaitis, T., Tax, D. M. J., & Morency, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.94 · Cited by 87 · see citing works
Video2vec Embeddings Recognize Events When Examples Are Scarce
Habibian, A., Mensink, T., & Snoek, C. G. M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2627563 · Cited by 76 · see citing works
Sparse Multigraph Embedding for Multimodal Feature Representation
Wang, S., & Guo, W. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2663324 · Cited by 75 · see citing works
Self-citationSemi-Supervised Image-to-Video Adaptation for Video Action Recognition
Zhang, J., Han, Y., Tang, J., Hu, Q., & Jiang, J. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2535122 · Cited by 68 · see citing works
Kernel CCA for multi-view learning of acoustic features using articulatory measurements.
Arora, R., & Livescu, K. (2012) · Cited by 61 · see citing works
Stochastic optimization for deep CCA via nonlinear orthogonal iterations
Wang, W., Arora, R., Livescu, K., & Srebro, N. (2015) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2015.7447071 · Cited by 59 · see citing works
CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation LearningPreprint
Peng, Y., Qi, J., & Yuan, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05106 · Cited by 58 · see citing works
Scalable and Effective Deep CCA via Soft Decorrelation
Chang, X., Xiang, T., & Hospedales, T. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00161 · Cited by 55 · see citing works
Deep correspondence restricted Boltzmann machine for cross-modal retrieval
Feng, F., Li, R., & Wang, X. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.12.020 · Cited by 54 · see citing works
Cross-modal Common Representation Learning by Hybrid Transfer Network
Huang, X., Peng, Y., & Yuan, M. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/263 · Cited by 54 · see citing works
Deep Cross-Media Knowledge Transfer
Huang, X., & Peng, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00921 · Cited by 47 · see citing works
Multi-source deep learning for information trustworthiness estimation
Ge, L., Gao, J., Xiao-yi, L., & Zhang, A. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487612 · Cited by 45 · see citing works
Cross-modal Bidirectional Translation via Reinforcement Learning
Qi, J., & Peng, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/365 · Cited by 41 · see citing works
AMC: Attention Guided Multi-modal Correlation Learning for Image Search
Chen, K., Bui, T., Chen, F., Wang, Z., & Nevatia, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.657 · Cited by 36 · see citing works
Mutlimodal Learning with Deep Boltzmann Machine for Emotion Prediction in User Generated Videos
Pang, L., & Ngo, C. (2015) · ACM International Conference on Multimedia Retrieval (ICMR) · doi:10.1145/2671188.2749400 · Cited by 32 · see citing works
Radical-Based Hierarchical Embeddings for Chinese Sentiment Analysis at Sentence Level.
Peng, H., Cambria, E., & Zou, X. (2017) · The Florida AI Research Society · Cited by 32 · see citing works
Bridging Languages through Images with Deep Partial Canonical Correlation Analysis
Rotman, G., Vulić, I., & Reichart, R. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1084 · Cited by 27 · see citing works
Acoustic Feature Learning via Deep Variational Canonical Correlation Analysis
Tang, Q., Wang, W., & Livescu, K. (2017) · Interspeech · doi:10.21437/interspeech.2017-1581 · Cited by 20 · see citing works
Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music RetrievalPreprint
Yu, Y., Tang, S., Raposo, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08976 · Cited by 15 · see citing works
Relation Classification via LSTMs Based on Sequence and Tree Structure
Dai, Y., Guo, W., Chen, X., & Zhang, Z. (2018) · IEEE Access · doi:10.1109/access.2018.2877934 · Cited by 15 · see citing works
Self-citationMultimodal Storytelling via Generative Adversarial Imitation Learning
Chen, Z., Zhang, X., Boedihardjo, A. P., Dai, J., & Lu, C. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/554 · Cited by 13 · see citing works
Inverting The Generator Of A Generative Adversarial NetworkPreprint
Creswell, A., & Bharath, A. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05644 · Cited by 13 · see citing works
Object co-segmentation via weakly supervised data fusion
Wang, S., Zhang, H., & Wang, H. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.11.004 · Cited by 10 · see citing works
Self-citationDeep Canonically Correlated LSTMsPreprint
Mallinar, N., & Rosset, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05407 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Y. Li, M. Yang, and Z. Zhang. (2016). ''A survey of multi-view represen- tation learning.'' [Online]. Available: https://arxiv.org/abs/1610.01206 · arXiv
- R. Sennrich, B. Haddow, and A. Birch, ''Neural machine translation of rare words with subword units,'' in Proc. 54th Annu. Meeting Assoc. Comput. Linguistics, vol. 1, 2016, pp. 1715-1725.
- M. Schuster and K. K. Paliwal, ''Bidirectional recurrent neural net- works,'' IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673-2681, Nov. 1997.
- A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, ''A kernel two-sample test,'' J. Mach. Learn. Res., vol. 13, pp. 723-773, Mar. 2012.
- N. Mor, L. Wolf, A. Polyak, and Y. Taigman. (2018). ''A universal music translation network.'' [Online]. Available: https://arxiv.org/abs/ 1805.07848
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ''BLEU: A method for automatic evaluation of machine translation,'' in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics, 2002, pp. 311-318.
- N. Srivastava and R. R. Salakhutdinov, ''Multimodal learning with deep Boltzmann machines,'' in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 2222-2230.
- C. K. I. Williams and M. Seeger, ''Using the Nyström method to speed up kernel machines,'' in Proc. Adv. Neural Inf. Process. Syst., 2001, pp. 682-688.
- Y. Takashima, T. Takiguchi, Y. Ariki, and K. Omori, ''Audio-visual speech recognition for a person with severe hearing loss using deep canonical correlation analysis,'' in Proc. 1st Int. Workshop Challenges Hearing Assistive Technol., 2017, pp. 77-81.
- A. Radford, L. Metz, and S. Chintala, ''Unsupervised representation learning with deep convolutional generative adversarial networks,'' in Proc. Int. Conf. Learn. Represent., 2016, pp. 1-16.
- X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, ''InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets,'' in Proc. 30th Int. Conf. Neural Inf. Process. Syst., 2016, pp. 2172-2180.
- V. Dumoulin et al., ''Adversarially learned inference,'' in Proc. Int. Conf. Learn. Represent., 2017, pp. 1-18.
- J. Donahue, P. Krähenbuhl, and T. Darrell, ''Adversarial feature learning,'' in Proc. Int. Conf. Learn. Represent., 2017, pp. 1-18.
- S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, ''Learning what and where to draw,'' in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 217-225.
- T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, ''Improved techniques for training GANs,'' in Proc. Adv. Neural Inf. Process. Syst., 2016, pp. 2234-2242.
- L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, ''Unrolled genera- tive adversarial networks,'' in Proc. Int. Conf. Learn. Represent., 2017, pp. 1-25.
- A. Vaswani et al., ''Attention is all you need,'' in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 5998-6008.
Cited by
Sorted by citations
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., & Yang, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.343 · Cited by 482 · see citing works
Long-term cancer survival prediction using multimodal deep learning
Vale-Silva, L. A., & Rohr, K. (2021) · Scientific Reports · doi:10.1038/s41598-021-92799-4 · Cited by 268 · see citing works
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Collaboration between clinicians and vision–language models in radiology report generation
Tanno, R., Barrett, D. G. T., Sellergren, A., Ghaisas, S., Dathathri, S., See, A., Welbl, J., Lau, C., Tu, T., Azizi, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03302-1 · Cited by 186 · see citing works
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126 · see citing works
A systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95 · see citing works
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71 · see citing works
Fusion of tactile and visual information in deep learning models for object recognition
Babadian, R. P., Faez, K., Amiri, M., & Falotico, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.032 · Cited by 67 · see citing works
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Rajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., & Wang, F. (2024) · Patterns · doi:10.1016/j.patter.2023.100913 · Cited by 47 · see citing works
Dynamic alignment and fusion of multimodal physiological patterns for stress recognition
Zhang, X., Wei, X., Zhou, Z., Zhao, Q., Zhang, S., Yang, Y., Li, R., & Hu, B. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3290177 · Cited by 38 · see citing works
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26 · see citing works
Learning cortical representations through perturbed and adversarial dreaming
Deperrois, N., Petrovici, M. A., Senn, W., & Jordan, J. (2022) · eLife · doi:10.7554/elife.76384 · Cited by 28 · see citing works
Multimodal Federated Learning: A Survey
Che, L., Wang, J., Zhou, Y., & Ma, F. (2023) · doi:10.20944/preprints202307.1420.v1 · Cited by 127 · see citing works
Self-citationA Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition
Liu, F., Chen, J., Tan, W., & Cai, C. (2021) · Entropy · doi:10.3390/e23101349 · Cited by 13 · see citing works
Ye, C., Zhang, R., Li, X., Deng, W., Wang, J., & Shao, S. (2025) · Information · doi:10.3390/info16121091 · Cited by 1 · see citing works
- Authors
- Guo, W., Wang, J., & Wang, S.
- Affiliations
- Fuzhou University (CN)
- Fujian Normal University (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 63373-63394
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2946165673
- Cited by
- 535
- References
- 154
- Citations by year
- 2026: 68 · 2025: 92 · 2024: 101 · 2023: 104 · 2022: 84 · 2021: 54 · 2020: 31 · 2019: 1
- FWCI
- 18.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61502104, 61672159)
- Fujian Collaborative Innovation Center for Big Data Application in Governments
- Technology Innovation Platform Project of Fujian Province (2014H2005)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- multimodal representation learning, deep learning, modality heterogeneity, joint representation learning, encoder-decoder architecture, generative adversarial networks, attention mechanism, multimodal data integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
