Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transformer models have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps.
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
The Abstract reports that MulT captures interactions across distinct time steps, and the Experiments report strong results on multimodal time-series. But the Proposed Method’s “Discussion about Attention & Alignment” explicitly says MulT does not resolve non-alignment by aligning the streams; it uses crossmodal attention to find relevant information instead. Thus the claim’s description of automatic alignment misstates the mechanism. (LLM, conf=0.84)
Experimental — accuracy not yet measuredMulT merged multiple pairwise cross-modal transformers for learning attention across modalities and then fused embedding from different modalities.
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
Multimodal fusion methods improve emotion recognition performance.
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
Different modalities are complementary for emotion expression.
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
TinyBERT reduces the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
The Introduction and Related Works sections describe MulT, a transformer for unaligned multimodal sequences, rather than TinyBERT. Searches for TinyBERT, knowledge distillation, model compression, and mobile deployment found no passage addressing the attributed claim; page numbers were not available in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
(2011) · Cited by 2,752 · see citing works
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · see citing works
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · cited 1× · see citing works
“Transformer network (Vaswani et al., 2017) was first introduced for neural machine translation (NMT) tasks, where the encoder and decoder side each leverages a self-attention (Parikh et al., 2016; Lin et al., 2017; Vaswani et al., 2017) transformer.”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · see citing works
Self-citationContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · see citing works
Self-citationFacial signs of emotional experience.
Ekman, P., Freisen, W. V., & Ancoli, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077722 · Cited by 888 · see citing works
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
Self-citationTools, Language and Cognition in Human Evolution.
Latour, B., Gibson, K. R., & Ingold, T. (1994) · Man · doi:10.2307/2804508 · Cited by 824 · see citing works
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · see citing works
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 600 · see citing works
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.12024 · Cited by 535 · see citing works
Self-citationPham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works
Self-citationWords Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 493 · see citing works
Self-citationThe Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G. F., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1008 · Cited by 484 · see citing works
Linguistically-Informed Self-Attention for Semantic Role Labeling
Strubell, E., Verga, P. W., Andor, D., Weiss, D., & McCallum, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1548 · Cited by 412 · see citing works
Learning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works
Self-citationCombining Language and Vision with a Multimodal Skip-gram Model
Lazaridou, A., Pham, N. T., & Baroni, M. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1016 · Cited by 283 · see citing works
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255 · see citing works
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · see citing works
Self-citationMultimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
Online Segment to Segment Neural Transduction
Yu, L., Buys, J., & Blunsom, P. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1138 · Cited by 77 · see citing works
A Decomposable Attention Model for Natural Language InferencePreprint
Parikh, A. P., Täckström, O., Das, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01933 · Cited by 57 · see citing works
Transformer-XL: Language Modeling with Longer-Term Dependency
Dai, Z., Yang, Z., Yang, Y., Cohen, W. W., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2018) · Cited by 38 · see citing works
Self-citationWhy Self-Attention? A Targeted Evaluation of Neural Machine Translation ArchitecturesPreprint
Tang, G., Müller, M., Rios, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08946 · Cited by 33 · see citing works
Sentiment Analysis using Imperfect Views from Spoken Language and Acoustic Modalities
Sheikh, I., Dumpala, S. H., Chakraborty, R., & Kopparapu, S. K. (2018) · doi:10.18653/v1/w18-3305 · Cited by 12 · see citing works
Online Segment to Segment Neural TransductionPreprint
Yu, L., Buys, J., & Blunsom, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08194 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
“This kind of hierarchically cross-modal interaction is also widely studied [137], [189], and alleviates the drawback of cross-attention.”
BackgroundYu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816 · see citing works
“In forward-guidance methods, researches are devoted to design interactive modules for capturing the cross-modal information (Zadeh et al. 2018a; Sun et al. 2020; Tsai et al. 2019).”
BackgroundMethodInfluentialIntegrating Multimodal Information in Large Pretrained Transformers
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A. B., Mao, C., Morency, L., & Hoque, M. E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.214 · Cited by 743 · see citing works
“MulT (Multimodal Transformer for Unaligned Multimodal Language Sequence) uses three sets of Transformers and combines their output in a late fusion manner to model a multimodal sequence (Tsai et al., 2019).”
BackgroundMethodInfluentialSelf-citationTrusted Multi-View Classification With Dynamic Evidential Fusion
Han, Z., Zhang, C., Fu, H., & Zhou, J. T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3171983 · Cited by 661 · see citing works
“For multi-view learning, traditional methods usually assume an equal value for different views [10], [11], [12], [13] or learn a fixed weight factor for each [14], [15], [16], [17], [18].”
BackgroundHan, W., Chen, H., & Poria, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.723 · Cited by 570 · see citing works
“…h t ∈ R d t ( d t =768) to lower ones h v ∈ R d v and h a ∈ R d a ( d v , d a ¡ 50) converges faster with higher accuracy; 2) many previous works (Tsai et al., 2019a; Hazarika et al., 2020) pointed out that from empirical study the text modality is predominate, which can integrate more…”
BackgroundMethodInfluentialCH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality
Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., & Yang, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.343 · Cited by 482 · see citing works
“The Multimodal Transformer (MULT) (Tsai et al., 2019) using the directional pairwise crossmodal attention to realize the interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another.”
BackgroundMethodInfluentialMultimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 368 · see citing works
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation
Hu, J., Liu, Y., Zhao, J., & Jin, Q. (2021) · doi:10.18653/v1/2021.acl-long.440 · Cited by 340 · see citing works
“To verify the effectiveness of MMGCN in multi-modal fusion, we compare it with other multimodal fusion methods, including early fusion, late fusion, fusion through gated attention and other representative fusion methods such as MFN(Zadeh et al., 2018) and MulT(Tsai et al., 2019).”
MethodTETFN: A text enhanced transformer fusion network for multimodal sentiment analysis
Wang, D., Guo, X., Tian, Y., Liu, J., He, L., & Luo, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109259 · Cited by 349 · see citing works
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410 · see citing works
“A multimodal transformer is proposed by (Tsai et al. 2019) that uses attention based cross-modal transformers to learn interactions between modalities.”
BackgroundCTNet: Conversational Transformer Network for Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3049898 · Cited by 300 · see citing works
“that multimodal features are already aligned in the utterance level [12], [17] or the word level [21], [22].”
BackgroundMethodInfluentialYu, J., Jiang, J., Li, Y., & Xia, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.306 · Cited by 305 · see citing works
“2.b, to learn better word representations with the guidance of associated images, we first employ an m -head cross-modal attention mechanism (Tsai et al., 2019), by treating V ∈ R d × 49 as queries, and R ∈ R d × ( n +1) as keys and values: where CA i refers to the i -th head of cross-modal…”
MethodBisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J. Y., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 325 · see citing works
“, 2018)) or further broadened to include audio (Tsai et al., 2019).”
BackgroundHybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Zheng, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3172360 · Cited by 301 · see citing works
“Following previous works [10], [12], we utilize 1,284 utterances for training, 229 utterances for validation, and 686 utterances for testing.”
MethodInfluentialUniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition
Hu, G., Lin, T.-E., Zhao, Y., Lu, G., Wu, Y., & Li, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.534 · Cited by 304 · see citing works
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation
Ma, X., Zhang, X., Pun, M.-O., & Liu, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3373033 · Cited by 270 · see citing works
“More recently, the transformer architecture has been considered for multimodal fusion in semantic segmentation for its outstanding capability in extracting contextual information [20], [26], [32], [54] and fusing different modalities [19], [38], [55], [56], [57], [58], [59].”
MethodSun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 283 · see citing works
“Recently, Multimodal Transformer (MulT) [12] has been proposed to directly model cross-modal correlations in unaligned multimodal sequences.”
BackgroundMethodResultInfluentialBi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Han, W., Chen, H., Gelbukh, A. F., Zadeh, A., Morency, L., & Poria, S. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479919 · Cited by 261 · see citing works
“Since it has been empirically shown that the text modality is most significant [26, 34], our model takes two text-related modality pairs, TV (text-visual) and TA (textacoustic), as respective inputs for its two bimodal learning modules.”
BackgroundMethodInfluentialSelf-citationSong, C., Ning, N., Zhang, Y., & Wu, B. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102437 · Cited by 256 · see citing works
Mai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237 · see citing works
“For modality translation methods, Multimodal Transformer (MulT) (Tsai et al. 2019a) transforms source modality to target modality using directional pairwise cross-modal attention.”
MethodSelf-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 247 · see citing works
“Using audio, vision and language to learn representations has also been explored in past work [11, 31, 38, 50, 83].”
BackgroundMissing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
Zhao, J., Li, R., & Jin, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.203 · Cited by 225 · see citing works
“Previous works have shown that these different modalities are complimentary for emotion expression, and proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇsaitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
BackgroundDecoupled Multimodal Distilling for Emotion Recognition
Li, Y., Wang, Y., & Cui, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00641 · Cited by 249 · see citing works
“[17] designed a progressive modality reinforcement method based on [28], it aims to learn the potential adaption and correlations from multimodal representation to unimodal representation.”
BackgroundMethodInfluentialLv, F., Chen, X., Huang, Y., Duan, L., & Lin, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00258 · Cited by 221 · see citing works
Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
Qu, L., Zhou, Y., Liang, P. P., Xia, Y., Wang, F., Adeli, E., Fei-Fei, L., & Rubin, D. L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00982 · Cited by 220 · see citing works
“This property has led to the prevalence of Transformers in self-supervised learning where heterogeneity is manifested via distribution shifts between unlabeled pretraining data and labeled test data [11], as well as in multimodal learning over fundamentally heterogeneous input modalities such as image and text [24, 60…”
BackgroundSelf-citationA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
“For example, Tsai et al. [114] proposed a multimodal Transformer (MULT) for unaligned multimodal language sequences.”
BackgroundLearning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
Zhang, H., Wang, Y., Yin, G., Liu, K., Liu, Y., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.49 · Cited by 213 · see citing works
“In MSA, this technique has been widely used for feature extraction, representation learning, and multimodal fusion (Tsai et al., 2019a; Huang et al., 2020; Liu et al., 2023b; Yuan et al., 2021).”
BackgroundMethodInfluentialMultimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification
Han, Z., Yang, F., Huang, J., Zhang, C., & Yao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02005 · Cited by 202 · see citing works
“Intermediate fusion strategy is widely adopted in multimodal learning, which allows multiple modalities to be fused at any layer through a well-designed network [6,26,29,30,32,33,37,38,58,66].”
BackgroundMethodFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
“To increase the expressiveness of CCA, several nonlinear extensions have been proposed including Kernel CCA [168], Deep CCA [21], and CCA Autoencoders [361].”
BackgroundMethodSelf-citationMultimodal Transformer Fusion for Continuous Emotion Recognition
Huang, J., Tao, J., Liu, B., Lian, Z., & Niu, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053762 · Cited by 187 · see citing works
“Tsai et al. [14] utilized Transformer to analyze human multimodal language and exhibited the best performance.”
BackgroundAre Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198 · see citing works
“So far, research on Trans-former models has been limited to developing new architectures for fusion [29, 35, 38] or exploring better self-supervised learning tasks [1, 6, 8, 47, 48].”
Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis
Yuan, Z., Li, W., Xu, H., & Yu, W. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475585 · Cited by 211 · see citing works
“Multimodal transformer (MulT) [22] uses a crossmodal attention mechanism to capture the relationship between different modalities.”
BackgroundMethodInfluentialConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis
Yang, J., Yu, Y., Niu, D., Guo, W., & Xu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.421 · Cited by 189 · see citing works
“, 2017), MulT (Tsai et al., 2019), MISA (Hazarika et al.”
BackgroundInfluentialA Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
Ma, H., Wang, J., Lin, H., Zhang, B., Zhang, Y., & Xu, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3271019 · Cited by 188 · see citing works
“Rahman et al. [35] fine-tuned large pre-trained transformer models for multimodal language by designing a multimodal adaptation gate (MAG).”
BackgroundA Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data
Ma, X., Zhang, X., & Pun, M.-O. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3165005 · Cited by 184 · see citing works
“More recently, it was shown in MulT [29] that the cross-attention mechanism can provide a latent crossmodal adaptation that fuses crossmodal information by directly attending to low-level features in other modalities.”
BackgroundState of the art: a review of sentiment analysis based on sequential transfer learning
Chan, J. Y.-L., Bea, K. T., Leow, S. M. H., Phoong, S. W., & Cheng, W. K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10183-8 · Cited by 187 · see citing works
Cross-Modal Enhancement Network for Multimodal Sentiment Analysis
Wang, D., Liu, S., Wang, Q., Tian, Y., He, L., & Gao, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3183830 · Cited by 191 · see citing works
“B ASED ON [54] AND [11] [2], [46], [47], AND [8], R ESPECTIVELY Self-MM(S), MAG-SentiLARE(S) which also use SentiLARE as their language model.”
BackgroundInfluentialLiu, J., Zang, H., Cheng, L., et al. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121160 · Cited by 175 · see citing works
Mai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182 · see citing works
“Furthermore, modality-translation methods such as Multimodal Cyclic Translation Network (MCTN) [24] and Multi-modal Transformer [6] aim at learning joint representation by translating source modality into the target one.”
BackgroundMethodTransformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel
Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1443 · Cited by 173 · see citing works
“…allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Lee et al., 2018; Wang et al., 2018; Tsai et al., 2019a).”
BackgroundMethodInfluentialSelf-citationCDTrans: Cross-domain Transformer for Unsupervised Domain AdaptationPreprint
Xu, T., Chen, W., Wang, P., Wang, F., Li, H., & Jin, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06165 · Cited by 195 · see citing works
“For multi-modal based networks, there are several works (Tsai et al., 2019; Li et al., 2021e; Hu & Singh, 2021) that apply cross-attention for multi-modal feature fusion, which demonstrates that attention mechanism is powerful at distilling noise and feature alignment.”
BackgroundMethodInfluentialCubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
Sun, H., Wang, H., Liu, J., Chen, Y., & Lin, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548025 · Cited by 176 · see citing works
“Tsai [33] proposed a Multimodal Transformer (MulT) and attempted to calculate cross-modal attention”
BackgroundMethodM2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation
Chudasama, V. M., Kar, P., Gudmalwar, A. P., Shah, N. J., Wasnik, P., & Onoe, N. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00511 · Cited by 170 · see citing works
“On a similar dataset, the CMU-Mosei methods like Loshchilov et al [3] and Tsai et al [31] use multi-head attention based fusion [32] for multi-modal emotion recognition.”
MethodSupervised Multimodal Bitransformers for Classifying Images and TextPreprint
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 271 · see citing works
MSER: Multimodal speech emotion recognition using cross-attention with deep fusion
Khan, M., Gueaieb, W., Saddik, A. E., & Kwon, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122946 · Cited by 162 · see citing works
Bayoudh, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 173 · see citing works
Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T. I., Billinghurst, M., & Nanayakkara, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 164 · see citing works
“Since the dataset is not uniformly distributed among the classes, we followed prior work [4], [40], and only”
BackgroundMethodInfluentialJaume, G., Vaidya, A. J., Chen, R. J., Williamson, D. F. K., Liang, P. P., & Mahmood, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01100 · Cited by 161 · see citing works
“In parallel, the use of Transformers for multimodal fusion has gained significant attention in classification and generative tasks [58, 67, 83].”
Self-citationTMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis
Huang, J., Zhou, J. W., Tang, Z., Lin, J., & Chen, C. Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111346 · Cited by 162 · see citing works
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
Shiri, F. M., Perumal, T., Mustapha, N., & Mohamed, R. (2023) · Journal of Artificial Intelligence · doi:10.32604/jai.2024.054314 · Cited by 537 · see citing works
Hierarchical Interactive Multimodal Transformer for Aspect-Based Multimodal Sentiment Analysis
Yu, J., Chen, K., & Xia, R. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3171091 · Cited by 148 · see citing works
“Note that an intuitive solution to capture the inter-modal interaction is to employ the Cross-Modal Transformer (CMT) layer [43].”
BackgroundMethodAOBERT: All-modalities-in-One BERT for multimodal sentiment analysis
Kim, K.-H., & Park, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.022 · Cited by 148 · see citing works
Multimodal Video Sentiment Analysis Using Deep Learning Approaches, a Survey
Abdu, S. A., Yousef, A. H., & Salem, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.003 · Cited by 147 · see citing works
Cross-Modal Multitask Transformer for End-to-End Multimodal Aspect-Based Sentiment Analysis
Yang, L., Na, J., & Yu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103038 · Cited by 149 · see citing works
Quantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143 · see citing works
“Multimodal Transformer (MulT) [8] takes a different approach by directly computing the inter-modal representations via the encoder of Transformer [34], resulting in a simpler but better-performed network.”
BackgroundMethodInfluentialLi, J., Wang, X., Lv, G., & Zeng, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3260635 · Cited by 146 · see citing works
Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 142 · see citing works
MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
Chen, J., & Ho, C. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00086 · Cited by 139 · see citing works
“Inspired by the strong progress made in learning contextualized language representations using transformers [13, 42, 53, 77], several recent works have extended uni-modal transformers to the multimodal setting by adding additional transformer stream(s) to learn joint vision-language representations [44, 52], visionaud…”
BackgroundAdvances in Sentiment Analysis - Techniques, Applications, and Challenges
Li, J. (2023) · Artificial intelligence · doi:10.5772/intechopen.111293 · Cited by 137 · see citing works
Kalateh, S., Estrada-Jimenez, L. A., Nikghadam-Hojjati, S., & Barata, J. (2024) · IEEE Access · doi:10.1109/access.2024.3430850 · Cited by 151 · see citing works
“Tsai et al. [205], [206] developed a multimodal transformer model that integrates language, acoustic, and visual data, using attention mechanisms to capture interactions between different modalities, leading to more accurate emotion predictions.”
BackgroundSign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
Camgöz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01004 · Cited by 164 · see citing works
Video sentiment analysis with bimodal information-augmented multi-head attention
Wu, T., Peng, J., Zhang, W., Zhang, H., Tan, S., Yi, F., Ma, C., & Huang, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107676 · Cited by 137 · see citing works
“The MulT [41] uses its cross-modal attention module to extract the key information inside each modality and then merges these features based on the Transformer [45] model.”
BackgroundMethodInfluentialMultimodal Sparse Transformer Network for Audio-Visual Speech Recognition
Song, Q., Sun, B., & Shutao Li, A. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3163771 · Cited by 127 · see citing works
“The effective way to fuse cross-modal information through a potential cross-modal adaptation has been verified in [36] and [37].”
BackgroundSiriwardhana, S., Reis, A., Weerasekera, R., & Nanayakkara, S. C. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1212 · Cited by 122 · see citing works
“Performance of the other models are taken from ICCCN (2019) [39] MULT (2019) [8]”
BackgroundMethodInfluentialZhu, X., Wang, Y., Cambria, E., Rida, I., López, J. S., Cui, L., & Wang, R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103268 · Cited by 130 · see citing works
Deep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128 · see citing works
Wu, Y., Lin, Z., Zhao, Y., Qin, B., & Zhu, L.-N. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.417 · Cited by 129 · see citing works
“Tsai et al. (2019) presented multimodal transformer (Mult), which uses the cross-modal attention to capture the bimodal interactions, motivated by the great success of transformer in NLP(Vaswani et al., 2017).”
BackgroundMethodZou, B., Han, J., Wang, Y., Liu, R., Zhao, S., Feng, L., Lyu, X., & Ma, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3181210 · Cited by 114 · see citing works
“Instead of early fusion, the second one is based on more advanced multimodal transformer fusion [56] which can handle inherent data non-alignment and long-range dependencies across modalities.”
BackgroundEnd-to-end multimodal affect recognition in real-world environments
Tzirakis, P., Chen, J., Zafeiriou, S., & Schuller, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.10.011 · Cited by 105 · see citing works
“Learning rate [10−3 − 10−6] Batch size (Text/audio) [10, 20, 30]”
A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning
Wang, L., Peng, J., Zheng, C., Zhao, T., & Zhu, L. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103675 · Cited by 111 · see citing works
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
DepMSTAT: Multimodal Spatio-Temporal Attentional Transformer for Depression Detection
Tao, Y., Yang, M., Li, H., Wu, Y., & Hu, B. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3350071 · Cited by 114 · see citing works
“…based on the Transformer and Attention mechanisms in various fields has contributed significantly to the development of their respective fields, e.g. humour detection [35], product detection [36], data alignment in time series modelling of human behaviour [37], dance choreography [38], and so on.”
BackgroundDistribution-Consistent Modal Recovering for Incomplete Multimodal Learning
Wang, Y., Cui, Z., & Li, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02013 · Cited by 111 · see citing works
“Multimodal machine learning dedicates to designing a strong model for understanding, reasoning, and learning by fusing multimodal data, such as language, acoustic, image, et al [15, 24].”
BackgroundMethodGated attention fusion network for multimodal sentiment classification
Du, Y., Liu, Y., Peng, Z. J., & Jin, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.108107 · Cited by 107 · see citing works
Feature Fusion Vision Transformer for Fine-Grained Visual Categorization
Wang, J., Yu, X., & Gao, Y. (2021) · Warwick Research Archive Portal (University of Warwick) · doi:10.48550/arxiv.2107.02341 · Cited by 106 · see citing works
“Transformer has achieved huge success in natural language processing [4, 25, 26].”
BackgroundWu, Y., Daoudi, M., & Amad, A. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3263907 · Cited by 114 · see citing works
Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 109 · see citing works
“Unsupervised explicit alignment aligns a modality without having any direct labels, as in [53], [157], [259], [395], and [406],”
BackgroundInfluentialMTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Yang, J., Wang, Y., Yi, R., Zhu, Y., Rehman, A., Zadeh, A., Poria, S., & Morency, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.79 · Cited by 104 · see citing works
“For basleine evaluations, we use Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) (Tsai et al., 2019a) as baselines.”
MethodResultInfluentialSelf-citationMultimodal sentiment analysis with unidirectional modality translation
Yang, B., Shao, B., Wu, L., & Lin, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.041 · Cited by 100 · see citing works
Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion
Ng, E., Joo, H., Hu, L., Li, H., Darrell, T., Kanazawa, A., & Ginosar, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01975 · Cited by 99 · see citing works
Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion
Xie, B., Sidulova, M., & Park, C. H. (2021) · Sensors · doi:10.3390/s21144913 · Cited by 108 · see citing works
“Nevertheless, our proposed network architecture extends the previous studies of multimodal emotion recognition with the crossmodal transformer [38,40], and the structure of the network can also be robustly expanded for a higher number of input modalities.”
BackgroundMethodInfluentialChen, J., Hu, Y., Lai, Q., Wang, W. W., Chen, J., Liu, H., Srivastava, G., Bashir, A. K., & Hu, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102017 · Cited by 99 · see citing works
M3Care: Learning with Missing Modalities in Multimodal Healthcare Data
Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., & Zhao, J. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539388 · Cited by 102 · see citing works
“• MulT [49] utilizes directional pairwise cross-modal transformers to attend to interactions between multimodal data.”
BackgroundMulti-Modal Attention for Speech Emotion Recognition
Pan, Z., Luo, Z., Yang, J., & Li, H. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1653 · Cited by 95 · see citing works
Ding, J., Chen, X., Lu, P., Yang, Z., Li, X., & Du, Y. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05439-1 · Cited by 95 · see citing works
“Multimodal transformer (MulT) [40], proposed by Tsai et al. (2021), can simulate the interactions between multimodal sequences across time steps, which information procession process is appropriate for our purposes of acting attitudes and behaviors among interlocutors.”
BackgroundMultimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
Tsai, Y.-H. H., Ma, M., Yang, M., Salakhutdinov, R., & Morency, L.-P. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.143 · Cited by 98 · see citing works
“Nonetheless, Graph-MFN did not separate the contributions among uni-modal and multimodal explanatory features, and MFM only provided the analysis on trimodal interaction feature.”
BackgroundMethodResultInfluentialSelf-citationAF: An Association-Based Fusion Method for Multi-Modal Classification
Liang, X., Qian, Y., Guo, Q. J., Cheng, H., & Liang, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3125995 · Cited by 98 · see citing works
Adaptive Multimodal Emotion Detection Architecture for Social Robots
Heredia, J., Lopes-Silva, E., Cardinale, Y., Díaz-Amado, J., Dongo, I., Graterol, W., & Aguilera, A. (2022) · IEEE Access · doi:10.1109/access.2022.3149214 · Cited by 95 · see citing works
“For instance, the document classification, based on text and visual data [47]; in a medical context, for early detection of Alzheimer’s disease stage using magnetic resonance imaging, genetic data, and clinical test data [48]; in music videos [49], [50]; or for multimodal languages [51]–[53].”
BackgroundLe, H.-D., Lee, G.-S., Kim, S.-H., Kim, S., & Yang, H.-J. (2023) · IEEE Access · doi:10.1109/access.2023.3244390 · Cited by 97 · see citing works
“Tsai et al. [7] proposed Multimodal Transformer (MulT) which uses pair-wise transformers to perform bidirectional cross-attention between visual-textual, visual-audio, and textual-audio for the unaligned multimodal affective recognition task.”
BackgroundMethodSKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis
Zhu, C., Chen, M., Zhang, S., Sun, C., Liang, H., Liu, Y., & Chen, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101958 · Cited by 93 · see citing works
Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis
Xiao, L., Wu, X., Yang, S., Xu, J., Zhou, J., & He, L. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103508 · Cited by 98 · see citing works
M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 6 · see citing works
“[69] utilized transformer attention mechanisms to learn both cross-modal alignment and interactions.”
BackgroundMethodInfluentialMultimodal End-to-End Sparse Model for Emotion Recognition
Dai, W., Cahyawijaya, S., Liu, Z., & Fung, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.417 · Cited by 91 · see citing works
“We follow the feature extraction procedure in the previous works (Zadeh et al., 2018b; Tsai et al., 2018, 2019; Rahman et al., 2020).”
BackgroundMethodInfluentialMultimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects
Warner, E., Lee, J., Hsu, W. W. Y., Syeda-Mahmood, T. F., Kahn, C. E., Gevaert, O., & Rao, A. U. K. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02032-8 · Cited by 93 · see citing works
“In the former, many studies attempt to form models that are bidirectional, whereby the intended output can be placed back as input and return an image similar to the original input image.”
BackgroundGkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 94 · see citing works
“Recently, attention mechanisms have been exploited for aligning different modalities, resulting in better-performed modality fusion approaches [9, 10, 11].”
BackgroundMethodInfluentialM-SENA: An Integrated Platform for Multimodal Sentiment Analysis
Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., & Gao, K. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.20 · Cited by 108 · see citing works
Improving multimodal fusion with Main Modal Transformer for emotion recognition in conversation
Zou, S., Huang, X., Shen, X., & Liu, H. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109978 · Cited by 91 · see citing works
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 91 · see citing works
Multi-modal Text Recognition Networks: Interactive Enhancements Between Visual and Semantic Features
Na, B., Kim, Y., & Park, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_26 · Cited by 91 · see citing works
“Multi-modal transformer [28], which consists of transformer layers processing multiple types of features at once, has been introduced in several domains such as visual question answering [10], vision-language navigation [4], autonomous driving [21], video retrieval [8], and many others.”
Background
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
