The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 101–200 of 1,004
Multimodal transformer with adaptive modality weighting for multimodal sentiment analysis
Wang, Y., He, J., Wang, D., Wang, Q., Wan, B., & Luo, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127181 · Cited by 104
SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3141237 · Cited by 92
“Recent works have verified the importance of intra- and cross-modal interactions for emotion recognition [21], [22].”
BackgroundMethodInfluentialBAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis
Tang, J., Liu, D., Jin, X., Peng, Y. K., Zhao, Q., Ding, Y., & Kong, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3218018 · Cited by 90
“MulT [21] proposed the directional pairwise cross-modality attention mechanism to capture the interactions between multimodal sequences across distinct time steps, and latently adapt the sequence from one modality to another.”
BackgroundInfluentialTeFNA: Text-centered fusion network with crossmodal attention for multimodal sentiment analysis
Huang, C., Zhang, J., Wu, X., Wang, Y., Li, M., & Huang, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110502 · Cited by 89
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89
“Differently, Multimodal Transformer ( MulT ) [26] uses a transformer [60] structure to translate each two modalities.”
BackgroundMethodMultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85
“Transformer models [158] are useful for temporal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
BackgroundMethodSelf-citationLiang, T., Lin, G., Feng, L., Zhang, Y., & Lv, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00804 · Cited by 93
Zhang, L., Wilson, R., Sumner, M., & Wu, Y. (2023) · Renewable Energy · doi:10.1016/j.renene.2023.118952 · Cited by 85
Tang, J., Li, K., Jin, X., Cichocki, A. S., Zhao, Q., & Kong, W. (2021) · doi:10.18653/v1/2021.acl-long.412 · Cited by 87
“Inspired by the success of the Transformer-based model, Tsai et al. introduced a directional cross-modality attention module to extend the standard Transformer network.”
MethodInfluentialPS-Mixer: A Polar-Vector and Strength-Vector Mixer Model for Multimodal Sentiment Analysis
Lin, H., Zhang, P., Ling, J., Yang, Z., Lee, L., & Liu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103229 · Cited by 85
MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00628 · Cited by 98
Pandey, A., & Vishwakarma, D. K. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.111206 · Cited by 84
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Chen, X., & Zhou, M. (2020) · arXiv (Cornell University) · Cited by 82
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81
“Besides, Transformer-like [42] architectures can be employed to model the complex temporal dynamics of emotion in the future.”
MethodDeep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition
Zhang, F., Li, X., Lim, C. P., Hua, Q., Dong, C.-R., & Zhai, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.07.006 · Cited by 81
CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition
Li, J., Wang, X., Liu, Y., & Zeng, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3389453 · Cited by 83
“In this paper, we adopt MHA networks to extract multimodal complementary and associative information, i.e., they are utilized to construct Attention based Cross-Modality Encoder (ACME).”
MethodDisentanglement Translation Network for multimodal sentiment analysis
Zeng, Y., Yan, W., Mai, S., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102031 · Cited by 85
Low Rank Fusion based Transformers for Multimodal Sequences
Sahay, S., Okur, E., Kumar, S. H., & Nachman, L. (2020) · doi:10.18653/v1/2020.challengehml-1.4 · Cited by 93
“We use the same setup as (Tsai et al., 2019) with 4 emotions (happy, sad, angry, neutral).”
BackgroundMethodInfluentialTailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition
Zhang, Y., Chen, M., Shen, J., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20895 · Cited by 85
“MulT (Tsai et al. 2019) fuses multi-modal information by directly attending Table 2: Predictive performance of TAILOR on multi-modal multi-label CMU-MOSEI dataset with aligned and unaligned multi-modal sequences compared with state-of-the-arts.”
BackgroundRobust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“We use multimodal transformer [53] as the base model for multimodal sentiment analysis.”
MethodMultimodal Sentiment Analysis With Two-Phase Multi-Task Learning
Yang, B., Wu, L., Zhu, J., Shao, B., Lin, X., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3178204 · Cited by 82
Self-attention fusion for audiovisual emotion recognition with incomplete data
Chumachenko, K., Iosifidis, A., & Gabbouj, M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956592 · Cited by 77
“In the context of emotion/affect recognition, notable works include [14], [16], [15].”
InfluentialNoise Imitation Based Adversarial Training for Robust Multimodal Sentiment Analysis
Yuan, Z., Liu, Y., Xu, H., & Gao, K. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3267882 · Cited by 87
“Moreover, following the previous works [46], [47], the Acc-2 and F1-Score metrics are utilized as negative/non-negative classification criteria.”
MethodMultimodal emotion recognition based on audio and text by using hybrid attention networks
Zhang, S., Yang, Y., Chen, C., Liu, R., Tao, X., Guo, W., Xu, Y., & Zhao, X. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105052 · Cited by 81
GPT-4V with emotion: A zero-shot benchmark for Generalized Emotion Recognition
Lian, Z., Sun, L., Sun, H., Chen, K., Wen, Z., Gu, H., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102367 · Cited by 99
Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection
Pramanick, S., Roy, A., & Johns, V. M. P. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00062 · Cited by 84
“Motivated by the success of Transformers [56, 23, 40, 14, 63, 31], several recent works have extended it to learn multimodal association by adding additional vision and speech modules to the Transformer framework [54, 33, 51, 37, 24, 25, 47].”
BackgroundSun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 81
“MulT [42] with its low-rank version LMF-MulT [43] both discard the traditional alignment for different modalities, but employs the stacked Transformer to expand the available temporal frames for alignment.”
BackgroundMethodInfluentialHumor Knowledge Enriched Transformer for Understanding Multimodal Humor
Hasan, M. K., Lee, S., Rahman, W., Zadeh, A., Mihalcea, R. F., Morency, L., & Hoque, M. E. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17534 · Cited by 80
“As our model is largely based on Transformer (Vaswani et al. 2017) architectures, we discuss some examples on how they have been expanded to model multiple modalities.”
BackgroundInfluentialSelf-citationSun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 92
“The most commonly used strategy is model-level fusion [6, 47, 48, 49, 50, 51, 52, 8].”
BackgroundMethodInfluentialFan, C., Zhu, K., Tao, J., Yi, G., Xue, J., & Lv, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3423671 · Cited by 81
“…are various methods of multimodal fusion, roughly divided into tensor-based [15], [23], [24], GAN-based [9], attention-based [16], [25], [26], [27], [28], graph-based [29], [30], operation-based [31], translation-based [32], [33], [34], routing-based methods [35], etc. [36] employ information…”
Zeng, Y., Li, Z., Tang, Z., Chen, Z., & Ma, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119240 · Cited by 74
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition
Li, D., Wang, Y., Funakoshi, K., & Okumura, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.996 · Cited by 92
MIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 77
“Specifically, MAG-BERT, MulT, and MISA increase accuracy scores of 1.28%, 0.92%, and 0.72% on twenty-class and 0.74%, 0.65%, and 0.36% on binary classification, respectively.”
BackgroundMethodInfluentialMulti-Task Momentum Distillation for Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282410 · Cited by 75
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
Yu, T., Dai, W., Liu, Z., & Fung, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.326 · Cited by 78
“Inspired by prior works (Yu et al., 2019; Tsai et al., 2019), we propose a vision guided multi-head attention mechanism for the text-vision fusion.”
MethodLu, P., Hu, L., Mitelpunkt, A., Bhatnagar, S., Lu, L. J., & Liang, H. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105669 · Cited by 80
“While many existing methods adopt a straightforward approach to integrate multimodal information by simply concatenating features extracted from different modalities, there is a growing interest in leveraging attention mechanisms to fuse multimodal data [36 – 38].”
AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis
Huang, C., Chen, J., Huang, Q., Wang, S., Tu, Y., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102725 · Cited by 76
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Lung Nodule Detection in Medical Images Based on Improved YOLOv5s
Ji, Z., Wu, Y., Zeng, X., An, Y., Li, Z., Wang, Z., & Ганчев, И. (2023) · IEEE Access · doi:10.1109/access.2023.3296530 · Cited by 72
“In 2017, Google used the Self-Attention model [36] to replace the Recurrent Neural Network (RNN) model [37], which was the most widely used in NLP tasks until then, causing a huge shock in the NLP field.”
MethodLi, M., Shi, J., Bai, L., Huang, C., Jiang, Y., Lü, K., Wang, S., & Hancock, E. R. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2024.111340 · Cited by 73
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73
“distribution matching across modalities due to two reasons: Firstly, as mentioned above, previous research has shown that emotion expression is not necessarily aligned at the frame level or word level across modalities [19, 32, 34, 46], forcing the matching at the frame level will lead to inherently poor optimization…”
BackgroundTime Series Analysis Based on Informer Algorithms: A Survey
Zhu, Q., Han, J., Chai, K., & Zhao, C. (2023) · Symmetry · doi:10.3390/sym15040951 · Cited by 76
“Following the formulas in references [18,19], the attention of the i-th query is defined as a kernel smoother in the form of probability: where p ( k j (cid:12)(cid:12) q i ) = k ( q i , k j ) / ∑ l k ( q i , k l ) and k ( q i , k j ) choose the asymmetric index kernel exp ( q i k j (cid:62) / √ d )…”
Toward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 87
“Most work uses attention to improve model performance without forcing an alignment in multimodal data, only visualizing attention activations for qualitative analysis [95], [96], [97], [98], [99], [100], [101].”
MethodTellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75
“In Appendix A, available online, we compare the proposed COLD fusion and a multimodal Transformer baseline [99] on 8)).”
MethodLu, Q., Sun, X., Gao, Z., Long, Y., Feng, J., & Zhang, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103538 · Cited by 69
Multimodal Phased Transformer for Sentiment Analysis
Cheng, J., Fostiropoulos, I., BOEHM, B. W., & Soleymani, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.189 · Cited by 69
“…from longer input sequences is absorbed by a shorter hidden state sequence iteratively for every layer instead of only the first layer (Rahman et al., 2020; Tsai et al., 2019; Huang et al., 2020; Pan et al., 2020) or recurrently on segments of the input sequence (Dai et al., 2019; Rae et al., 2020).”
BackgroundMethodResultInfluentialTargeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching
Yu, J., Wang, J., Xia, R., & Li, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/622 · Cited by 67
Emotion Recognition for Multiple Context Awareness
Yang, D., Huang, S., Wang, S., Liu, Y., Zhai, P., Su, L., Li, M., & Zhang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_9 · Cited by 71
“Benefiting from the excellent performance of deep learning technologies in processing diverse signals [26, 71, 9, 37, 64, 35], many researchers [70, 50, 59, 58, 55, 38, 6] emotion recognition by combining diverse modalities ( e.g., face, audio, and language) from the recognized agent.”
BackgroundLearning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking Factor
Ya, S., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3101421 · Cited by 65
Full-Circuit Implementation of Transformer Network Based on Memristor
Yang, C., Wang, X., & Zeng, Z. (2022) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2021.3136355 · Cited by 69
Cheng, H., Yang, Z., Zhang, X., & Yang, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3265653 · Cited by 70
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 75
Li, M., Yang, D., Lei, Y., Wang, S., Wang, S., Su, L., Yang, K., Wang, Y., Sun, M., & Zhang, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28871 · Cited by 75
“Mainstream works (Zadeh et al. 2017, 2018a; Tsai et al. 2019; Hazarika, Zimmermann, and Poria 2020; Han, Chen, and Poria 2021) enhanced the MSA performance by designing complex structures, interaction mechanisms, or fusion paradigms.”
BackgroundMultimodal sentiment analysis based on multi-layer feature fusion and multi-task learning
Cai, Y., Li, X., Zhang, Y., Li, J., Zhu, F., & Rao, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-85859-6 · Cited by 68
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
Guo, Z., Jin, T., & Zhao, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.94 · Cited by 77
Ji, X., Dong, Z., Han, Y., Lai, C. S., & Qi, D. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3275708 · Cited by 66
“Following the previous studies [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], 80% data is distributed in training dataset and the remaining 20% is distributed in testing dataset.”
BackgroundMethodInfluentialSentiment Analysis: Comprehensive Reviews, Recent Advances, and Open Challenges
Lu, Q., Sun, X., Long, Y., Gao, Z., Feng, J. H., & Sun, T. S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294810 · Cited by 77
“There are some studies that attempt to model multimodal alignment [161], [162], [163].”
BackgroundNo Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures
Ahuja, C., Lee, D. W., Ishii, R., & Morency, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.170 · Cited by 65
“, 2017) and cross-attention models (Tsai et al., 2019) to alleviate the need of explicit”
BackgroundMethodInfluentialSelf-citationAttend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
Seo, A., Kang, G.-C., Park, J., & Zhang, B. (2021) · doi:10.18653/v1/2021.acl-long.481 · Cited by 67
Husformer : A Multimodal Transformer for Multimodal Human State Recognition
Wang, R., Jo, W., Zhao, D., Wang, W., Gupte, A., Yang, B., Chen, G., & Min, B. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3357618 · Cited by 77
Deep Learning in Virtual Try-On: A Comprehensive Survey
Islam, T., Miron, A., Liu, X., & Li, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3368612 · Cited by 75
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64
“Previous MSA benchmarks focus more on the study of the multimodal fusion process and commonly utilize a joint representation learning framework to learn a unified multimodal representation [29, 43, 44].”
BackgroundMethodMEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis
Qi, Q., Lin, L., Zhang, R., & Xue, C. (2022) · IEEE Access · doi:10.1109/access.2022.3157712 · Cited by 66
“To solve this problem, following the work of [13], we use sin and cos functions to encode the position information of the sequence of length T , and the frequency is determined by the feature dimension index.”
BackgroundMethodInfluentialZhang, X., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.732 · Cited by 62
“In addition, a cross-modal transformer was integrated that learns attention between two-modal features, thus enabling implicit enhancement of the target modality (Tsai et al., 2019).”
BackgroundMmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis
Paraskevopoulos, G., Georgiou, E., & Potamianos, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746418 · Cited by 62
Pan, Y., Shang, Y., Shao, Z., Liu, T., Guo, G., & Ding, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3296318 · Cited by 58
“To fuse the audio and visual features, the MulT [17] model is utilized as the fusion layer.”
MethodInfluentialMulti-grained fusion network with self-distillation for aspect-based multimodal sentiment analysis
Yang, J., Xiao, Y., & Du, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111724 · Cited by 60
Quantum-inspired Neural Network for Conversational Emotion Recognition
Li, Q., Gkoumas, D., Sordoni, A., Nie, J., & Melucci, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17567 · Cited by 60
“On the one hand, unimodal features are merged into a joint multimodal utterance representation, in which interactions between different modalities are captured (i.e. multimodal fusion) (Liang et al. 2018; Zadeh et al. 2018a; Tsai et al. 2019a; Zadeh et al. 2017; Zhang et al. 2020).”
BackgroundMethodChen, Q., Huang, G., & Wang, Y. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3192728 · Cited by 58
“MulT [30]: The core of the Multimodal Transformer (MulT) is the cross-modal attention mechanism that fuses multimodal representations through directly focusing on interactions among multimodal sequential data.”
BackgroundResultXing, Z., He, Y., & Zhang, W. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3222663 · Cited by 58
“The cross attention (CA) [24] fuses the extracted information from the RFBU-CS and RFBU-CW to further improve the accuracy of the fault diagnosis.”
MethodContrastive-Based Removal of Negative Information in Multimodal Emotion Analysis
Wang, R., Wang, Y.-Y., Cambria, E., Fan, X.-H., Yu, X.-H., Huang, Y., Xiaosong, E., & Zhu, X.-X. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10463-9 · Cited by 70
“Zadeh et al. employ tensor fusion techniques for creating a joint multimodal representation [23], Kim et al. explore feature fusion schemes ranging from bimodal to trimodal configura-tions [26], and Tsai et al. utilize a cross-modal transformer to handle unaligned data [27].”
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58
“MCTN [58] [70] UTA {A, T, V} Train-Val-Test 3-1-1 Sessions 84.”
BackgroundMethodInfluentialErdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
“…self-attention mechanism, some Transformer-based models also utilize a cross-attention mechanism in tasks that involve attending to multimodal data on the input side, such as VQA or multimodal Machine Translation, and vision and language grounding problems (Liu et al., 2019a; Tsai et al., 2019).”
MethodMultimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling
Yan, X., Xue, H., Jiang, S., & Liu, Z. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.2000688 · Cited by 57
“, 2018b), financial and political forecasting Xing, Cambria, and Welsch (2018), Ebrahimi, Yazdavar, and Sheth (2017), community detection Young et al. (2018), dialog systems Majumder et al.”
BackgroundInfluentialLearning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 61
“In particular, a large portion of existing transformer-based models, which inject multimodal information into the popular transformer-based architectures, are receiving more attention for their excellent performance to extract high-level sentiment-related features [1], [28], [29], [30].”
BackgroundMethodInfluentialTEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis
Wang, F., Tian, S.-W., Yu, L., Liu, J., Wang, J., Li, K., & Wang, Y. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10073-9 · Cited by 55
Wei, Y., Yuan, S., Yang, R., Shen, L., Li, Z., Wang, L., & Chen, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.287 · Cited by 62
“In contrast, text modality is more indicative of sentiment owing to higher information density (Tsai et al., 2019; Sun et al., 2021).”
BackgroundSentiment analysis based on text information enhancement and multimodal feature fusion
Liu, Z., Cai, L., Yang, W., & Liu, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110847 · Cited by 59
Zheng, J., Zhang, S., Wang, Z., Wang, X., & Zeng, Z. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3144885 · Cited by 63
“Previous works [6] [10] [15] [61] have shown that the interaction between modalities is essential for emotion recognition.”
BackgroundMethodInfluentialMultimodal Entity Linking with Gated Hierarchical Fusion and Contrastive Training
Wang, P., Wu, J., & Chen, X. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531867 · Cited by 63
“Both of them are built upon the Cross-Modal Transformer (CMT) layer [38].”
BackgroundExcavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57
Ma, X., Zhang, X., Wang, Z., & Pun, M.-O. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3240982 · Cited by 61
“Furthermore, the proposed MBATrans employs the same Q, K , and V matrices [33] for both self-attention and cross-attention computation.”
MethodFu, F., Ai, W., Yang, F., Shou, Y., Meng, T., & Li, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112825 · Cited by 57
“MulT [15] model uses cross-modal transformers to capture long-range dependencies.”
Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion
Liu, X., Xu, Z., & Huang, K. (2023) · Computational Intelligence and Neuroscience · doi:10.1155/2023/9645611 · Cited by 55
“As the MFN, MCTN, RAVEN, and MulT models mainly use a single traditional network (LSTM or CNN) to extract modality features, such networks fail to capture long-distance and local information at the same time in the speech and text modality and can easily miss the key frame information in the video.”
MethodLin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 59
“Hybrid fusion methods [43], [44], [45] perform multimodal fusion at input and output levels to exert the advantages of both early and late fusion methods.”
BackgroundMethodInfluentialTrustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
Xie, Z., Yang, Y. Z., Wang, J., Liu, X., & Li, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376564 · Cited by 65
“6) ICCN: Interaction canonical correlation network (ICCN) [4], which takes text modality as the primary modality and learns text-based audio features and text-based video features.”
BackgroundMultimodal sentiment analysis with unimodal label generation and modality decomposition
Zhu, L., Zhao, H., Zhu, Z., Zhang, C., & Kong, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102787 · Cited by 55
Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing
Zhang, D., Ju, X., Zhang, W., Li, J., Li, S., Zhu, Q., & Zhou, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17686 · Cited by 55
“Second, compared with uni-modal multi-label approaches, multi-modal approaches typically perform better, especially MulT .”
BackgroundInfluentialShi, Q., Fan, J., Wang, Z., & Zhang, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108837 · Cited by 56
Modality-invariant temporal representation learning for multimodal sentiment classification
Sun, H., Liu, J., Chen, Y., & Lin, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.031 · Cited by 57
A survey on semantic processing techniques
Mao, R., He, K., Zhang, X., Chen, G., Ni, J., Yang, Z., & Cambria, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101988 · Cited by 55
TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112220 · Cited by 55
Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation
Du, Z., Li, X., Li, F., Lü, K., Zhu, L., & Li, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02206 · Cited by 57
“To achieve this, we use the cross-attention [47] mechanism to pass information between the two branches, which has shown great success in modeling multimodal interactions [11, 44].”
MethodLarge language models meet text-centric multimodal sentiment analysis: a survey
Yang, H., Zhao, Y., Wu, Y., Wang, S., Zheng, T., Zhang, H., Ma, Z., Che, W., Wang, S., Wei, S., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4593-8 · Cited by 104
“Inspired by related research in the multimodal learning field, in [104], the authors proposed using a cross-modal attention mechanism to learn the alignment relationships between pairs of modalities and developed a transformer-based multimodal sentiment analysis model named MulT.”
Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition
Zhang, S., Chen, M., Chen, J., Li, Y.-F., Wu, Y., Li, M., & Zhu, C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107340 · Cited by 54
Yang, M., Wu, Y., Tao, Y., Hu, X., & Hu, B. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3322271 · Cited by 55
“…fusion analysis, such as Deep Belief Net (DBN) [12], Recurrent Neural Network (RNN) [13], Autoencoder (AE) [14], Convolutional Neural Network (CNN) [15], Transformer [16], etc. Deep learning often requires a large number of samples [17], but our depression assessment research has limited data.”
Dynamic hypergraph convolutional network for multimodal sentiment analysis
Huang, J., Pu, Y., Zhou, D., Cao, J., Gu, J., Zhao, Z., & Xu, D. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126992 · Cited by 57
A Multi-Modal Vertical Federated Learning Framework Based on Homomorphic Encryption
Gong, M., Zhang, Y., Gao, Y., Qin, A. K., Wu, Y., Wang, S., & Zhang, Y. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3340994 · Cited by 54
“MulT [41]: an end-to-end transformer structure to handle both aligned and unaligned multi-modal data.”
BackgroundIs Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?
Rajan, V., Brutti, A., & Cavallaro, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746924 · Cited by 53
“Cross-modal transformers use cross-attention to calculate the relevance of each time-step in a target modality representation using a different sourcemodality [13, 17].”
BackgroundMethodResultInfluential
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
