The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 301–400 of 1,004
Gomaa, A., Huang, Y., Hagag, A., Schmitter, C., Höfler, D., Weißmann, T., Breininger, K., Schmidt, M. A., Stritzelberger, J., Delev, D., et al. (2024) · Neuro-Oncology Advances · doi:10.1093/noajnl/vdae122 · Cited by 35
“42 Cross-attention is a type of attention mechanism used in machine learning models, particularly in natural language processing and computer vision tasks, 42,43 to facilitate interactions between 2 different inputs or sequences.”
MethodMulti-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning
Wang, W., Mao, L., Wang, R., & Min, B. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611322 · Cited by 34
MDKAT: Multimodal Decoupling With Knowledge Aggregation and Transfer for Video Emotion Recognition
Wang, J., Wang, C., Guo, L., Zhao, S., Wang, D., Zhang, S., Zhao, X., Yu, J., Wang, Y., Yang, Y., et al. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3571534 · Cited by 38
“MulT [36]: Multimodal Transformer (MulT) uses a cross-modal attention to fuse unaligned multimodal sequences, capturing long-range dependencies without requiring explicit alignment between modalities.”
InfluentialMulti-Task Learning for Jointly Detecting Depression and Emotion
Zhang, Y., Li, X., Rong, L., & Tiwari, P. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669546 · Cited by 36
“The proposed AMM model achieves the best micro-F1 of 68.5% as compared to micro-F1 of 67.5% of the state-of-the-art system (i.e., Multi-modal Transformer).”
InfluentialMulti-transSP: Multimodal Transformer for Survival Prediction of Nasopharyngeal Carcinoma Patients
Zheng, H., Lin, Z., Zhou, Q., Peng, X., Xiao, J., Zu, C., Jiao, Z., & Wang, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_23 · Cited by 34
EmoInt-Trans: A Multimodal Transformer for Identifying Emotions and Intents in Social Conversations
Singh, G. V., Firdaus, M., Ekbal, A., & Bhattacharyya, P. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3224287 · Cited by 36
A Survey of Quantum-cognitively Inspired Sentiment Analysis Models
Liu, Y., Li, Q., Wang, B., Zhang, Y., & Song, D. (2023) · ACM Computing Surveys · doi:10.1145/3604550 · Cited by 42
“, the encoder-decoder architecture) have been made to “translate” the representation of a single sentence under one modality to the representation of another modality and take the hidden unit as the joint representation of the sentence [94, 109].”
BackgroundChumachenko, K., Iosifidis, A., & Gabbouj, M. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00470 · Cited by 34
“Specifically, we compare with: • Addition in original space, i.e., without compression and expansion (ADD); • Multimodal Transformer following [45], where we introduce 2 multi-head self-attention blocks, audio-to-vision, and vision-to-audio, outputs of which are added to corresponding branches…”
Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 36
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
Dai, W., Liu, Z., Yu, T., & Fung, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.30 · Cited by 33
“, 2015) has been widely applied to do modality fusion and emphasis (Zadeh et al., 2018a; Pham et al., 2018; Tsai et al., 2019a).”
BackgroundMethodInfluentialInteractive Multimodal Attention Network for Emotion Recognition in Conversation
Ren, M., Huang, X., Shi, X., & Nie, W. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3078698 · Cited by 33
“Here, inspired by [27], we employ cross-modal attention to capture the cross-modal interactions between two different modalities as shown in Fig.”
BackgroundMethodQuantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis
Gkoumas, D., Li, Q., Dehdashti, S., Melucci, M., Yu, Y., & Song, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16165 · Cited by 34
“• For SOTA, we replicated a) MulT (Tsai et al. 2019), consisting of pairwise crossmodal transformers, the outputs of which are concatenated to build the multimodal embedding utterance, b) RAVEN (Wang et al. 2019b), an RNN based model with an attention gating mechanism to model crossmodal interactions, and c) TFN (Zade…”
BackgroundMethodInfluentialMultimodal Sentiment Analysis based on Recurrent Neural Network and Multimodal Attention
Cai, C., He, Y. L., Sun, L., Lian, Z., Liu, B., Tao, J., Xu, M., & Wang, K. (2021) · doi:10.1145/3475957.3484454 · Cited by 33
“Tsai et al. [32] extended the transformer with cross-modal attention module to learn representations from multimodal features.”
BackgroundAn autoencoder-based feature level fusion for speech emotion recognition
Peng, S., Kai, C., Tian, T., & Jingying, C. (2022) · Digital Communications and Networks · doi:10.1016/j.dcan.2022.10.018 · Cited by 38
i-Code: An Integrative and Composable Multimodal Learning Framework
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Yi‐chong, X., Qian, Y., Gao, M., Chen, Y., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26290 · Cited by 45
Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542927 · Cited by 63
“The baselines for comparison include Early Fusion LSTM (EF-LSTM), Late Fusion LSTM (LFLSTM), TFN [64], MFN [65], MFRM [32], MulT [49], and MTAG [57].”
BackgroundMethodInfluentialWang, X., Lyu, J., Kim, B., Parameshachari, B. D., Li, K., & Li, Q. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3419140 · Cited by 38
“Tsai et al. [34] introduced a transformer-based network for multimodal feature extraction.”
Si, X., Zhang, S., Yang, Z., Yu, J., & Ming, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126081 · Cited by 41
Cross-Modality Relevance for Reasoning on Language and Vision
Chen, Z., Guo, Q., & Kordjamshidi, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.683 · Cited by 31
“Researchers also extended Transformers with both textual and visual modalities (Li et al., 2019b; Sun et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Tsai et al., 2019).”
BackgroundTemporal Multimodal Graph Transformer With Global-Local Alignment for Video-Text Retrieval
Feng, Z., Zeng, Z., Guo, C., & Li, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3207910 · Cited by 33
“Tsai et al. [31] solve the problem of multi-modal language analysis with the multi-modal transformer, indicating to fully understand the human language with not only words but also facial gestures and acoustic behaviors.”
BackgroundFV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference
Wei, Q., Huang, X., & Zhang, Y. (2022) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2022.3215245 · Cited by 32
“After being processed by the first Transformer layer of the audio module, the 16 attention maps shown in “Layer 1” are obtained.”
BackgroundMethodInfluentialTowards Robust Multimodal Sentiment Analysis Under Uncertain Signal Missing
Li, M., Yang, D., & Zhang, L. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3324552 · Cited by 33
“Most works [15], [16], [17], [18], [19] are based on a strong assumption that all modalities are complete during the training and testing phases.”
BackgroundLei, Y., Yang, D., Li, M., Wang, S., Chen, J., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-981-99-9119-8_18 · Cited by 35
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from TextPreprint
Hill, F., Mokrá, S., Wong, N., & Harley, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09382 · Cited by 83
“BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g. Tsai et al. [2019], Lu et al. [2019], which in our application provides an explicit pathway for the agent to bind visual experience to specific (contextual) word representations.”
BackgroundEnhancing Chinese Character Representation With Lattice-Aligned Attention
Zhao, S., Hu, M., Cai, Z., Zhang, Z., Zhou, T., & Liu, F. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3114378 · Cited by 32
“3(a)] aims to capture fine-grained correlations between character and word feature representations, which is a variant of recently proposed m -head cross-modal attention mechanism [33], by treating X as queries and Y as keys and values.”
BackgroundRBA-GCN: Relational Bilevel Aggregation Graph Convolutional Network for Emotion Recognition
Yuan, L., Huang, G., Li, F., Yuan, X., Pun, C., & Zhong, G. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3284509 · Cited by 31
Multimodal Sentiment Analysis with Preferential Fusion and Distance-aware Contrastive Learning
Ma, F., Zhang, Y., & Sun, X. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00237 · Cited by 31
“[8] proposed a directional pairwise cross-modal attention method that attends to the interactions between multimodal sequences.”
BackgroundMethodResultInfluentialLi, M., Zhu, Z., Li, K., & Pei, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3430045 · Cited by 33
“…trend has evolved from simple feature-weighted concatenation fusion methods [10], [11] to exploring complex fusion strategies involving tensors [5], [12], attention mechanisms [13], [14], Transformers [15], [16], [17], and other technologies [18], [19], [20] to capture inter-modal interactions.”
A vision and language hierarchical alignment for multimodal aspect-based sentiment analysis
Zou, W., Sun, X., Lu, Q., Wang, X., & Feng, J. H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111369 · Cited by 32
Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors
Wu, Y., Zhao, Y., Yang, H., Chen, S., Qin, B., Cao, X., & Zhao, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.109 · Cited by 33
Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion
Yao, Y., & Mihalcea, R. F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.143 · Cited by 36
“On the other hand, the multimodal interaction methods enable the encoders to interact with each other via cross-modal attention mechanisms (Yu et al., 2017a; Tan and Bansal, 2019; Tsai et al., 2019).”
BackgroundMethodDeng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211197 · Cited by 32
“Tsai et al. [8] conducted cross-modal interactions by trans-former encoders, which could not decompose multimodal information into inter-modality and intra-modality interactions.”
BackgroundMethodInfluentialMultimodal Sentiment Recognition With Multi-Task Learning
Zhang, S., Yin, C., & Yin, Z. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3224929 · Cited by 31
“MulT [27] is based on multi-head transformer model that”
BackgroundMethodCOM: Contrastive Masked-attention model for incomplete multimodal learning
Qian, S., & Wang, C. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.003 · Cited by 33
Multimodal Named Entity Recognition with Bottleneck Fusion and Contrastive Learning
Wang, P., Xiaohang, C. H. E. N., Shang, Z., & Ke, W. (2023) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2022edp7116 · Cited by 33
Elastic Graph Transformer Networks for EEG-Based Emotion Recognition
Jiang, W.-B., Yan, X., Zheng, W., & Lu, B. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096511 · Cited by 32
“Cross-modal Transformer Unit Inspired by the work in [17], we propose the Cross-modal Transformer Unit to fuse modalities so that the model is able to make use of the complementary properties of different modalities.”
MethodZhang, X., Cui, W., Hu, B., & Li, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3354382 · Cited by 30
Cross-modal contrastive learning for multimodal sentiment recognition
Yang, S., Cui, L., Wang, L., & Wang, T. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05355-8 · Cited by 31
“Currently, state-of-the-art approaches focus on designing fusion models capable of effectively integrating information from diverse modalities [3, 4].”
MLG-NCS: Multimodal Local–Global Neuromorphic Computing System for Affective Video Content Analysis
Ji, X., Dong, Z., Zhou, G., Lai, C. S., & Qi, D. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3392732 · Cited by 30
“Recently, with the rapid development of AI, various deep learning algorithms have achieved tremendous successes in affective video content analysis, which can recognize emotion accurately and automatically [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20].”
BackgroundNoise-Resistant Multimodal Transformer for Emotion Recognition
Liu, Y., Zhang, H., Zhan, Y., Chen, Z., Yin, G., Wei, L., & Chen, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02304-3 · Cited by 35
“For the MOSI and MOSEI datasets, referring to the existing methods (Tsai et al., 2019; Haz-arika et al., 2020; Mao et al., 2022), we used the features provided in the dataset, which had been extracted by the OpenFace (Baltrusaitis et al., 2016).”
InfluentialMultimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
Zhu, W., Wang, X. E., Fu, T.-J., Yan, A., Narayana, P., Sone, K., Basu, S., & Wang, W. Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.103 · Cited by 35
“The models usually adopt the Transformer structure to encode the visual features and the textual features (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2020; Sun et al., 2019; Li et al., 2019; Huang et al., 2020b; Luo et al., 2020; Li et al., 2020; Zheng et al., 2020; Wei et al., 2020; Tsai et al., 2019).”
MethodStroke constrained attention network for online handwritten mathematical expression recognition
Wang, J., Du, J., Zhang, J., Wang, B., & Ren, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108047 · Cited by 32
“Although [46,47] utilized cross-modal self-attention to achieve the encoder fusion, the unaligned issue was still existed as the alignments acquired by cross-modal self-attention could not be guaranteed to be exactly accurate.”
BackgroundA Large-Scale Chinese Multimodal NER Dataset with Speech Clues
Sui, D., Tian, Z., Chen, Y., Liu, K., & Zhao, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.218 · Cited by 32
Improving Multimodal fusion via Mutual Dependency Maximisation
Colombo, P., Chapuis, É., Labeau, M., & Clavel, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.21 · Cited by 36
“Current state-of-the-art on these datasets uses architectures based on pre-trained transformers (Tsai et al., 2019; Siriwardhana et al., 2020) such as MultiModal Bert (MAGBERT) or Mular X iv :2 10 9.”
MethodMAG+: An Extended Multimodal Adaptation Gate for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Li, W., Gao, L., & Tang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746536 · Cited by 30
PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3192469 · Cited by 33
“For descriptions of the remaining baselines, we recommend readers refer to GMFN [49], RLC [5], DialogueTRM [75], MulT [76], ICCN [52], M3ER [77], QMF [78], and DialogXL [79].”
MethodHwang, Y., & Kim, J.-H. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.2 · Cited by 31
“3 The authors of SPT (Cheng et al., 2021) claim that the number of parameters is reduced to 10% of (Tsai et al., 2019) which utilizes the vanilla Transformer encoder.”
BackgroundMethodResultInfluentialWang, K., Shan, S., Dou, W., Wei, H., & Zhang, K. (2024) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2024.3494266 · Cited by 31
“Common MML fusion methods were selected to evaluate the noise robustness of DCCA-LF, including LF, Tensor Fusion (TF) [38], Low-rank Tensor Fusion (LRTF) [39], Gradient-Blending (GB) [40], Matrix Interaction (MI) [41], and Multi-modal Transformer (MulT) [42].”
SSLMM: Semi-Supervised Learning with Missing Modalities for Multimodal Sentiment Analysis
Wang, Y., Jian, H., Zhuang, J., Guo, H., & Leng, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103058 · Cited by 34
Xie, W., Fang, Y., Yang, G., Yu, K., & Li, W. (2023) · Biomolecules · doi:10.3390/biom13091391 · Cited by 29
“Detailed explanations of these two modules can be found in the literature, [17] and [25], and we have integrated these two methods for multi-modal data fusion.”
MethodCoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities
Kontras, K., Chatzichristos, C., Phan, H., Suykens, J. A. K., & Vos, M. D. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3354388 · Cited by 32
He, J.-J., Shi, X.-H., Li, X.-F., & Toda, T. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp48485.2024.10446548 · Cited by 40
“CME Block is structured akin to a standard transformer layer, featuring an h -head cross-attention module [28], residual connections, and FC layers.”
MethodText-centered cross-sample fusion network for multimodal sentiment analysis
Huang, Q., Chen, J., Huang, C., Huang, X., & Wang, Y. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01421-w · Cited by 29
“The CMU-MOSEI dataset contains more complex and diverse data, than most traditional methods, such as those based on tensor fusion [7, 8, 46], traditional attentional mechanisms [9, 24, 48], or simple auxiliary tasks [11, 11, 12] do not perform well on CMU-MOSEI.”
InfluentialWang, R., Yang, Q., Tian, S., Yu, L., He, X., & Wang, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129163 · Cited by 30
Decoupled cross-attribute correlation network for multimodal sentiment analysis
Zhao, X., Li, X., Jiang, R., & Tang, B. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102897 · Cited by 31
Gu, X., Zhao, S., Zhao, S., et al. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110138 · Cited by 28
Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition
Delbrouck, J.-B., Tits, N., & Dupont, S. (2020) · doi:10.18653/v1/2020.nlpbt-1.1 · Cited by 30
“Sad Neu. avg MAT (ours) 0.68 0.71 0.75 0.80 0.73 MNT (ours) 0.66 0.71 0.72 0.80 0.72 NT (ours) 0.67 0.69 0.69 0.78 0.70 P (L+A, ours) 0.63 0.65 0.68 0.78 0.67 MulT (2019) 0.60 0.70 0.74 0.82 0.71 MCTN (2018) 0.49 0.66 0.72 0.78 0.66 RAVEN (2019) 0.60 0.64 0.66 0.77 0.66 Table 2: IEMOCAP: F1-scores per emotion class.”
BackgroundInfluentialMRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering
You, C., Chen, N., & Zou, Y. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/549 · Cited by 32
TFF-Former: Temporal-Frequency Fusion Transformer for Zero-training Decoding of Two BCI Tasks
Li, X., Wei, W., Qiu, S., et al. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548269 · Cited by 28
“Meanwhile, Multimodal Transformer [30] can achieve interactions between multimodal sequences across latently adaptive streams from one modality to another, which can be used to achieve the feature fusion between temporal information and frequency information.”
BackgroundQuan, Z., Chen, Q., Zhang, M., Hu, W., Zhao, Q., Hou, J., Li, Y., & Liu, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3255832 · Cited by 28
“Cross-view attention [60] can Fig.”
MethodMai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26
MBCFNet: A Multimodal Brain–Computer Fusion Network for human intention recognition
Li, Z., Zhang, G., Okada, S., Wang, L., Zhao, B., & Dang, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111826 · Cited by 29
UEFN: Efficient uncertainty estimation fusion network for reliable multimodal sentiment analysis
Wang, S., Ratnavelu, K., & Shibghatullah, A. S. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06113-6 · Cited by 30
“2) The multimodal transformer [50] employs an attention mechanism and transformer to enhance token representations by incorporating nonverbal signals.”
He, X., Fang, Y., Xiang, N., Li, Z., Wang, Q., Yang, C., Wang, H. H., & Pan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129532 · Cited by 31
“MulT [24] MulT employs directional pairwise cross-modal attentions to facilitate interactions between modalities.”
Depression detection methods based on multimodal fusion of voice and text
Xu, Z., Gao, Y., Wang, F., Zhang, L., Zhang, L., Wang, J., & Shu, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-03524-4 · Cited by 30
Graph-based Multimodal Ranking Models for Multimodal Summarization
Zhu, J., Xiang, L., Zhou, Y., Zhang, J., & Zong, C. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3445794 · Cited by 27
“Precision and recall are commonly used to evaluate recommendation systems [16] and information retrieval task [26, 39, 52].”
BackgroundRadar Ghost Target Detection via Multimodal Transformers
Wang, L., Giebenhain, S., Anklam, C., & Goldlüecke, B. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3100176 · Cited by 27
Multi-modal Sarcasm Detection Based on Contrastive Attention Mechanism
Zhang, X., Chen, Y., & Li, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_66 · Cited by 30
“[16] used Transformers [17] to model each modality and used its multi-head attention mechanism to capture multi-modal interaction information.”
MethodZeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31
“6) Interpretable Modality Fusion (IMR) (Tsai et al., 2020), which improves the interpretable ability of MULT by introducing the multimodal routing mechanism.”
BackgroundWang, K., Song, Y., Sheng, H., Xu, J., Zhang, S., & Qin, J. (2022) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2022.05.010 · Cited by 31
A Vegetable Leaf Disease Identification Model Based on Image-Text Cross-Modal Feature Fusion
Feng, X., Zhao, C., Wang, C., et al. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.918940 · Cited by 27
“Tsai et al. (2019) proposed to use Multimodal Transformer for multimodal feature fusion.”
MethodCrossmodal Transformer Based Generative Framework for Pedestrian Trajectory Prediction
Su, Z., Huang, G., Zhang, S., et al. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812226 · Cited by 27
USAF: Multimodal Chinese named entity recognition using synthesized acoustic features
Liu, Y., Huang, S., Li, R., Yan, N., & Du, Z. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103290 · Cited by 27
Attention-Based Cross-Modal CNN Using Non-Disassembled Files for Malware Classification
Kim, J., Paik, J., & Cho, E.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3253770 · Cited by 27
“Recently, cross-modal learning [29], [31], [32], a new multi-modal learning method that focuses on the adaptive combination of multiple modalities, has emerged.”
BackgroundTowards Multimodal Sentiment Analysis Debiasing via Bias Purification
Yang, D., Li, M., Xiao, D., Liu, Y., Yang, K., Chen, Z., Wang, Y., Zhai, P., Li, K. J., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73636-0_27 · Cited by 30
Contextualized Attention-Based Knowledge Transfer for Spoken Conversational Question Answering
You, C., Chen, N., & Zou, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-110 · Cited by 40
“A 2 : a bottle R 2 : a bottle translate speech content into text forms, and then apply the state-of-the-art Text-QA models on the transcribed text documents [8, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 19, 22].”
MethodMissing-modality enabled multi-modal fusion architecture for medical data
Wang, M., Fan, S., Li, Y., Xie, Z., & Chen, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104796 · Cited by 31
A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism
Qiu, K., Zhang, Y., Zhao, J., Zhang, S., Wang, Q., & Chen, F. (2024) · Electronics · doi:10.3390/electronics13101922 · Cited by 26
Li, Q., Yao, C., Yao, X., Zhou, Z., & Ren, K. (2024) · Remote Sensing · doi:10.3390/rs16152688 · Cited by 28
“Compared with other existing methods, the model performs better in predicting the deformation trend of the reservoir bank slope.”
MethodText-Guided Reconstruction Network for Sentiment Analysis With Uncertain Missing Modalities
Shi, P., Hu, M. Q., Nakagawa, S., Zheng, X., Shi, X., & Ren, F. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3541743 · Cited by 29
“However, numerous studies have shown that the text modality plays a central and significant role [2], [3], [10], [11], [12], [13], [14].”
InfluentialTime-Continuous Audiovisual Fusion with Recurrence vs Attention for In-The-Wild Affect Recognition
Karas, V., Tellamekala, M. K., Mallol-Ragolta, A., Valstar, M., & Schuller, B. W. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00266 · Cited by 27
Lin, R., & Hu, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.36 · Cited by 34
“Modality translation is a popular method to explicitly translate source modality to the target one, which directly manipulates the commonalities across modalities (Tsai et al., 2019a; Wu et al., 2021; Zhao et al., 2021).”
BackgroundMethodInfluentialMOCAS: A Multimodal Dataset for Objective Cognitive Workload Assessment on Simultaneous Tasks
Jo, W., Wang, R., Cha, G.-E., Su, S., Senthilkumaran, R. K., Foti, D., & Min, B. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3414330 · Cited by 27
Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling FusionPreprint
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13572 · Cited by 26
“Specifically, our model surpasses the state-of-the-art unaligned cross-modal fusion method MulT [36] in terms of all metrics, except 7-class accuracy on CMU-MOSI, which demonstrates the effectiveness of graph convolution and graph pooling in learning sequential data, compared to Transformer which is employed in MulT.”
MethodInfluentialQuan, Z., Sun, T., Su, M., & Wei, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4767437 · Cited by 25
Dai, Y., Li, J., Li, Y., & Lu, G. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111954 · Cited by 27
Modeling inter-modal incongruous sentiment expressions for multi-modal sarcasm detection
Ou, L., & Li, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128874 · Cited by 26
Li, Z.-H., Liu, P.-B., Pan, Y.-S., Ding, W.-P., Yu, J., Chen, H.-R., Liu, W.-H., Luo, Y.-M., & Wang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128940 · Cited by 25
Large Language Models in Psychiatry: Current Applications, Limitations, and Future Scope
Liu, Z., Bao, Y., Zeng, S., Qian, R., Deng, M., Gu, A., Li, J., Wang, W., Cai, W., Li, W., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020046 · Cited by 25
Visual Entity Linking via Multi-modal Learning
Zheng, Q., Wen, H., Wang, M., & Qi, G. (2021) · Data Intelligence · doi:10.1162/dint_a_00114 · Cited by 28
“[24, 25, 26, 27, 28] proposed multi-modal entity-aware models to achieve image caption generation, which is also different from our visual entity linking task.”
BackgroundMultimodal Music Emotion Recognition with Hierarchical Cross-Modal Attention Network
Zhao, J., Ru, G., Yu, Y., Wu, Y., Li, D., & Li, W. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859812 · Cited by 27
“The Cross-Modal Attention (CMA) module [19] is used for fusing the features from two modalities.”
MethodVocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices
Kadandale, V. S., Montesinos, J. F., & Haro, G. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10861 · Cited by 25
CMJRT: Cross-Modal Joint Representation Transformer for Multimodal Sentiment Analysis
Xu, M. Y., Liang, F., Su, X., & Fang, C. (2022) · IEEE Access · doi:10.1109/access.2022.3219200 · Cited by 25
“As for heterogeneous modalities, Tsai et al. [9] design Multimodal Transformer (MulT), which attends to information adaption between bimodality with the crossmodal attention mechanism.”
BackgroundMethodInfluentialIlias, L., & Askounis, D. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110834 · Cited by 26
QAP: A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition
Li, Z., Zhou, Y., Liu, Y., Zhu, F., Yang, C., & Hu, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.772 · Cited by 26
“MulT (Tsai et al., 2019): For modalities unaligned, MulT uses cross-modal attention to integrate modalities in pairs and does not pay attention to modal priority as above baselines.”
BackgroundMethodMuti-Modal Emotion Recognition via Hierarchical Knowledge Distillation
Sun, T., Wei, Y., Ni, J., Liu, Z., Song, X., Wang, Y., & Nie, L. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3385180 · Cited by 26
“In particular, the prior studie [3], [13], [17] predicted emotions in a seven-class or binary-class task, corre-spondingtoAcc-7andAcc-2scores.”
BackgroundMethodInfluentialMultimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 34
“For multimodal emotion recognition, following the state-of-the-art methods in IEMOCAP [16], [19], we utilize GloVe [76] to extract the features of the text modality.”
BackgroundMethodInfluentialDu, J., Jin, J., Zhuang, J., & Zhang, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-54872-6 · Cited by 25
“Multi-modal sentiment analysis has attracted extensive attention in the multimedia community in recent years 20,21 , because of the vivid and interesting information in multi-modal data, In the following, we mainly present the related works on the traditional MSA model without cross-instance…”
BackgroundMultimodal PEAR Chain-of-Thought Reasoning for Multimodal Sentiment Analysis
Li, Y., Lan, X., Chen, H., Lü, K., & Jiang, D. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3672398 · Cited by 29
“To evaluate the performance of our models, we adopt the following metrics based on previous works: binary accuracy (Acc 2 ), binary F1 score (F1), 7-class accuracy (Acc 7 ), Mean Absolute Error (MAE) , and the correlation between model predictions and human annotations (Corr).”
Sentiment analysis of social media comments based on multimodal attention fusion network
Liu, Z., Yang, T., Chen, W., Chen, J., Li, Q., & Zhang, J. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112011 · Cited by 25
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
