The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 501–600 of 1,004
ArunaGladys, A., & Vetriselvi, V. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111553 · Cited by 36
Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers
Rizzoli, G., Shenaj, D., & Zanuttigh, P. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00070 · Cited by 20
Yang, X., Xu, Z., Liu, H., Shull, P. B., Redmond, S. J., Liu, G., & Wang, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3400022 · Cited by 19
“Inspired by the development of Transformer [13], Tsai et al. [14] proposed a cross-modal attention mechanism for analyzing human multimodal language.”
MethodJoint training strategy of unimodal and multimodal for multimodal sentiment analysis
Meng, L., Zhu, Z., Li, K., Zhou, L., Zhao, Z., & Pei, H. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105172 · Cited by 18
Simple and Effective Multimodal Learning Based on Pre-Trained Transformer Models
Miyazawa, K., Kyuragi, Y., & Nagai, T. (2022) · IEEE Access · doi:10.1109/access.2022.3159346 · Cited by 19
“MulT Multimodal transformer (MulT) [34] is a model designed for unaligned multimodal language sequences, wherein a directional pairwise cross-modal transformers are adopted to merge the multimodal information.”
BackgroundMethodResultEnd-to-End Prediction of Knee Osteoarthritis Progression With Multimodal Transformers
Panfilov, E., Saarakkala, S. S., Nieminen, M. T., & Tiulpin, A. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3536170 · Cited by 17
“Lately, thanks to advances in DL-based methods, unified attention-based methods, namely Transformers, were developed [41], [42].”
SDRS: Sentiment-Aware Disentangled Representation Shifting for Multimodal Sentiment Analysis
Zhao, S., Yang, Z., Shi, H., Feng, X., Meng, L., Qin, B., Yan, C. C., Tao, J., & Ding, G. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3539225 · Cited by 18
“With the success of transformer [34], a number of attention-level multimodal feature fusion solutions have been proposed to alleviate these problems [35], [36], [37], [38], [39], [40].”
Self-Supervised Bi-Channel Transformer Networks for Computer-Aided Diagnosis
Gong, R., Han, X., Wang, J., Ying, S., & Shi, J. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3153902 · Cited by 18
“Someworksaimto provide a latent adaptation between unaligned multi-view features by a transformer [23], [24], indicating the effectiveness of multi-view transformer for information interaction.”
BackgroundInstance-sequence reasoning for video question answering
Liu, R., & Han, Y. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-1248-1 · Cited by 17
Multimodal false information detection method based on Text-CNN and SE module
Liang, Y., Tohti, T., & Hamdulla, A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277463 · Cited by 17
“(2) Early Concatenation [30–33], this model concatenates text features and image features and then inputs them into the Transformer layer to fuse the features of different modalities.”
MethodVideoPro: A Visual Analytics Approach for Interactive Video Programming
He, J., Wang, X., Kam-Kwai, W., Huang, X., Chen, C., Chen, Z., Wang, F., Zhu, M., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326586 · Cited by 17
“This classification scheme is established according to the experts’ requirements and previous work practices [3, 62].”
MethodModality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition
Chen, C., & Zhang, P. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3640343 · Cited by 18
“Inspired by the power of Transformer [60], researchers have looked into ways to implicitly align multimodal sequences during model training [56][64][38][25].”
BackgroundInfluentialMeta Noise Adaption Framework for Multimodal Sentiment Analysis With Feature Noise
Yuan, Z., Zhang, B., Xu, H., & Gao, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3362600 · Cited by 19
“In particular, typical methods including the Multimodal Transformer (MulT) [53], the Modality-Invariant and -Specific Representations (MISA) [54], and the Self-supervised Multi-task Multimodal sentiment analysis network (Self-MM) [39] are first selected.”
MethodYin, B., Chen, Z., & Tao, M. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3364673 · Cited by 17
A Three-stage multimodal emotion recognition network based on text low-rank fusion
Zhao, L., Yang, Y., & Ning, T. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01345-5 · Cited by 21
“Multimodal Transformer (Mult) [35] uses the Transformer to complete the cross-modal interaction between the two modalities to achieve the mutual complementation and reinforcement of the modalities.”
A prompt construction method for the reverse dictionary task of large-scale language models
Tian, S., Huang, S., Li, R., & Wei, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108596 · Cited by 19
Exploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
Qin, J., Zong, L., & Liu, F. (2024) · Applied Sciences · doi:10.3390/app14177720 · Cited by 19
Wang, P., Zhang, X., Chen, Y., & Cao, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104013 · Cited by 21
Multimodal sentiment analysis method based on image-text quantum transformer
Wu, H., Kong, D., Wang, L., Li, D., Zhang, J., & Han, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130107 · Cited by 17
MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., & Tao, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653457 · Cited by 71
“According to the overall results, MulT [63] is the best sequence-level algorithm.”
Cross-modal context-gated convolution for multi-modal sentiment analysis
Wen, H., You, S., & Fu, Y. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.025 · Cited by 19
In Your Eyes: Modality Disentangling for Personality Analysis in Short Video
Sun, X., Liu, B., Ai, L., Liu, D., Meng, Q., & Cao, J. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3161708 · Cited by 17
“Recently, video-based research studies [26] have gradually attracted researchers’ attention because video streams usually contain multimodal data [27] and are potentially improve the personality detection performance.”
BackgroundAn Ensemble-Learning-Based Technique for Bimodal Sentiment Analysis
Shah, S., Ghomeshi, H., Vakaj, E., Cooper, E., & Mohammad, R. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7020085 · Cited by 21
Interpretable Multimodal Capsule Fusion
Wu, J., Mai, S., & Hu, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3178236 · Cited by 16
MMT: Multi-way Multi-modal Transformer for Multimodal Learning
Tang, J., Li, K., Hou, M., Jin, X., Kong, W., Ding, Y., & Zhao, Q. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/480 · Cited by 16
Sense-aware BERT and Multi-task Fine-tuning for Multimodal Sentiment Analysis
Fang, L., Liu, G., & Zhang, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892116 · Cited by 16
“Tsai et al. [13] constructed Multimodal Transformer (MulT) that focuses on interactions between multimodal sequences spanning different time steps.”
BackgroundMethodInfluential3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos
Gupta, V., Mittal, T., Mathur, P., Mishra, V., Maheshwari, M., Bera, A., Mukherjee, D., & Manocha, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02039 · Cited by 16
“[68] is a multimodal transformerbased model that uses both visual and audio modalities and has shown high performance on other emotion recognition datasets.”
BackgroundBennett, C. C., Bae, Y., Yoon, J. H., Chae, Y., Yoon, E., Lee, S., Ryu, U., Kim, S., & Weiss, B. (2023) · Computer Speech and Language · doi:10.1016/j.csl.2023.101521 · Cited by 15
“There are also existing research challenges in regards to effectively integrating multi-modal audio and visual data streams in real time, which remain to be addressed (Tsai et al., 2019).”
BackgroundDSAMR: Dual-Stream Attention Multi-hop Reasoning for knowledge-based visual question answering
Sun, Y., Zhu, Z., Zuo, Z., Li, K., Gong, S., & Qi, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123092 · Cited by 17
Cross-Modal Dynamic Transfer Learning for Multimodal Emotion Recognition
Hong, S., Kang, H., & Cho, H. (2024) · IEEE Access · doi:10.1109/access.2024.3356185 · Cited by 20
“For example, MulT [19] uses cross-attention, which learns how to attend to one modality based on another modality, or MAG [20] learns how to capture the characteristics of each modality and fuse them with a gating mechanism.”
BackgroundKnowledge enhancement for speech emotion recognition via multi-level acoustic feature
Zhao, H., Huang, N., & Chen, H. (2024) · Connection Science · doi:10.1080/09540091.2024.2312103 · Cited by 16
“Inspired by Tsai et al. (2019), the crossmodal transformer allows for potential adaptation between two modalities through its cross-fusion module, which forms its core.”
InfluentialAttA-NET: Attention Aggregation Network for Audio-Visual Emotion Recognition
Fan, R., Liu, H., Li, Y., Guo, P., Wang, G., & Wang, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447640 · Cited by 16
“MulT [20] 78.50% - - MMTM [21] 73.12% 83.58% 31.97M MSAF [22] 74.86 % 83.12% 25.94M TAV [12] 77.75% - - TVA [12] 76.00% - - ERANNs [4] 74.80% 86.94% 23.60M John et al. [13] 51.01% - - Franceschini et al. [23] 78.54% - - Chumachenko et al. [1] 81.58% - - Guo et al. [15] 78.45% 90.53% 18.40M Human…”
InfluentialAn Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning
Gong, L., He, X., & Jianzhong, Y. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2371712 · Cited by 17
Multimodal Detection of Bots on X (Twitter) Using Transformers
Ilias, L., Kazelidis, I. M., & Askounis, D. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3435138 · Cited by 17
“Motivated by [57], [58], and [59], we exploit two cross-modal attention layers, i.e., one from textual f t to visual features f v and one from visual to textual features.”
MethodLai, S., Li, J., Guo, G., Hu, X., Li, Y., Tan, Y., Song, Z., Liu, Y., Ren, Z., Wang, C., et al. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651442 · Cited by 16
PROSE: Predicting Multiple Operators and Symbolic Expressions using multimodal transformers
Liu, Y., Zhang, Z., & Schaeffer, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106707 · Cited by 34
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17
“Multimodal Transformer [43] proposes cross-attention mechanisms, for cross-modal learning on all pairs of video-audio-text data, which Pellegrain et al. [31] extends to longer sequences.”
BackgroundLi, M., Chen, Y., Lu, Z., Ding, F., & Hu, B. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3551439 · Cited by 16
“Generally, multimodal fusion methods generally combine features extracted from multiple channels through a fully connected layer [50], [51].”
Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems
Lin, T.-E., Wu, Y., Huang, F., Si, L., Sun, J., & Li, Y. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539209 · Cited by 17
“The key to multimodal modeling includes multimodal fusion [53, 56], consistency and difference [16, 55], modality alignment [47].”
BackgroundChaib, S., Mansouri, D. E. K., Omara, I., Hagag, A., Dhelim, S., & Bensaber, D. A. (2022) · Remote Sensing · doi:10.3390/rs14225817 · Cited by 16
C4AV: Learning Cross-Modal Representations from Transformers
Luo, S., Dai, H., Shao, L., & Ding, Y. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_3 · Cited by 16
River water quality forecasting: a novel LSTM-Transformer approach enhanced by multi-source data
Huan, J., Zhang, C., Xu, X., Qian, Y., Zhang, H., Fan, Y., Hu, Q., Mao, Y., & Zhao, X. (2025) · Environmental Monitoring and Assessment · doi:10.1007/s10661-025-14494-5 · Cited by 15
“This architecture demonstrates significant advantages in long-range time-series prediction tasks through global dependency modeling (Tsai et al., 2019; Devlin et al., 2019).”
Xu, X., Chen, J., Fu, C., & Lv, Z. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3605133 · Cited by 15
“Tsai et al. [44] developed directional pairwise cross-modal attention to address the challenges of data non-alignment and long-range dependencies in multi-modal human language time-series data.”
Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification
Zhang, Y., Tiwari, P., Rong, L., Chen, R., Alnajem, N. A., & Hossain, M. S. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3527175 · Cited by 15
“In line with Multi-modal Transformer [41], we will use a similar manner that could constantly force a target modality with the low-level features from the source modality via the bidirectional mapping, and is applied to unaligned multi-modal language.”
MethodFen, L., Chen, J., Li, K., Tan, W., Cai, C., & Ayub, M. S. (2022) · Entropy · doi:10.3390/e24121836 · Cited by 16
“To tackle this problem, the low-rank multi-modal fusion (LMF) [26], the multi-modal, factorized, bilinear pooling (MFB) model [27], the memory fusion network (MFN) [28], and the multi-modal transformer (MuLT) [14] have been proposed to further improve the processing efficiency and evaluation.”
BackgroundMethodInfluentialPace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion
Guo, X., Kot, A. C., & Kong, A. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3252270 · Cited by 17
“Recently, research works [21], [25], [27], [29], [37], [38] proposed more promising fusion mechanisms using sophisticated strategies.”
BackgroundMethodInfluentialTMMDA: A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., Zang, X., Xiang, Y., & Tang, B. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583406 · Cited by 15
“[27] proposed Multimodal Transformer which extends the standard Transformer network to learn representations directly from unaligned multimodal streams.”
BackgroundMethodResultInfluentialBeyond superficial emotion recognition: Modality-adaptive emotion recognition system
Kang, D., Kim, D. H., Kang, D., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121097 · Cited by 15
Contrastive topic-enhanced network for video captioning
YaWen, Z., Wang, Y., Liao, D., Li, G., Xu, J. H., Man, H., Liu, B., & Xu, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121601 · Cited by 18
Learning to Ground Instructional Articles in Videos through Narrations
Mavroudi, E., Afouras, T., & Torresani, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01395 · Cited by 15
“As shown in Figure 2, our proposed VINA model follows the popular paradigm of leveraging Transformers for modeling multimodal interactions [56, 62].”
MethodHardan, S., Shaaban, M. A., Abdalla, J., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-66812-5 · Cited by 15
“We also adopted three other fusion mechanisms: tensor fusion 17,18 , attention fusion 19 , and Multimodal InfoMax (MMIM) 20 .”
BackgroundMethodPrivacy-Preserving Gaze-Assisted Immersive Video Streaming
Jin, Y., Zhang, W., Xu, Z., Wang, F., & Liu, X. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3452510 · Cited by 15
“(cid:2) Viewport Prediction: Our new viewport prediction al-gorithm incorporates the crossmodal attention mechanism [15] based on the attention mechanism of transformer [16].”
BackgroundMethodJointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition
Goncalves, L., Chou, H.-C., Salman, A. N., Lee, C.-C., & Busso, C. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3530274 · Cited by 17
Multi-modal Sarcasm Detection on Social Media via Multi-Granularity Information Fusion
Ou, L., & Li, Z. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3715139 · Cited by 17
“To further validate the efficacy and efficiency of our proposed multi-modal fusion strategy, we conducted a comparison between the default GLIM employed by CAP and the Local – Local Interaction Method (LLIM) [38].”
Wang, H., Du, Q., & Xiang, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110262 · Cited by 15
An Entanglement-driven Fusion Neural Network for Video Sentiment Analysis
Gkoumas, D., Li, Q., Yu, Y., & Song, D. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/239 · Cited by 15
Utilizing BERT Intermediate Layers for Multimodal Sentiment Analysis
Zou, W., Ding, J., & Wang, C. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9860014 · Cited by 17
“MulT (Multimodal Transformer) [4] merges multimodal time-series via multiple directional pairwise crossmodal transformers.”
BackgroundInfluentialDeep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
Shi, P., Hu, M. Q., Shi, X., & Ren, F. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634706 · Cited by 15
“With transformer: Multimodal Transformer (MulT) [48], Multimodal Routing (MR) [49], Multi-scale Cooperative Multimodal Transformer (MCMulT) [28], Modality-Invariant and -Specific Representations For MSA (MISA) [15], Hybrid Contrastive learning (HyCon) [29], Multi-modal Sentiment Recognition With…”
BackgroundMethodInfluentialZhao, X., Li, X., Jiang, R., & Tang, B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102745 · Cited by 18
A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis
Tao, C., Li, J., Zang, T., & Gao, P. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i2.32146 · Cited by 18
“MulT (Tsai et al. 2019) integrates directional pairwise cross-modal attention to enhance three sets of Transform-ers, facilitating end-to-end multimodal data interaction and latent stream adaptation without explicit alignment.”
MULTICAUSENET temporal attention for multimodal emotion cause pair extraction
Ma, J., Chaudhry, H. N., Kulsoom, F., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-01221-w · Cited by 14
MAIN: Multimodal Attention-based Fusion Networks for Diagnosis Prediction
An, Y., Zhang, H., Sheng, Y., Wang, J., & Chen, X. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669634 · Cited by 15
Multimodal Sentiment Analysis on Unaligned Sequences Via Holographic Embedding
Ma, Y., & Ma, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747646 · Cited by 14
Zhao, X., Chen, Y., Chen, Y.-T., Liu, S., & Tang, B. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859747 · Cited by 14
“The baselines include the following state-of-the-art methods: Multimodal Transformer (MULT) [3], Interaction Canonical Correlation Network (ICCN) [18], Multimodal Adaptation Gate BERT (MAG-BERT) [4], Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis (MISA) [9],…”
BackgroundMethodA Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition
Gong, P., Liu, J., Wu, Z., Han, B., Wang, Y. K., & He, H. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.028291 · Cited by 15
CLAP: Contrastive Language-Audio Pre-training Model for Multi-modal Sentiment Analysis
Zhao, T., Kong, M. C., Liang, T., Zhu, Q., Kuang, K., & Wu, F. Q. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592296 · Cited by 16
“Related works propose fusion structures to make more efficient multi-modal information integration, such as cross-attention [33], feature splicing [41], and maximum mutual information (MI) [7].”
BackgroundMethodBlinkLinMulT: Transformer-Based Eye Blink Detection
Fodor, Á., Fenech, K., & Lörincz, A. (2023) · Journal of Imaging · doi:10.3390/jimaging9100196 · Cited by 15
“The architecture is motivated by Tsai et al. [27]; however, we replaced the quadratic attention modules with linear versions [28], providing similar performance [21] while being easier to train.”
MethodZero-shot multitask intent and emotion prediction from multimodal data: A benchmark study
Singh, G. V., Firdaus, M., Chauhan, D. S., Ekbal, A., & Bhattacharyya, P. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127128 · Cited by 15
Multimodal Sentiment Analysis Based on a Cross-Modal Multihead Attention Mechanism
Deng, L., Liu, B., & Li, Z. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.042150 · Cited by 18
Semisupervised Hierarchical Subspace Learning Model for Multimodal Social Media Sentiment Analysis
Han, X., Cheng, H., Ding, J., & Yan, S. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3350696 · Cited by 14
“Tsai et al. [16] developed a trans-former based multimodal feature extraction network, which used attention mechanisms to learn feature representations between modalities and enhance the ability of fused features to represent target information.”
BackgroundMultimodal consistency-specificity fusion based on information bottleneck for sentiment analysis
Liu, W., Cao, S., & Zhang, S. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.101943 · Cited by 14
“• MulT (Tsai et al., 2019b).”
Zhang, L., Jin, L., Xu, G., Li, X., Xu, C., Wei, K., Liu, N., & Liu, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28787 · Cited by 16
“For example, (Tsai et al. 2019)”
QEAN: quaternion-enhanced attention network for visual dance generation
Zhou, Z., Huo, Y., Huang, G., Zeng, A., Chen, X., Huang, L., & Li, Z. (2024) · The Visual Computer · doi:10.1007/s00371-024-03376-5 · Cited by 18
“However, a notable challenge in motion graphs was the generation of plausible transitions, and certain methods sought to overcome this by introducing parameter-izations for transitions [27].”
MethodView-interactive attention information alignment-guided fusion for incomplete multi-view clustering
Shu, Z., Luo, Y., Huang, Y., Mao, C., & Yu, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124258 · Cited by 14
SIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition
Li, S., Zhang, T., & Chen, C. L. P. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3409715 · Cited by 14
“Therefore, it is introduced into MER to capture cross-modal interactive relations, such as cross-modal Transformer (CT) [46], BMAN [47], CMRoBERTa [48], CHFN [49], MTSA [50], SPMN [51], and CTNet [52].”
MethodMultifaceted and deep semantic alignment network for multimodal sarcasm detection
Yu, B., Wang, H., & Xi, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112298 · Cited by 16
Enhancing multimodal depression detection with intra- and inter-sample contrastive learning
Li, M., Wei, Y., Zhu, Y., Wei, S., & Wu, B. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121282 · Cited by 16
Fu, J., Fu, Y., Xue, H., & Xu, Z. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01724-5 · Cited by 16
“• MulT [15] builds an architecture and achieves fusion using attention mechanisms by employing separate uni-modal and cross modal transformer networks.”
InfluentialMultimodal GRU with directed pairwise cross-modal attention for sentiment analysis
Qin, Z., Luo, Q., Zang, Z., & Fu, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-93023-3 · Cited by 20
“Furthermore, while the multimodal Transformer (MulT) model, proposed by Tsai et al. 33 , has demonstrated the power of attention mechanisms for multimodal fusion, it often faces significant computational challenges due to the heavy reliance on the Transformer architecture.”
A vector quantized masked autoencoder for audiovisual speech emotion recognition
Sadok, S., Leglaive, S., & Séguier, R. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104362 · Cited by 19
“In recent years, advances in artificial intelligence technology and hardware acceleration have enabled a shift towards multimodal processing in the field of emotion recognition [6, 7, 8].”
Qu, D., & Ma, Y. (2025) · Mathematics · doi:10.3390/math13172740 · Cited by 20
Multi-modal Fusion for Video Sentiment Analysis
Li, R., Zhao, J., Hu, J., Guo, S., & Jin, Q. (2020) · doi:10.1145/3423327.3423671 · Cited by 13
“Tsai et al.[27] proposed MMT (Multi-Modal Transformer) which uses a feed-forward fusion process to merge the multimodal series, which consisted of multiple crossmodal Transformer units.”
MethodA deep multimodal generative and fusion framework for class-imbalanced multimodal data
Li, Q., Yu, G., Wang, J., & Liu, Y. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09227-4 · Cited by 14
Liang, S. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2419609 · Cited by 13
“Tsai et al. (2019) employed a directed bidirectional cross-modal attention mechanism to enable interaction between multimodal sequences characterized by varying time steps.”
Huo, T., Wu, W., Chen, X., Xue, M., Liu, P., Zhang, J., Xie, Y., Wang, H., Zhou, H., Yan, Z., et al. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.06.005 · Cited by 15
Multi-Modal Pre-Training for Automated Speech Recognition
Chan, D. M., Ghosh, S., Chakrabarty, D., & Hoffmeister, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746449 · Cited by 13
Zhang, Y., Deng, Y., Zhou, Z., Zhang, X., Jiao, P., & Zhao, Z. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.1021400 · Cited by 13
Liu, P., Zheng, X. W., Li, H., Liu, J., Ren, Y., Zhu, H., & Sun, L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096470 · Cited by 13
“MulT [6] proposes the multimodal transformer, where a cross-modal attention module is learned for reinforcing one Model MOSI MOSEI MAE Corr Acc-2 F1 MAE Corr Acc-2 F1 TFN 0.”
BackgroundMethodA Collaborative Multimodal Learning-Based Framework for COVID-19 Diagnosis
Gao, Y., Gong, M., Ong, Y.-S., Qin, A. K., Wu, Y., & Xie, F. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3290188 · Cited by 13
“…translation [27] that translates data from one modality to another with the open-ended and subjective relationship between them; alignment [28] that identifies the direct relations between subelements from diverse modalities; fusion [29] that joins information from different modalities to…”
BackgroundPredict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection
Tiwari, D., Kanojia, D., Ray, A., Nunna, A., & Bhattacharyya, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.988 · Cited by 16
“Efficient multimodal fusion approaches have also been discussed in (Sahay et al., 2020; Tsai et al., 2019; Liu et al., 2018).”
BackgroundLow-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis
Miao, X., Zhang, X., & Zhang, H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18032-8 · Cited by 13
“MulT [45].”
Li, J., Cai, S., Li, L., Sun, R., Yuan, G., & Zhu, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123655 · Cited by 14
Cheng, S., Shang, P., Zhang, Y., Guan, J., Chen, Y., Lv, Z., Huang, S., Liu, Y., & Xie, H. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106646 · Cited by 13
Mual: enhancing multimodal sentiment analysis with cross-modal attention and difference loss
Deng, Y., Li, Y., Xian, S., Li, L., & Qiu, H. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00340-w · Cited by 14
“[22] proposed a multimodal Transformer for sentiment analysis, which has been applied to different domains with good results [23, 24].”
Yang, D., Li, M., Qu, L., Yang, K., Zhai, P., Wang, S., & Zhang, L. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3435561 · Cited by 16
“The task-related losses are selected to follow the consensus of previous mainstream works [6], [12], [25] for a fair comparison.”
InfluentialGraph Reconstruction Attention Fusion Network for Multimodal Sentiment Analysis
Hu, R., Yi, J., Chen, L., & Jin, Z. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3452204 · Cited by 13
“MulT [6], EMT [14], TeFNA Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
Xiang, P., Lin, C., Wu, K., & Bai, O. (2024) · International Conference on Pattern Recognition Systems (ICPRS) · doi:10.1109/icprs62101.2024.10677820 · Cited by 22
Early diagnosis of Alzheimer’s Disease based on multi-attention mechanism
Yang, X., Hong, K., Zhang, D., & Wang, K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310966 · Cited by 19
Li, X., Zhang, F., Hua, Q., & Dong, C.-R. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121515 · Cited by 13
Enhancing video rumor detection through multimodal deep feature fusion with time-sync comments
Yin, M., Chen, W., Zhu, D., & Jiang, J. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103935 · Cited by 15
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
