The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 701–800 of 1,004
Personalized emotion analysis based on fuzzy multi-modal transformer model
Liu, J., Ang, M. C., Chaw, J. K., Ng, K. W., & Kor, A.-L. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05954-5 · Cited by 13
“To align with previous studies [67, 99], we employ accuracy (Acc.) and F1-score (F1) as evaluation metrics for assessing model performance on the IEMOCAP dataset.”
Tripartite interaction representation learning for multi-modal sentiment analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Yin, W., & Lu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126279 · Cited by 10
Exploring Emotion Expression Recognition in Older Adults Interacting With a Virtual Coach
Palmero, C., deVelasco, M., Hmani, M. A., Mtibaa, A., Letaifa, L. B., Buch-Cardona, P., Justo, R., Amorese, T., González–Fraile, E., Fernández-Ruanova, B., et al. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3558141 · Cited by 15
“For the latter, a flexible temporal synchronization able to capture long-term cross-modal dependencies would compensate for the unaligned nature of communication [102].”
Xu, Z., Chen, N., Li, G., Li, J., Zhu, H., & Zhu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128035 · Cited by 10
Bimodal speech emotion recognition via contrastive self-alignment learning
Wang, C., Wen, G., Yang, P., & Liu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128605 · Cited by 11
Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis
Tuerhong, G., Fu, F., & Wushouer, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-11848-4 · Cited by 13
“Currently, a common approach is to combine features of different modalities in pairs (Tsai et al. 8 ) and use cross-attention mechanisms to weight and mark important regions of each modality.”
Multimodal sentiment analysis using image and text fusion for emotion detection
Deshpande, U. U., Shanbhag, S., Sukhasare, A., Dixit, M. M., Patil, R., Sangani, S., Srinivasaiah, S. H., Goudar, S. I., & Managuli, M. (2025) · Discover Computing · doi:10.1007/s10791-025-09756-2 · Cited by 11
“Modality-specific and modality-invariant representations were presented [13] for multimodal representation learning.”
Improving EEG-based Emotion Recognition by Fusing Time-Frequency and Spatial Representations
Zhu, K., Zhang, X., Wang, J., et al. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097171 · Cited by 10
Multimodal Sentiment Analysis Using BiGRU and Attention-Based Hybrid Fusion Strategy
Liu, Z., Zhou, B., Meng, L., & Huang, G. (2023) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.038835 · Cited by 10
MSFNet: modality smoothing fusion network for multimodal aspect-based sentiment analysis
Xiang, Y., Cai, Y., & Guo, J. (2023) · Frontiers in Physics · doi:10.3389/fphy.2023.1187503 · Cited by 10
“3 Multi-channel attention We verified the effectiveness of the multi-channel attentionbased interaction (MCA) by deleting it or replacing it with the Cross-Modal Transformer (CMT) [20].”
MethodInfluentialDynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis
Wang, X., Zhang, M., Chen, B., Wei, D., & Shao, Y. (2023) · Electronics · doi:10.3390/electronics12132986 · Cited by 13
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning
Mai, S., Ya, S., & Hu, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.209 · Cited by 13
“To highlight important information across modalities, many algorithms apply cross-modal attention mechanisms (Zadeh et al., 2018b; Tsai et al., 2019; Shenoy and Sardana, 2020).”
M2FNet: Multimodal Fusion Network for Airport Runway Subsurface Defect Detection Using GPR Data
Li, N., Wu, R., Li, H., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3308205 · Cited by 10
“all multimodal features are of the same dimension, the 1-D convolutional layer is used as an alignment tool [37].”
MethodBalance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks
Wang, Y., Jian, X., & Xue, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.652 · Cited by 15
Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection
Zhang, X., Yang, H., Wang, T., Li, X., & Yin, L. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00597 · Cited by 10
“The results show that our model reaches and surpasses the state-of-the-art.”
BackgroundM2P2: Multimodal Persuasion Prediction Using Adaptive Fusion
Bai, C., Chen, H., Kumar, S., Leskovec, J., & Subrahmanian, V. S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3134168 · Cited by 12
“In order to capture long-term dependencies, M2P2 uses the Transformer encoder [34, 33] to learn latent embeddings for modalities.”
MethodMultimodal high-order relational network for vision-and-language tasks
Pan, H., & Huang, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.03.071 · Cited by 10
Gao, C., Marsic, I., Sarcevic, A., Gestrich-Thompson, W. V., & Burd, R. S. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3580798 · Cited by 10
“Recent methods [23, 61] have focused on aligning different modalities along the time axis, achieving better performance than unaligned methods.”
Improved Speech Emotion Classification Using Deep Neural Network
Saeed, M. H. (2023) · Circuits Systems and Signal Processing · doi:10.1007/s00034-023-02446-8 · Cited by 10
“These feature extraction packages have been utilized by the authors in [30–37] for their SER studies.”
MethodUnsupervised Multimodal Language Representations using Convolutional AutoencodersPreprint
Koromilas, P., & Γιαννακόπουλος, Θ. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03007 · Cited by 10
Detecting Change Talk in Motivational Interviewing using Verbal and Facial Information
Nakano, Y., Hirose, E., Sakato, T., Okada, S., & Martin, J. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556607 · Cited by 9
A Novel Rumor Detection Method Based on Non-Consecutive Semantic Features and Comment Stance
Zhu, Y., Wang, G., Li, S., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3284308 · Cited by 9
“Z = Crossmodal_Transformer(Spi ,Upi ) (3) Cross-modal Transformer is an extension of the Transformer that can handle tasks with information from multiple modalities.”
BackgroundInfluentialXu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_6 · Cited by 9
Multimodal Sentiment Analysis Method Based on Hierarchical Adaptive Feature Fusion Network
Zhang, H. (2024) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.335918 · Cited by 9
“The following is a comparative of the proposed model with TFN (Zadeh et al., 2017), MulT (Tsai et al., 2019), MISA (Hazarika et al., 2020), Self-MM (Yu et al., 2021), and ConFEDE (Yang et al., 2023) based on the CH-SIMS dataset: (1) TFN (Zadeh et al., 2017): First, create multidimensional tensors…”
MethodInfluentialAction recognition in compressed domains: A survey
Ming, Y., Zhou, J., Hu, N., Feng, F., Zhao, P., Lyu, B., & Yu, H. L. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127389 · Cited by 14
A Novel Multimodal Sentiment Analysis Model Based on Gated Fusion and Multi-Task Learning
Sun, X., Ren, X., & Xie, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446040 · Cited by 9
“(v) Transformer-based methods, such as MulT [7] and BBFN [6].”
MethodYu, B., & Shi, Z. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06422-0 · Cited by 10
“The cross-modal transformer encoder structure used in this paper comes from MulT [12], which is a deep stack of cross-attention blocks with positional embedding and layer normalization.”
Automatic movie genre classification & emotion recognition via a BiProjection Multimodal Transformer
Moreno-Galván, D. A., López, R., González, L. C., Montes-y-Gómez, M., Sánchez-Vega, F., & López-Monroy, A. P. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102641 · Cited by 14
Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality
Che, L., Wang, J., Liu, X., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70378-2_25 · Cited by 9
A temporal–spectral fusion transformer with subject-specific adapter for enhancing RSVP-BCI decoding
Li, X., Wei, W., Qiu, S., & He, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106844 · Cited by 12
“…EEG decoding and obtain more discriminative high-level representations [35, 36, 37], we integrate the temporal and spectral information of EEG signals within a multi-view learning network based on Transformer, which is compatible with diverse modalities and can realize cross-view interaction [38].”
BackgroundLi, C., Xie, L., Wang, X., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125822 · Cited by 9
Adaptive Multimodal Graph Integration Network for Multimodal Sentiment Analysis
Gong, P., Liu, J., Zhang, X., Li, X., Wei, L., & He, H. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3507576 · Cited by 11
“Examplesofsuch methodsincludeLMF[27],MFN[9],TFN[26],ICCN[28], MulT[13],BIMHA[52],CITN-DAF[53],BBFN[54],MAG-BERT[31],SSE-FT[55],BAFN[56],MTAG[17],TGCN[19], Multimodal Graph [57], GraphCAGE [58], MAGCN [41], and DEAN [59].”
Global distilling framework with cognitive gravitation for multimodal emotion recognition
Li, H., Zhong, H., Xu, C., Liu, X., Wen, G., & Liu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2024.129306 · Cited by 9
Zhang, Y., Zhong, H., Alhusaini, N., Chen, G., & Wu, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113121 · Cited by 9
Chen, L., Hu, Z., Chen, J., & Sun, Y. (2025) · Remote Sensing · doi:10.3390/rs17050868 · Cited by 12
Yin, X., Wu, T., Yu, J., He, X., & Xu, L. (2024) · Acta Oceanologica Sinica · doi:10.1007/s13131-024-2450-4 · Cited by 11
“Inspired by a multi-modal transformer (Tsai et al., 2019), this study assumes that a good way to fuse multiple characteristics is to provide cross-characteristic latent adaptation, that is, b to a.”
Qin, J., Liu, F., & Zong, L. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107449 · Cited by 9
Detecting sarcasm in user-generated content integrating transformers and gated graph neural networks
Qin, Z., Luo, Q., Zang, Z., & Fu, H. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2817 · Cited by 12
Multi-modal sentiment recognition with residual gating network and emotion intensity attention
Wang, Y., Guo, X., Hou, X., Miao, Z., Yang, X., & Guo, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107483 · Cited by 9
Wen, Y., Li, Q., Zhou, Y., Gao, Y., Wen, Z., Tao, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3578618 · Cited by 12
“To comprehensively validate the performance of our model, we compare our method with several advanced and SOTA baselines in Tables II and III: TFN [30], LMF [31], MulT [14], MAG-BERT [57], MISA [58], Self-MM [48], MMIM [32], and ConFEDE [43].”
InfluentialA dual path graph neural network framework for dementia diagnosis
Zhang, D., & Zhu, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-06519-3 · Cited by 13
Sparse Information Perception Network for Remote Sensing Cross-Modal Retrieval
Ouyang, P., Ma, Q., & Bai, C. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3597055 · Cited by 9
Wu, X., Sun, H., Wang, Y., Nie, J., Zhang, J., Wang, Y., Xue, J., & He, L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00854 · Cited by 12
Dharia, S. Y., Liu, Q., Smith, S. D., & Valderrama, C. E. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108390 · Cited by 11
Multimodal Classification and Out-of-Distribution Detection for Multimodal Intent Understanding
Zhang, H., Zhou, Q., Xu, H., Su, J., Evans, R. H., & Gao, K. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3618541 · Cited by 19
“These include TEXT, MulT [7], MAG-BERT [9], MMIM [46], SDIF [10], SPECTRA [70], TCL-MAP [11], as well as MIntOOD (R) and MIntOOD (T), two variants of our method.”
InfluentialIncorporating Multimodal Sentiments into Conversational Bots for Service Requirement Elicitation
Yu, D., Tian, J., Su, T., Tu, Z., Xu, X., & Wang, Z. (2021) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose52839.2021.00014 · Cited by 9
Quan, Z., Sun, T., Su, M., Wei, J., Zhang, X., & Zhong, S. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945514 · Cited by 10
“• MULT : Multimodal Transformer (MULT) [6] extends the multimodal converter architecture using directed pairwise cross-attention, which converts one modality to another using directed pairwise cross-attention.”
BackgroundInfluentialMultimodal Learning for Predicting Mortality in Patients with Pulmonary Arterial Hypertension
Suvon, M. N. I., Tripathi, P. C., Alabed, S., Swift, A. J., & Lu, H. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995597 · Cited by 9
“related research in multimodal transformers [26]–[29] and other baselines like concatenation and MLP.”
BackgroundMMATR: A Lightweight Approach for Multimodal Sentiment Analysis Based on Tensor Methods
Koromilas, P., Nicolaou, M. A., Γιαννακόπουλος, Θ., & Panagakis, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097030 · Cited by 9
“…models the parameters of the unimodal feature extraction networks (e.g., BERT) are not included in the final number of parameters, since it is a common practice [12, 10, 13, 14, 15, 16, 17, 4, 8, 2, 3, 1] to use pre-extracted features in order to evaluate the performance on multimodal sentiment.”
BackgroundMethodInfluentialTraffic Flow Prediction Based on Two-Channel Multi-Modal Fusion of MCB and Attention
Qin, X. (2023) · IEEE Access · doi:10.1109/access.2023.3280068 · Cited by 9
“MulT [32]: For any target mode, cross-modal Transformer is used to convert the information of other modes to the target mode.”
MethodTowards Arabic Multimodal Dataset for Sentiment Analysis
Haouhat, A., Bellaouar, S., Nehar, A., & Cherroun, H. (2023) · International Conference on Intelligent Data Science Technologies and Applications (IDSTA) · doi:10.1109/idsta58916.2023.10317847 · Cited by 14
“The initial state-of-the-art model, known as the Multimodal transformer (Mult) [19], is a transformer-based model that deploys an attention mechanism.”
BackgroundMethodResultVAD: A Video Affective Dataset With Danmu
Wang, S., Li, X., Zheng, F., Pan, J., Li, X., Chang, Y., Zhu, Z., Li, Q., Wang, J., & Xiao, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3382503 · Cited by 9
Text-image semantic relevance identification for aspect-based multimodal sentiment analysis
Zhang, T., Zhou, G., Lu, J., Li, Z., Wu, H., & Liu, S. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1904 · Cited by 12
Transformer-based adaptive contrastive learning for multimodal sentiment analysis
Hu, Y., Huang, X., Wang, X., Lin, H., & Zhang, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19152-5 · Cited by 13
“Increasingly,researchershaverecognizedthisissue [3,8],notingasignificantdeclineinmodelaccuracyinMSAtasksupontheexclusionof thetextmodality[9].”
InfluentialTowards Asynchronous Multimodal Signal Interaction and Fusion via Tailored Transformers
Yang, D., Kuang, H., Yang, K., Li, M., & Zhang, L. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3409211 · Cited by 9
“For instance, MulT [21] proposed a multimodal transformer to perform directional cross-modal interactions between two paired signals.”
BackgroundMethodInfluentialEnhancing personalized gene expression prediction from DNA sequences using genomic foundation models
Ramprasad, P., Pai, N., & Pan, W. (2024) · Human Genetics and Genomics Advances · doi:10.1016/j.xhgg.2024.100347 · Cited by 14
End-to-End Learning for Multimodal Emotion Recognition in Video With Adaptive Loss
Huynh, V. T., Yang, H.-J., Lee, G.-S., & Kim, S.-H. (2021) · IEEE Multimedia · doi:10.1109/mmul.2021.3080305 · Cited by 9
MMTrans-MT: A Framework for Multimodal Emotion Recognition Using Multitask Learning
Shen, J., Zheng, J., & Wang, X. (2021) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci52617.2021.9435906 · Cited by 9
“This work uses BiLSTM [9] and Multimodal Transformer (MulT) [14] as two baseline models for both CMU-MOSEI dataset and IEMOCAP dataset.”
MethodLanguage Reinforced Superposition Multimodal Fusion for Sentiment Analysis
He, J., & Hu, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3180668 · Cited by 10
Improving Targeted Multimodal Sentiment Classification with Semantic Description of Images
An, J., Zainon, W. M. N. W., & Zhang, H. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.038220 · Cited by 9
Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos
Alnajjar, K., Hämäläinen, M. K., & Zhang, S. (2022) · Workshop on Figurative Language Processing (FigLang) · doi:10.18653/v1/2022.flp-1.4 · Cited by 9
Gate-Fusion Transformer for Multimodal Sentiment Analysis
Xie, L., & Zhang, X.-Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59830-3_3 · Cited by 9
Integrating Multimodal Information in Large Pretrained TransformersPreprint
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A., Mao, C., Morency, L., & Hoque, M. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05787 · Cited by 39
Self-citationUsing Large Pre-Trained Models with Cross-Modal Attention for Multi-Modal Emotion RecognitionPreprint
N, K. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09669 · Cited by 11
“Attention-based methods [3,34,16,37] have been used for emotion recognition task.”
TEASEL: A Transformer-Based Speech-Prefixed Language ModelPreprint
Arjmand, M., Dousti, M. J., & Moradi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05522 · Cited by 21
“Finally, previous works have shown that the speech modality serves more information in multimodal language settings [6, 7].”
BackgroundMethodInfluentialScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally DescriptorsPreprint
Luo, H., Ji, L., Huang, Y., Wang, B., Ji, S., & Li, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01368 · Cited by 20
“…(Rahman et al., 2020) -/84.7 -/84.5 - - MAG-XLNet (Rahman et al., 2020) -/85.6 -/85.7 - - TFN (Zadeh et al., 2017) -/82.5 -/82.1 0.593 0.700 MulT (Tsai et al., 2019) -/81.6 -/81.6 0.591 0.694 ICCN (Sun et al., 2020) -/84.2 -/84.2 0.565 0.713 MISA (Hazarika et al. 2020) 83.6/85.5 83.8/85.3 0.555…”
InfluentialSparse Fusion for Multimodal TransformersPreprint
Ding, Y., Rich, A., Wang, M. L., Stier, N., Turk, M., Sen, P., & Höllerer, T. (2022) · doi:10.31219/osf.io/f7jvn · Cited by 9
A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cues
Armitage, J., Impett, L., & Sennrich, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00115 · Cited by 8
“Methods for Aligning and Localising Features in Linguistic and Visual Sequences Alignment in multimodal tasks is often posited as an implicit subprocess in an attention-based component of a transformer (Tsai et al., 2019; Zhu et al., 2021b).”
BackgroundXie, Y., Niu, J., Zhang, Y., & Ren, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3307972 · Cited by 12
“For tri-modal fusion (such as text-visual-audio), previous research [15], [16], [27] mostly uses ternary-symmetric architectures, in which bi-directional relationships in each multi-modal pair are modeled in some way.”
BackgroundMethodInfluentialExploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction
Yi, G., Yang, Y., Pan, Y., Cao, Y., Yao, J., Lv, X., Fan, C., Lv, Z., Tao, J., Liang, S., et al. (2023) · doi:10.1145/3606039.3613109 · Cited by 8
“Based on numerous experiments, we have found that traditional Multimodal Transformers [27] are still simple and effective.”
BackgroundQuan, Z., Chen, Q., Wang, W., Zhang, M., Li, X., Li, Y., & Liu, Z. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3337367 · Cited by 10
“To capture intermodal interaction information, we employ the cross-view transformer method [44].”
BackgroundMethodMultimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception
Wang, L. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2023.1324831 · Cited by 8
“The experimental results show that the GRU model performs well on both Music Emotion Dataset and Multimodal DataSet with significant performance advantages.”
BackgroundInfluentialFlexible Material Quality Assessment Based on Visual–Tactile Fusion
Xu, S., Xu, H., Mao, F., Su, W., Ji, M., Gan, H., & Yang, W. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3386205 · Cited by 8
“The basic attention-based multimodal feature fusion module calculates a weighted sum based on the input query vector and key-value pairs of each modality to represent the interaction between different modalities [35].”
BackgroundSemantic-wise guidance for efficient multimodal emotion recognition with missing modalities
Liu, S., Wang, Y., Wang, K., Li, B., Yang, F., & Yang, S. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01310-2 · Cited by 8
“Tsai et al. [31] proposed Multimodal Transformer (MulT) consisting of paired cross-modality attention mechanisms.”
From Extraction to Generation: Multimodal Emotion-Cause Pair Generation in Conversations
Ma, H., Yu, J., Wang, F., Cao, H., & Xia, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3446646 · Cited by 9
“Most existing studies on MEAC primarily focus on recognizing the emotion category expressed in each utterance of a conversation [5], [7], [8], [12], [13].”
TriSAT: Trimodal Representation Learning for Multimodal Sentiment Analysis
Huan, R., Zhong, G., Chen, P., & Liang, R. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3458812 · Cited by 9
“Results on IEMOCAP dataset: TriSAT exhibits substantial improvements in Acc2 for the happy, sad, and neutral emotional categories, with gains of 0.4%, 0.1%, and 1.5%, respectively, compared to the previous best MulT [8].”
BackgroundMethodResultInfluentialTowards Multimodal Prediction of Spontaneous Humor: A Novel Dataset and First Results
Christ, L., Amiriparian, S., Kathan, A., Müller, N., Konig, A. S., & Schuller, B. W. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3475736 · Cited by 32
Ji, M., Wei, N., Zhou, J., & Wang, X. (2024) · Electronics · doi:10.3390/electronics13214175 · Cited by 9
GRACE: GRadient-based Active Learning with Curriculum Enhancement for Multimodal Sentiment Analysis
Li, X., Ye, W., Zhang, Y., & Sun, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681617 · Cited by 11
Liu, M., Wang, Z., & Liu, F. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822358 · Cited by 9
“Like the MulT model [18], the CrossMamba module comprises three branches, each enhancing its modality-specific modal features through cross-modal attention mechanisms while iteratively leveraging features from the other modalities.”
Multi-level language interaction transformer for multimodal sentiment analysis
Li, Y., Liu, A., & Lu, Y. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00923-x · Cited by 11
“It has been widely used in multimodal sentiment analysis (Shi et al., 2024; Tsai et al., 2019; Yu et al., 2022; Sun et al., 2022; Guarascio et al., 2024).”
Affect and Personality Aided Modeling of Transcribed Speech for Depression Severity Estimation
Gönç, K., & Dibeklioğlu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3560476 · Cited by 9
“Pair-wise cross-attention in both directions, often termed bidirectional cross-attention, captures dependencies between feature sets, as seen in works fusing visual, auditory, and text data for applications like affect recognition and diagnosis [44], [45], [46], [47], [48], [49].”
Mamba-assisted modality subspace complementary fusion for multimodal sentiment analysis
Chen, Q., Tang, Y., & Liu, H. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.03.021 · Cited by 9
SAGE-Net: Single-layer augmented gated encoder network for efficient multimodal sentiment analysis
Zhou, J., Kang, X., Ding, W., Wang, L., Ding, F., Matsumoto, K., Zhang, C., & Chi, H. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113665 · Cited by 10
Chen, J., Zhong, Y., Huang, Q., Huang, C., Jiang, F., Huang, X., & Wang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3606964 · Cited by 9
“Driven by the development of attention mechanisms and the widespread application of Transformers [33], Tsai et al. introduced cross-modal attention [34] to enhance interaction between different modalities.”
InfluentialShen, Y., Lin, X., & Fan, W. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209824 · Cited by 8
Jin, X. H., Yu, J., Ding, Z., Xia, R., Zhou, X., & Tu, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60457-8_4 · Cited by 8
Transformer-Based Physiological Feature Learning for Multimodal Analysis of Self-Reported Sentiment
Katada, S., Okada, S., & Komatani, K. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556576 · Cited by 8
“(4) CrossModal Transformer (CMTr): The CMTr is a core component of the MulT and was proposed in [27].”
BackgroundMethodInfluentialChen, Z.-Y., Fan, Z., & Sun, M. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3267858 · Cited by 10
“Moreover, instead of the commonly used query-key-value attention model for the vision-and-language cross-modal correlation/alignment [16], [35], [51], the additive attention model is suitable for the diverse information fusion for inventory management problems.”
MethodHuang, Y., Wang, M., Zheng, Z., Ma, M., Fei, X., Wei, L., & Chen, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104427 · Cited by 9
He, P., Qi, H., Wang, S., & Cang, J. (2023) · Applied Sciences · doi:10.3390/app13137489 · Cited by 8
Multimodal attention for lip synthesis using conditional generative adversarial networks
Vidal, A., & Busso, C. (2023) · Speech Communication · doi:10.1016/j.specom.2023.102959 · Cited by 8
“This approach has been successfully used in multimodal fusion problems (Goncalves and Busso, 2022; Tsai et al., 2019).”
MethodXu, C., Du, Y., Wang, J., Zheng, W., Li, T., & Yuan, Z.-S. (2023) · Computational Intelligence · doi:10.1111/coin.12607 · Cited by 9
Fen, L., Chen, J., Li, K., Bai, J., Tan, W., Cai, C., & Ayub, M. S. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104265 · Cited by 8
Huang, Q., Cai, P., Nie, T., & Zeng, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446825 · Cited by 8
“MulT [5] constructed unimodal and cross-modal Transformers to accomplish the fusion process through attention.”
BackgroundMethodResultLi, Y., Weng, W., & Liu, C. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09634-w · Cited by 8
“• MulT [24]: It is a network structure based on the attention mechanism to construct unimodal and cross-modal transformers.”
Forecasting of 3D Whole-Body Human Poses with Grasping Objects
Yan, H., Cui, Q., Xie, J., & Guo, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00170 · Cited by 8
“Trans-former encoders have demonstrated efficacy in fusing multimodal features [31, 49, 54].”
Semantic Feature Scheduling and Rate Control in Multi-Modal Distributed Network
Gao, H., Yu, G., He, Y., & Liu, Y. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3480352 · Cited by 11
“We compare the performance with the existing multi-modal fusion models like MulT [43] and ICCN [45] on both MOSI and MOSEI datasets as shown in Table I. Specifically, MulT learns representations from unaligned multi-modal time series using directional pairwise cross-modal attention.”
M3: MultiModal Masking Applied to Sentiment Analysis
Georgiou, E., Paraskevopoulos, G., & Potamianos, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1739 · Cited by 8
“In the future we plan to integrate M 3 with more architectures (Transformers) and experiment with more diverse multimodal tasks, e.g. Multimodal ASR and VQA, which would involve more than three modalities.”
BackgroundMethodInfluentialHuman-Guided Modality Informativeness for Affective States
Wörtwein, T., Sheeber, L., Allen, N. B., Cohn, J. F., & Morency, L. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3481004 · Cited by 8
“Modality informativeness: Many models that focus on multi-modal fusion implicitly or explicitly estimate the informativeness of modalities [25, 37, 38].”
BackgroundSelf-citation
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
