The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 601–700 of 1,004
ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactions
Luo, C. X., Song, S., Xie, W., Spitale, M., Ge, Z., Shen, L., & Güneş, H. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3490613 · Cited by 34
“A straightforward yet effective approach to unifying different modalities has been the adoption of multimodal transformers, as demonstrated by Tsai et al. [73].”
Nguyen, Q. H., Nguyen, M.-V. T., & Nguyen, K. V. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01558-8 · Cited by 16
“To achieve this goal, we apply Cross-modal Attention [69] to model the interaction between context and image, which treats the hidden states of the input sequence as a query, and the image features v I k as keys and values, as follows: where ∈ ℝ d × N is the generated image-based context representation.”
WavFusion: Towards Wav2vec 2.0 Multimodal Speech Emotion Recognition
Li, F., Luo, J., & Xia, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2071-5_24 · Cited by 13
Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition
He, P., Yu, J., Ge, C., Yu, Y., Xu, W., Wang, L., Liu, T., & Kan, Z. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3711865 · Cited by 16
“The Multimodal Transformer (MulT) [57] approach extends the self-attention mechanism of the original Transformer network to learn cross-modal dependencies between elements [2, 15, 21, 31, 70].”
InfluentialTemporal text-guided feedback-based progressive fusion network for multimodal sentiment analysis
Yang, Z., He, Q., Du, N., & He, Q. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.117 · Cited by 13
Zhu, M., He, X., Qiao, B., Luo, Y., Li, Z., & Pan, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113249 · Cited by 15
Hemamou, L., Guillon, A., Martin, J., & Clavel, C. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3113159 · Cited by 14
“Models based on memory cells [79], attention mechanisms [78], transformers architectures [70], shift of word embeddings [73] or joint representations [52] have been showing increased performance on emotion prediction task.”
BackgroundMulti-Modal Learning for AU Detection Based on Multi-Head Fused Transformers
Zhang, X., & Yin, L. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667030 · Cited by 13
“Inspired by these vision transformer works [10], [6], and multi-modal transformer works [32], [27], we explore the methods that are utilized transformer architecture for both feature representation and fusion.”
BackgroundMethodDementia Detection by Fusing Speech and Eye-Tracking Representation
Sheng, Z.-Y., Guo, Z., Li, X., Li, Y., & Ling, Z.-H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747054 · Cited by 13
AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model
Mingyu, J., Zhou, J., & Ning, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273936 · Cited by 16
“Referring to the literature [36], we use 1D temporal convolutional layer as a sequence alignment tool to ensure that both are of the same dimension.”
BackgroundMethodLearning Cross-Modal Audiovisual Representations with Ladder Networks for Emotion Recognition
Goncalves, L., & Busso, C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096138 · Cited by 13
“Multimodal representation learning approaches have been explored more extensively lately with advancements in computational resources and multimodal deep learning methods [4, 5, 10, 12, 19, 20].”
BackgroundInfluentialCross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition
Hu, Y., Li, R., Chen, C., Zou, H., Zhu, Q., & Chng, E. S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/564 · Cited by 13
Weakly-supervised Multi-task Learning for Multimodal Affect RecognitionPreprint
Dai, W., Cahyawijaya, S., Bang, Y., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11560 · Cited by 15
“Although the human annotation has nine emotion categories, following the prior works (Haz-arika et al., 2018; Wang et al., 2019a; Tsai et al., 2019; Dai et al., 2020a), we take four categories: neutral , happy , sad , and angry .”
BackgroundMethodInfluentialIntegrating multimodal data and machine learning for entrepreneurship research
Shrestha, Y. R., & He, V. F. (2025) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1546 · Cited by 15
Dynamic Cross-Feature Fusion for American Sign Language Translation
Ananthanarayana, T., Kotecha, N., Srivastava, P., Chaudhary, L., Wilkins, N., & Nwogu, I. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667027 · Cited by 13
Dutt, A., Zare, A., & Gader, P. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3217485 · Cited by 12
Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion
Wu, S., Dai, D., Qin, Z., Liu, T., Lin, B., Cao, Y., & Sui, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.124 · Cited by 18
Multimodal Deep Learning Approach for Real-Time Sentiment Analysis in Video Streaming
Tejashwini, S. G., & Aradhana, D. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140881 · Cited by 12
Brain-Inspired Recognition System Based on Multimodal In-Memory Computing Framework for Edge AI
Xiao, H., Xie, D., Hu, X., Zhou, Y., Zhou, G., & Duan, S. (2023) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2023.3337532 · Cited by 12
“To evaluate the effectiveness of our proposed system in MSA and ERC tasks, we compared it with 16 baseline meth-ods [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62].”
MethodMulti-layer cross-modality attention fusion network for multimodal sentiment analysis
Yin, Z., Du, Y., Liu, Y., & Wang, Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17685-9 · Cited by 13
“However, Tsai et al. [11] believe that different modality feature is not necessary to be aligned explicitly due to variable sampling rates of the sequence, so they propose an information extraction approach based on non-alignment Transformer to achieve text-audio sentiment analysis.”
Balanced sentimental information via multimodal interaction model
Luo, Y., Wu, R., Liu, J., & Tang, X. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01208-5 · Cited by 12
“The second technique regards a value less than 0 as negative and the value greater than 0 as positive, excluding the value equal to 0 [12].”
InfluentialText-Centric Multimodal Contrastive Learning for Sentiment Analysis
Peng, H., Gu, X. F., Li, J., Wang, Z., & Xu, H. (2024) · Electronics · doi:10.3390/electronics13061149 · Cited by 15
Variational Mixture of Stochastic Experts Auto-Encoder for Multi-Modal Recommendation
Yi, J., & Chen, Z. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3384058 · Cited by 15
“Then, we train our proposed model, where we search for the optimal parameters for the multi-modal autoencoder, and set a small learning rate as 1e-5 for the collaborative model, with the weight of the similarity constraint searched in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].”
BackgroundMethodInfluentialA Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis
Yi, G., Fan, C., Tao, J., Lv, Z., Wen, Z., Pei, G., & Li, T. (2024) · Intelligent Computing · doi:10.34133/icomputing.0081 · Cited by 12
Weakly Correlated Multimodal Sentiment Analysis: New Dataset and Topic-Oriented Model
Liu, W., Li, W., Ruan, Y.-P., Shu, Y., Chen, J., Li, Y., Yu, C., Zhang, Y., Guan, J., & Zhou, S. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3396144 · Cited by 14
CCMA: CapsNet for audio–video sentiment analysis using cross-modal attention
Li, H., Guo, A., & Li, Y. (2024) · The Visual Computer · doi:10.1007/s00371-024-03453-9 · Cited by 12
“In the MOSEI datasets comparison experiment, we compare with MULT [31], TAV [29], VAE-AMDT [49] and SAER [47].”
An autoencoder-based self-supervised learning for multimodal sentiment analysis
Feng, W., Wang, X., Cao, D., & Lin, D. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120682 · Cited by 13
Liu, W., Xu, H., Hua, Y., Chi, Y., & Gao, K. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200410 · Cited by 13
Zhang, H., Yan, Y., Cai, Z., Zhan, P., Chen, B. Y., Jiang, B., & Xie, B. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112346 · Cited by 14
Li, Z., Liu, P., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., & Wang, H. (2024) · Applied Intelligence · doi:10.1007/s10489-024-06150-1 · Cited by 14
TriagedMSA: Triaging Sentimental Disagreement in Multimodal Sentiment Analysis
Luo, Y., Liu, W., Sun, Q., Li, S., Li, J., Wu, R., & Tang, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3524789 · Cited by 15
Chakkarapani, V., Poornapushpakala, S., & Suresh, S. (2025) · SN Computer Science · doi:10.1007/s42979-024-03601-x · Cited by 12
“Few researches as outlined in [16, 17] highlights the usage the multi-modal data shows better performance in diagnosing skin cancer detection compared to models using single modality.”
Li, Y.-Y., Bai, Y., Wang, C., Qu, M., Lu, Z., Soria, R., & Liu, J. (2025) · Intelligent Computing · doi:10.34133/icomputing.0110 · Cited by 18
Multimodal hypergraph network with contrastive learning for sentiment analysis
Huang, J., Jiang, K., Pu, Y., Zhao, Z., Yang, Q., Gu, J., & Xu, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129566 · Cited by 15
Adaptive Multi-Scale Language Reinforcement for Multimodal Named Entity Recognition
Li, E., Li, T., Luo, H., Chu, J., Duan, L., & Lv, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3543105 · Cited by 16
“In order to consider the effect of the image modality, existing works [15], [54] reinforce language representations by fusing visual information into textual features via the cross-modal attention mechanism [57].”
Conv-Enhanced Transformer and Robust Optimization Network for robust multimodal sentiment analysis
Sun, B., Jia, L., Cui, Y., Wang, N., & Jiang, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129842 · Cited by 12
TEMSA:Text enhanced modal representation learning for multimodal sentiment analysis
Chen, J., Song, S., Tan, Y., & Xia, H. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104391 · Cited by 13
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Yang, Q., Shi, Q., Wang, T., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02300 · Cited by 13
Hurley, N. C., Spatz, E. S., Krumholz, H. M., Jafari, R., & Mortazavi, B. J. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3417958 · Cited by 13
“The transformer architecture presents opportunities to further enhance model performance of time-series signals by allowing for additional flexibility to point the deep learning model at which portions of signals to focus on [78] that may not be properly aligned [188].”
BackgroundEmotion Dynamics Modeling via BERT
Yang, H., & Shen, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533860 · Cited by 12
“The aim is to identify correct emotions from multi-modal expressions [37]–[39].”
BackgroundSentiment Analysis of Fashion Related Posts in Social Media
Yuan, Y., & Lam, W. P. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498423 · Cited by 12
“In the vision text composition module, for the multimodal Transformer, we set the number of multi-head attention blocks as 2 followed by a self-attention layer which only contains one attention block.”
BackgroundMethodInfluentialMulti-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model
Zhang, Y., Rong, L., Li, X., & Chen, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_35 · Cited by 12
Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning
He, M., Du, W., Wen, Z., Du, Q., Xie, Y., & Wu, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3225549 · Cited by 12
“In order to improve video representation, most existing methods [21], [22], [23], [24], [28], [29], [30], [31] aggregate different modalities such as video and audio.”
MethodMultimodal interaction enhanced representation learning for video emotion recognition
Xia, X., Zhao, Y. H., & Jiang, D. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1086380 · Cited by 12
“It should also be mentioned that, the first five methods (LF-LSTM, LF-TRANS, EmoEmbs, MulT, and BIMHA) are based on hand-crafted features, where 142-dimensional audio features are extracted using the DisVoice toolkit (Vasquez-Correa et al., 2019), 35-dimensional visual features are extracted via the OpenFace toolkit (Ba…”
BackgroundMethodInfluentialGu, R., Wang, T., Deng, J., & Cheng, L. (2023) · Applied Sciences · doi:10.3390/app13074299 · Cited by 12
“[16] proposed MulT, which uses multiple crossmodal Transformers to fuse the three modal features of text, audio and vision.”
MethodWAY: Estimation of Vessel Destination in Worldwide AIS Trajectory
Kim, J. S., Park, H. J., Shin, W., Park, D. I., & Han, S. W. (2023) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3269729 · Cited by 14
Adapted Multimodal Bert with Layer-Wise Fusion for Sentiment Analysis
Chlapanis, O. S., Paraskevopoulos, G., & Potamianos, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094923 · Cited by 13
Li, X., Zhang, Z., Zhang, X., Wang, T., Li, Z., Yang, H., Çiftçi, U. A., Ji, Q., Cohn, J. F., & Yin, L. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3286838 · Cited by 12
“For example, MulT [54] first attempted to fuse multi-modalities by transformer, which was applied to image, audio, and text.”
MethodBPM_MT: Enhanced Backchannel Prediction Model using Multi-Task Learning
Jang, J. Y., Kim, S., Jung, M., Shin, S., & Gweon, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.277 · Cited by 12
Non-Uniform Attention Network for Multi-modal Sentiment Analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Cao, Q., & Chao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_48 · Cited by 12
Incorporating Stock Market Signals for Twitter Stance Detection
Conforti, C., Berndt, J., Pilehvar, M. T., Giannitsarou, C., Toxvaerd, F., & Collier, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.281 · Cited by 13
“Multimodal learning has proven successful for many NLP tasks (Tsai et al., 2019; Zadeh et al., 2020), including grounding (Beinborn et al., 2018), visual question answering (Ben-Younes et al., 2017; Yu et al., 2018), sentiment analysis (Rahman et al., 2020), and humor detection (Hasan et al., 2019).”
BackgroundUnbiased Missing-Modality Multimodal Learning
Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., & He, T. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02272 · Cited by 11
“We employ multimodal Transformers [46] to fuse the features from ˆ X miss ∪ X ava .”
Bimodal fusion network with multi-head attention for multimodal sentiment analysis
Zhang, R., Xue, C., Qi, Q., Lin, L., Zhang, J., & Zhang, L. (2023) · Applied Sciences · doi:10.3390/app13031915 · Cited by 11
Pan, Y., Mirheidari, B., Blackburn, D. J., & Christensen, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533363 · Cited by 13
“To address this, cross-attention has been proposed to balance modalities by using an attention mask to emphasise features from one modality in another before fusion [65], [66], [67].”
Multi-level fusion with fine-grained alignment for multimodal sentiment analysis
Li, X., Ma, Y., An, X., Xing, J., Liu, R. P., & Ren, Y. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00094-3 · Cited by 12
“The multimodal transformer designs the multi-head attention module in the Transformer framework to enable directed pairwise cross-modal attention (Tsai et al. 2019).”
Li, X., Dong, Z., Cheng, X., Miao, D., Zhang, H., Wang, T., Zhang, X., & Li, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129341 · Cited by 11
Multi-Level Contrastive Learning for Multimodal Sentiment Analysis
Yan, Z., Bai, W., Zhang, Y., Deng, J., Hu, Z., Zhang, X., & Ren, F. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3613116 · Cited by 16
“To effectively capture the multimodal interactions, we employ an architecture that incorporates multiple cross-attention blocks with layer normalization, a technique that is consistent with current state-of-the-art methods [4], [8].”
InfluentialHong, W. C., Shen, C., & Wu, P.-Y. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02233-x · Cited by 11
“Recently, many studies have utilized cross-attention mechanism for feature fusion, especially in multi-domain tasks, e.g. vision-to-text (Tsai et al., 2019; Hu & Singh, 2021) and vision-to-vision (Xu et al., 2022; Li et al., 2022).”
Capturing High-Level Semantic Correlations via Graph for Multimodal Sentiment Analysis
Qian, F., Han, J., Guan, Y., Song, W., & He, Y. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3359570 · Cited by 11
“The classical methods in MSA include tensor-based [4], [5], memory networks-based [6], [7], and cross transformer-based [8], [9] methods.”
MethodLenton‐Brym, A. P., Collins, A., Lane, J. E. M., Busso, C., Ouyang, J., Fitzpatrick, S., Kuo, J. R., & Monson, C. M. (2024) · British Journal of Clinical Psychology · doi:10.1111/bjc.12468 · Cited by 17
“For example, multimodal processing, which combines text, audio and optionally video data, has been shown to provide stronger ML performance on tasks such as sentiment analysis and emotion recognition (Sourav & Ouyang, 2021; Tsai et al., 2019), compared to approaches that focus on only one modality.”
CogAware: Cognition-Aware framework for sentiment analysis with textual representations
Zhang, Z., Wu, C., Chen, H., & Chen, H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112094 · Cited by 11
Prompt Learning for Multimodal Intent Recognition with Modal Alignment Perception
Chen, Y., Zhu, W., Yu, W., Xue, H., Fu, H., Lin, J., & Jiang, D. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10328-7 · Cited by 12
“The baselines used in this study are state-of-the-art multi-modal fusion methods, including MAG-BERT (Rahman et al. [11]) and MulT (Tsai et al. [9]).”
BTMuda: A Bi-Level Multisource Unsupervised Domain Adaptation Framework for Breast Cancer Diagnosis
Yang, Y., Zeng, X., Zeng, P., Yan, B., Wu, X., Zhou, J., & Wang, Y. (2024) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2024.3453401 · Cited by 12
“Inspired by multimodal networks [22], [23] that employ cross-attention for feature fusion, we propose to use the cross-attention strategy to build a fine-grained feature alignment process to generate a proper intermediate feature space among the source and target domains, thus relieving the domain…”
Deng, Y., Bian, J., Wu, S., Lai, J., & Xie, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102711 · Cited by 12
Robust Multimodal Representation under Uncertain Missing Modalities
Lan, G., Du, Y., & Yang, Z. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3702003 · Cited by 11
“Multi-modal Transformer (MulT) [42] utilized pairwise cross-modal attention to capture interactions between modalities, addressing the long-range dependencies between elements across modalities.”
Zhu, Z., Cheng, X., Chen, Z., Chen, Y., Zhang, Y., Wu, X., Zheng, Y., & Xing, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681623 · Cited by 13
Multimodal transformer for early alarm prediction
Strem, N., Dhami, D. S., Schmidt, B., & Kersting, K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109643 · Cited by 12
A Social Media Dataset and H-GNN-Based Contrastive Learning Scheme for Multimodal Sentiment Analysis
Peng, J., He, Y., Chang, Y., Lu, Y., Zhang, P., Ou, Z., & Yu, Q. (2025) · Applied Sciences · doi:10.3390/app15020636 · Cited by 11
“Tsai et al. [22] introduce a multimodal transformer to model unaligned language sequences, improving the integration of text and other modalities like images or audio.”
Huang, Z., Bai, Y., Liu, H., et al. (2025) · Applied Sciences · doi:10.3390/app15052305 · Cited by 11
Fahad, S. K. A., Zhengkui, D. W., Ng, P. C., Wong, N., Ng, A. B., & See, S. C. W. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-80334-5_19 · Cited by 13
Multi-grained feature gating fusion network for multimodal sentiment analysis
Yu, B., Li, C., & Shi, Z. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02446-x · Cited by 13
“For example, Tsai et al. [13] proposed a multimodal transformer (MulT), which effectively captures modality consistency through cross-modal attention mechanisms.”
MDFormer: Transformer-Based Multimodal Fusion for Robust Chest Disease Diagnosis
Liu, X., Pan, F., Song, H., Cao, S., Li, C., & Li, T. (2025) · Electronics · doi:10.3390/electronics14101926 · Cited by 14
Reward-Based Gradient Modulation for Multimodal Emotion Recognition With LoRA
Liu, F., Fu, Z., & Wang, Y. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3566373 · Cited by 11
Transformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 116
Sterpu, G., Saam, C., & Harte, N. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2480 · Cited by 11
“This concept has also been applied to emotion recognition [6, 7], or speech grounding from video [8].”
MethodMultimodal metaphor detection based on distinguishing concreteness
Su, C., Chen, W., Fu, Z., et al. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.11.051 · Cited by 11
Speech Emotion Recognition via Multi-Level Cross-Modal Distillation
Li, R., Zhao, J., & Jin, Q. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-785 · Cited by 11
“Inspired by [26, 27], we apply attention mechanism to learn the audio-to-text alignment via a soft-align network block, which consist three weight matrices: where σ denotes the GELU function, ˆ h sn denotes the soft aligned acoustic sequences, which has the same shape with the teacher’s word hidden…”
MethodDXM‐TransFuse U-net: Dual cross-modal transformer fusion U-net for automated nerve identification
Xie, B., Milam, G., Ning, B., et al. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102090 · Cited by 11
“Furthermore, a Transformer module that was inspired by Tsai et al. (2019) is implemented at the bottleneck for the fusion of different modalities.”
BackgroundMethodMulti-Modal Sentiment Analysis Based on Interactive Attention Mechanism
Wu, J., Zhu, T., Zheng, X., & Wang, C. (2022) · Applied Sciences · doi:10.3390/app12168174 · Cited by 13
Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text
Liu, J., Wumaier, A., Fan, C., & Guo, S. (2023) · Electronics · doi:10.3390/electronics12081775 · Cited by 17
Efficient Water Segmentation with Transformer and Knowledge Distillation for USVs
Zhang, J., Gao, J., Liang, J., Wu, Y., Li, B., Zhai, Y., & Li, X. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11050901 · Cited by 11
Image-text fusion transformer network for sarcasm detection
Liu, J., Tian, S., Yu, L., et al. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17252-2 · Cited by 11
“This allows for the fusion of relevant information across different modalities, enabling better utilization of the multimodal data [24].”
Nguyen, C.-D. T., Nguyen, T. T., Vu, D., & Luu, A. N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.980 · Cited by 15
Talemi, N. A., Kashiani, H., & Nasrabadi, N. M. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2023.3349218 · Cited by 11
“Moreover, results prove that among recent feature integration studies [72], [73], [74], our proposed module establishes the most effective integration approach for enhancing the performance of FR feature representation.”
ResultMulti-Modal Hierarchical Empathetic Framework for Social Robots With Affective Body Control
Gao, Y., Fu, Y., Sun, M., & Gao, F. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3356511 · Cited by 12
“Following [41], evaluation metrics are composed of binary accuracy and F1 score.”
BackgroundMethodInfluentialMulti-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos
Ma, L., Yu, Y., Liang, T., & Liu, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0351-0_21 · Cited by 11
Point-of-Interest Type Prediction using Text and Images
Villegas, D. S., & Αλέτρας, Ν. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.614 · Cited by 13
“To model cross-modal interactions, we adapt the cross-attention mechanism (Tsai et al., 2019; Tan and Bansal, 2019) to combine text and image information for multimodal POI type prediction (MM-XAtt).”
CtnR: Compress-then-Reconstruct Approach for Multimodal Abstractive Summarization
Zhang, C., Zhang, Z., Li, J., Liu, Q., & Zhu, H. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534082 · Cited by 11
“In multimodal text classification task, MulT [30] has been proved to be able to capture different modal information and MTN [31] has been verified in the fields of multimodal end-to-end video grounded dialogue systems.”
BackgroundText Is No More Enough! A Benchmark for Profile-Based Spoken Language Understanding
Xiao, X., Qin, L., Chen, K., Wu, G., Li, L., & Che, W. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21411 · Cited by 12
On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis
Ando, A., Masumura, R., Takashima, A., Suzuki, S., Makishima, N., Suzuki, K., Moriya, T., Ashihara, T., & Satō, H. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022548 · Cited by 12
Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)
(2020) · doi:10.18653/v1/2020.challengehml-1 · Cited by 11
“We use the same setup as (Tsai et al., 2019) with 4 emotions (happy, sad, angry, neutral).”
BackgroundMethodInfluentialResidual bayesian attention networks for uncertainty quantification in regression tasks
Chen, Y., Guan, W., & Azzam, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-24093-6 · Cited by 10
Research on Multimodal Fusion of Temporal Electronic Medical Records
Ma, M., Wang, M., Gao, B., Li, Y., Huang, J., & Chen, H. (2024) · Bioengineering · doi:10.3390/bioengineering11010094 · Cited by 11
“…method—fusion-convolutional neural network (Fusion-CNN) and fusion-LSTM [33]; • Transformer-based fusion models—Multimodal Transformer (MulT) [34], Crossformer [35], and patch time series transformer (PatchTST) [36]; • Models applied to time series EMR—Multivariate irregularly sampled time…”
MethodA Fine-Grained Tri-Modal Interaction Model for Multimodal Sentiment Analysis
Zhi, Y., Li, J., Wang, H., Chen, J., & Cao, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447872 · Cited by 11
“Feature-level fusion[4][5] learns all features into one vector and performs downstream tasks.”
BackgroundMethodInfluentialZhang, J., Kang, J., Zhang, K., Zhang, L., Liu, P., Liu, X., Sun, W., & Wang, G. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.212994 · Cited by 11
Multimodal Consistency-Based Teacher for Semi-Supervised Multimodal Sentiment Analysis
Yuan, Z., Fang, J., Xu, H., & Gao, K. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430543 · Cited by 14
“The Multimodal Transformer [54] extends transformer architecture fusion the source modality into the target modality using directional pairwise cross-attention mechanism.”
DiffusionEIT: Diffusion Model for Electrical Impedance Tomography
Liu, J., Shi, F., Xiong, H., & Zhou, Y. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3472911 · Cited by 11
“The transformer structure, which performs well in the NLP field, has been proven to be a good mechanism for multimodal information fusion [10], [11], [12], [13].”
BackgroundJing, Y., & Zhao, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681599 · Cited by 10
Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
Kim, J., Hong, J., & Choi, Y. (2024) · Applied Sciences · doi:10.3390/app142311397 · Cited by 12
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
