The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 201–300 of 1,004
Goncalves, L., & Busso, C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3216993 · Cited by 55
“Tsai et al. [17] proposed a multimodal transformer architecture for human language time-series data.”
BackgroundInfluentialMultimodal transformer network for incomplete image generation and diagnosis of Alzheimer’s disease
Gao, X., Shi, F., Shen, D. G., & Liu, M. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102303 · Cited by 52
A Comprehensive Survey of Recommender Systems Based on Deep Learning
Zhou, H., Xiong, F., & Chen, H. (2023) · Applied Sciences · doi:10.3390/app132011378 · Cited by 78
Xia, X., Guo, Y., Wang, Y., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123972 · Cited by 52
Wang, S., He, X., Jian, Z., Li, J., Xu, C., Chen, Y., Liu, Y., Chen, H., Huang, C., Hu, J., et al. (2024) · Eye and Vision · doi:10.1186/s40662-024-00405-1 · Cited by 59
“Feature-level fusion, or early fusion, combines modality features into a joint representation for decision-making [15, 16], while decision-level fusion, or late fusion, predicts results from unimodal features and then combines these results [17].”
BackgroundTransformer-Based Self-Supervised Learning for Emotion Recognition
Vazquez-Rodriguez, J., Lefebvre, G., Cumin, J., & Crowley, J. L. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956027 · Cited by 59
Wang, W., Wang, Y., Huang, Y., Mu, C., Sun, Z., Tong, X., & Cai, Z. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109206 · Cited by 51
Huan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 60
“Recent works have been dedicated specifically to unaligned multimodal sequences [27], [28], [29], [30], [31], [32], [33], [34].”
BackgroundMethodInfluentialMultimodal Variational Auto-encoder based Audio-Visual Segmentation
Mao, Y., Zhang, J., Xiang, M., Zhong, Y., & Dai, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00094 · Cited by 62
“Such as audio-visual similarity modeling by computing the correlation matrix [6, 7], audio-visual cross attention [12, 13, 17], audio-guided Grad-CAM [18], or using a multimodal transformer for modeling the long-range dependencies between elements across modalities directly [19, 20].”
MethodVideo multimodal sentiment analysis using cross-modal feature translation and dynamical propagation
Gan, C., Tang, Y., Fu, X., Zhu, Q., Jain, D. K., & García, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111982 · Cited by 52
Learning fine-grained representation with token-level alignment for multimodal sentiment analysis
Li, X., Zhang, H., Dong, Z., Cheng, X., Liu, Y., & Zhang, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126274 · Cited by 57
CIME: Contextual Interaction-Based Multimodal Emotion Analysis With Enhanced Semantic Information
Wang, R., Guo, C., Cambria, E., Rida, I., Yuan, H., Piran, M. J., Feng, Y., & Zhu, X. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3572495 · Cited by 63
“For instance, the MulT model [17] employs a nonaligned multimodal language sequence framework that leverages cross-modal self-attention mechanisms to capture long-range dependencies across modalities.”
Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
Hollenstein, N., Renggli, C., Glaus, B., Barrett, M., Tröndle, M., Langer, N., & Zhang, C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.659410 · Cited by 57
“Tsai et al. (2019) train a multi-modal sentiment analysis model from natural language, facial gestures, and acoustic behaviors.”
BackgroundDialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50
“The performance of MulT is limited because the model forces all the modalities to use context-dependent settings.”
BackgroundMethodInfluentialShared-Private Memory Networks For Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., & Tang, B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3222023 · Cited by 50
“In the perspective of multimodal fusion [11], [12], [13], [14], crossmodal attention [1], additive attention [4] and gating mechanism [15] are usually used to fuse multimodal information.”
BackgroundMethodResultInfluentialSurvey: Transformer based video-language pre-training
Ruan, L., & Jin, Q. (2022) · AI Open · doi:10.1016/j.aiopen.2022.01.001 · Cited by 56
Song, C., Ning, N., Zhang, Y., & Wu, B. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.077 · Cited by 51
Detecting dementia from speech and transcripts using transformers
Ilias, L., Askounis, D., & Psarras, J. E. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101485 · Cited by 55
“Next, we employ the crossmodal attention [73, 59, 64].”
Li, Z., Guo, Q., Pan, Y., Ding, W., Yu, J., Zhang, Y., Liu, W., Chen, H., Wang, H., & Xie, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101891 · Cited by 49
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis
Yu, Y., Zhao, M. M., Qi, S.-A., Sun, F., Wang, B., Guo, W., Wang, X. L., Yang, L., & Niu, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.860 · Cited by 58
Dual-Perspective Fusion Network for Aspect-Based Multimodal Sentiment Analysis
Wang, D., Tian, C., Xiao, L., Zhao, L., He, L., & Wang, Q. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321435 · Cited by 50
“For multimodal sentiment analysis tasks, recent approaches [22], [23], [24], [25], [26], [27], [28] use features from different modalities as transformer input for modality fusion.”
BackgroundMethodIncongruity-aware multimodal physiology signals fusion for emotion recognition
Li, J., Chen, N., Zhu, H., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102220 · Cited by 49
Heo, Y., Kim, E., Choi, W. S., & Zhang, B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.29 · Cited by 49
Zheng, W., Yu, J., Xia, R., & Wang, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.861 · Cited by 51
“4https://ffmpeg.org/ 5https://github.com/TadasBaltrusaitis/OpenFace A.2 Pseudo-code of MARIO We provide the pseudocode for training the proposed MARIO model, where θSwin, θT , θself−attn, and θCMT represent the parameters of SwinTransformer, the text encoder, self-attention Transformer, and Cross-Modal Transformer, re…”
MethodInfluentialChen, C., Gao, R., Calamia, P. T., & Grauman, K. L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01829 · Cited by 48
“Audio-visual attention [36, 57, 58] has been recently studied to capture the correlation between visual features and audio features.”
BackgroundLi, X., Liu, J., Xie, Y., Gong, P., Zhang, X., & He, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111126 · Cited by 49
Liu, H., Wei, R., Tu, G., Lin, J., Liu, C., & Jiang, D. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102353 · Cited by 50
Multi-modal Multi-label Emotion Detection with Modality and Label Dependence
Zhang, D., Ju, X., Li, J., Li, S., Zhu, Q., & Zhou, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.291 · Cited by 48
“( 9) MulT 10 (Tsai et al., 2019), which addresses long-range dependencies between elements across modalities in an end-to-end manner.”
BackgroundMethodInfluentialSelf-Supervised Learning with Cross-Modal Transformers for Emotion Recognition
Khare, A., Parthasarathy, S., & Sundaram, S. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383618 · Cited by 47
“We use the cross-modal transformer for emotion recognition since it showed state of the art results on sentiment analysis [15].”
InfluentialAdaptive Fusion Techniques for Multimodal Data
Sahu, G., & Vechtomova, O. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.275 · Cited by 60
“More recently, Multimodal Transformer (MulT) (Tsai et al., 2019) was proposed to align data from different modalities implicitly.”
BackgroundMethodMF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis
He, J., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3139856 · Cited by 48
Sentiment Analysis and Emotion Recognition from Speech Using Universal Speech Representations
Atmaja, B. T., & Sasou, A. (2022) · Sensors · doi:10.3390/s22176369 · Cited by 58
“Some references [9,21] did not provide the number of instances, whereas references [22,23] mentioned the use of 23,454 instances.”
BackgroundSemi-supervised Video Paragraph Grounding with Contrastive Encoder
Jiang, X., Xu, X., Zhang, J., Shen, F., Cao, Z., & Shen, H. T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00250 · Cited by 48
“Following that, a recent informal work [30] proposed a method that utilized Transformer [34] to tackle this task.”
BackgroundInfluentialYang, D., Liu, Y., Huang, C., Li, M., Zhao, X., Wang, Y., Yang, K., Wang, Y., Zhai, P., & Zhang, L. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110370 · Cited by 47
MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis
Li, S., Zhang, T., Chen, B., & Chen, C. L. P. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3259010 · Cited by 48
“If follows the extension method in work [20], it will lead to large growth in parameters and FLOPs as the number of modalities increases, which limits the generalization of the model.”
MethodInfluentialGubelmann, R. (2023) · Grazer Philosophische Studien · doi:10.1163/18756735-00000182 · Cited by 47
Learning discriminative multi-relation representations for multimodal sentiment analysis
Tang, Z., Xiao, Q., Zhou, X., Li, Y., Chen, C., & Li, K. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119125 · Cited by 50
FMSA-SC: A Fine-Grained Multimodal Sentiment Analysis Dataset Based on Stock Comment Videos
Song, L., Chen, S., Meng, Z., Sun, M., & Shang, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3363641 · Cited by 49
“Multimodal Transformer (MulT) [62]: It is an extended Trans-former network designed to learn representations from un-aligned multimodal streams.”
BackgroundLooking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation
Lee, J., Chung, S.-W., Kim, S., Kang, H.-G., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00139 · Cited by 47
“More related to our work are affinity-based multi-modal approaches in various other challenging tasks, such as music sound separation [44], emotion recognition [45], language understanding [46], and self-supervised learning [47].”
BackgroundA Package for Learning on Tabular and Text Data with Transformers
Gu, K., & Budhkar, A. (2021) · doi:10.18653/v1/2021.maiworkshop-1.10 · Cited by 54
A multimodal fusion method for sarcasm detection based on late fusion
Ding, N., Tian, S., & Yu, L. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12122-9 · Cited by 49
“In order to verify the comparison results between the deep learning models, we have added the experimental results of several models, they are: EF-LSTM [38], MFN [47], LMF [22], TFN [44] and MULT [36].”
ResultMALN: Multimodal Adversarial Learning Network for Conversational Emotion Recognition
Ren, M.-J., Huang, X., Liu, J., Liu, M., Li, X., & Liu, A. (2023) · IEEE transactions on circuits and systems for video technology (Print) · doi:10.1109/tcsvt.2023.3273577 · Cited by 50
Nguyen, C.-V. T., Mai, T., The, S., Kieu, H.-D., & Le, D.-T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.937 · Cited by 55
“Inspired by the idea (Tsai et al., 2019), we design the Pairwise Cross-modal Feature Interaction (P-CM) method into our proposed framework for conversation understanding.”
MethodCMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Zadeh, A. B., Cao, Y., Hessner, S., Liang, P. P., Poria, S., & Morency, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.141 · Cited by 46
“The Transformer hidden unit size is 40 with 4 cross-modal blocks and 10 attention heads.”
BackgroundMethodInfluentialSelf-citationVisual language navigation: a survey and open challenges
Park, S., & Kim, Y. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10174-9 · Cited by 49
“In particular, with the recent advent of the attention model, the use of transformers (Landi et al. 2021; Tsai et al. 2019) in the multi-modal domain is increasing as well as CNN (Joze et al.”
MethodA fine-grained modal label-based multi-stage network for multimodal sentiment analysis
Peng, J., Wu, T., Zhang, W., Cheng, F., Tan, S., Yi, F., & Huang, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119721 · Cited by 47
Tian, X., Bai, X., & Zhou, F. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3275991 · Cited by 48
“Then, multihead cross-modal attention is achieved by “Mul-tihead Attention” [42], i.e., where X H ∈ R N H × d H and X J ∈ R N J × d J denote feature sequences of the two modalities; N ( · ) and d ( · ) denote the sequence length and feature dimension, respectively; h is the number of heads; In…”
BackgroundHybrid cross-modal interaction learning for multimodal sentiment analysis
Fu, Y., Zhang, Z., Yang, R., & Yao, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127201 · Cited by 46
A Pre-Trained Audio-Visual Transformer for Emotion Recognition
Tran, M. T., & Soleymani, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747278 · Cited by 46
“We adapt the Multimodal Transformer (MulT) architecture [14] for the pretraining task.”
InfluentialAIA-Net: Adaptive Interactive Attention Network for Text–Audio Emotion Recognition
Zhang, T., Li, S., Chen, B., Yuan, H., & Chen, C. L. P. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3195739 · Cited by 47
“Compared to the state-of-the-art model MulT [22] and C-MIB [38], AIA-Net significantly outperforms them on neutral emotion, which further proves the effectiveness of AIA-Net in weak categories.”
BackgroundMethodResultInfluentialMultimodal Mutual Attention-Based Sentiment Analysis Framework Adapted to Complicated Contexts
He, L., Wang, Z., Wang, L., & Li, F. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3276075 · Cited by 45
“Tsai et al. [51] proposed a transformer-based multimodal fusion algorithm to map one modal to the other modal feature space using the attention between the two modal features.”
BackgroundMethodInfluentialA multimodal fusion emotion recognition method based on multitask learning and attention mechanism
Xie, J., Wang, J., Wang, Q., Yang, D., Gu, J., Tang, Y., & Varatnitski, Y. I. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126649 · Cited by 47
Fu, Z., Liu, F., Xu, Q., Fu, X., & Qi, J. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2444-y · Cited by 44
Yang, J., Xu, M., Xiao, Y., & Du, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127222 · Cited by 46
Weakening the Dominant Role of Text: CMOSI Dataset and Multimodal Semantic Enhancement Network
Jin, C., Luo, C., Yan, M. G., Zhao, G., Zhang, G., & Zhang, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3282953 · Cited by 44
Grounded Multimodal Named Entity Recognition on Social Media
Yu, J., Li, Z., Wang, J., & Xia, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.508 · Cited by 50
“Next, a Cross-Model Transformer layer (Tsai et al., 2019) is utilized to model the interaction between the text and visual representations as follows: H = CMT ( V , T , T ) , where V and T are regarded as queries and keys/values, and H = { h 1 , . . . , h K } is the generated hidden representation.”
MethodCo-space Representation Interaction Network for multimodal sentiment analysis
Shi, H., Pu, Y., Zhao, Z., Huang, J., Zhou, D., Xu, D., & Cao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111149 · Cited by 46
HRFuser: A Multi-Resolution Sensor Fusion Architecture for 2D Object Detection
Brödermann, T., Sakaridis, C., Dai, D., & Gool, L. V. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422432 · Cited by 46
“To adaptively fuse two modalities with each other, cross-attention [35] was introduced.”
MethodSee, hear, read: Leveraging multimodality with guided attention for abstractive text summarization
Atri, Y. K., Pramanick, S., Goyal, V., & Chakraborty, T. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107152 · Cited by 47
“As presented in Table 3, the MulT, and FMT multimodal baselines and the proposed FLORAL model beat most of the unimodal systems by a large margin, on both the datasets.”
MethodInfluentialGraph Capsule Aggregation for Unaligned Multimodal Sequences
Wu, J., Mai, S., & Hu, H. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479931 · Cited by 45
Zhang, Q., Shi, L., Liu, P., Zhu, Z., & Xu, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03343-4 · Cited by 43
“Multimodal languages mainly include Language (L), Visual (V), and acoustic(A) modalities.”
BackgroundMethodInfluentialRetractedA feature-based restoration dynamic interaction network for multimodal sentiment analysis
Zeng, Y., Li, Z., Chen, Z., & Ma, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107335 · Cited by 42
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Zhou, Q., Xu, H., Li, H., Zhang, H., Zhang, X., Wang, Y., & Gao, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29656 · Cited by 45
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 105
“This dataset emphasizes semantic reasoning and object localization, but the increased task complexity poses significant challenges for existing models in handling semantic inference and object interaction.”
AuxFormer: Robust Approach to Audiovisual Emotion Recognition
Goncalves, L., & Busso, C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747157 · Cited by 41
“Baseline 2 was derived from the architecture presented in Tsai et al. [8] which proposed a multimodal transformer architecture for human language time-series data.”
BackgroundMethodInfluentialSignNet II: A Transformer-Based Two-Way Sign Language Translation Model
Chaudhary, L., Ananthanarayana, T., Hoq, E., & Nwogu, I. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3232389 · Cited by 47
VLP2MSA: Expanding vision-language pre-training to multimodal sentiment analysis
Yi, G., Fan, C., Zhu, K., Lv, Z., Liang, S., Wen, Z., Pei, G., Li, T., & Tao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111136 · Cited by 43
Rate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data
He, Y., Yu, G., & Cai, Y. (2023) · IEEE Transactions on Communications · doi:10.1109/tcomm.2023.3335977 · Cited by 46
“We choose three existing multi-modal fusion models for the task of MSA: MulT [34] which utilizes the self-attention mechanism to learn cross-modality representations, ICCN [35] which uses the outer product between modalities for canonical correlation analysis, and MISA [36] which combines modality…”
MethodEMOE: Modality-Specific Enhanced Dynamic Emotion Experts
Fang, Y., Huang, W., Wan, G., Su, K., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01335 · Cited by 52
“Following the typical multimodal emotion recognition [14, 44], our goal is to detect sentiments in videos by leveraging multimodal signals.”
Multimodal and Multiresolution Speech Recognition with Transformers
Paraskevopoulos, G., Parthasarathy, S., Khare, A., & Sundaram, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.216 · Cited by 48
“Tsai et al. (2019) present Multimodal Transformers, relying on a deep pair-wise cascade of cross-modal attention mechanisms to map between modalities for multimodal sentiment analysis.”
BackgroundMethodLearning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition
Li, H., Ding, W., Wu, Z., & Liu, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-158 · Cited by 40
“At last, we replace our pre-calculated word-level binary alignment matrix A with the CTC predicting alignment matrix (w/ CTC Align) [31], we find the model’s performance suffers a great loss, which indicates that the pre-calculated alignment matrixA is more compatible with our proposed algorithm.”
Factorized Multimodal Transformer for Multimodal Sequential LearningPreprint
Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., & Morency, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09826 · Cited by 51
“We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (Zadeh et al., 2018b) on the left side of /, and negative vs. positive (Tsai et al., 2019) on the right side.”
BackgroundMethodResultInfluentialSelf-citationMemobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition
Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39
“MulT [4] is a transformer-based framework, which handles both non-aligned multi-modal sequences and long-range dependencies.”
BackgroundKim, D., & Kang, P. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.035 · Cited by 39
Learning speaker-independent multimodal representation for sentiment analysis
Wang, J., Wang, S., Lin, M., Xu, Z., & Guo, W. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.116 · Cited by 40
Learning from the global view: Supervised contrastive learning of multimodal representation
Mai, S., Zeng, Y., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101920 · Cited by 44
Versatile Audio-Visual Learning for Emotion Recognition
Goncalves, L., Leem, S.-G., Lin, W.-C., Şişman, B., & Busso, C. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3433386 · Cited by 51
TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107222 · Cited by 43
Multimodal Reconstruct and Align Net for Missing Modality Problem in Sentiment Analysis
Luo, W., Xu, M., & Lai, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-27818-1_34 · Cited by 39
Fcdnet: Fuzzy Cognition-Based Dynamic Fusion Network for Multimodal Sentiment Analysis
Liu, S., Luo, Z., & Fu, W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3407739 · Cited by 40
Triple disentangled representation learning for multimodal affective analysis
Zhou, Y., Liang, X., Chen, H., Zhao, Y., Chen, X., & Yu, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102663 · Cited by 41
NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis
Fu, Z., Liu, F., Xu, Q., Qi, J., Fu, X., Zhou, A., & Li, Z. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859836 · Cited by 37
“However, some works [21, 22, 20, 9, 7] consider that the inherent data is unaligned due to the variable sampling rates of audio and visual modalities, so they use multiple cross-modal attention to model unaligned multimodal sequences, typically MulT [8], PMR [7], and SPT [9].”
BackgroundCheng, X., Wei, W., Du, C., Qiu, S., Tian, S., Ma, X., & He, H. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548367 · Cited by 37
“To learn the inter-modality relations, we employ the cross-modal Transformer [27] based on cross-attention for multi-modality fusion, which only focuses on the inter-modality.”
MethodMulti-modal medical Transformers: A meta-analysis for medical image segmentation in oncology
Andrade-Miranda, G. X., Jaouen, V., Tankyevych, O., Rest, C. C. L., Visvikis, D., & Conze, P.-H. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102308 · Cited by 41
TCMT: Target-oriented Cross Modal Transformer for Multimodal Aspect-Based Sentiment Analysis
Zou, W., Sun, X., Wu, W., Lu, Q., Zhao, X., Bo, Q., & Yan, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125818 · Cited by 41
Yang, Z., Li, G., Xue, G., He, B., Song, Y., & Li, X. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112188 · Cited by 39
A knowledge-enhanced network for joint multimodal entity-relation extraction
Huang, S., Cai, Y., Yuan, L., & Wang, J. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104033 · Cited by 39
A Unimodal Reinforced Transformer With Time Squeeze Fusion for Multimodal Sentiment Analysis
He, J., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3078074 · Cited by 37
“Note that different from Multimodal Transformer (MULT) [14] (see Fig.”
BackgroundMethodInfluentialMultimodal Transformer for Property Prediction in Polymers
Han, S., Kang, Y., Park, H., Yi, J., Park, G., & Kim, J. (2024) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.4c01207 · Cited by 36
Token-disentangling Mutual Transformer for multimodal emotion recognition
Yin, G., Liu, Y., Liu, T., Zhang, H., Fang, F., Tang, C., & Jiang, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108348 · Cited by 43
Li, Z., Huang, Z., Pan, Y., Yu, J., Liu, W., Chen, H., Luo, Y., Wu, D., & Wang, H. H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124236 · Cited by 36
Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal TransformersPreprint
Yang, R., Zhang, M., Hansen, N., Xu, H., & Wang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03996 · Cited by 76
“We fuse the proprioceptive state and depth image inputs using Transformers (Vaswani et al., 2017; Tsai et al., 2019) for RL, which enables the model to reason with complementary information from both modalities.”
MethodGuideRender: large-scale scene navigation based on multi-modal view frustum movement prediction
Qin, Y., Chi, X., Sheng, B., et al. (2023) · The Visual Computer · doi:10.1007/s00371-023-02922-x · Cited by 35
“Therefore, we introduce the attention mechanism [30] and create an attentional guidance fusion (AGF) module, shown in Fig.”
MethodFine-Grained Disentangled Representation Learning For Multimodal Emotion Recognition
Sun, H., Zhao, S., Wang, X., Zeng, W., Chen, Y., & Qin, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447667 · Cited by 37
“Second, researchers have been committed to exploring the correlations between different modal information to obtain more robust modal semantics and, consequently, improve the accuracy of emotion recognition tasks [7, 8, 9].”
BackgroundInteract before Align: Leveraging Cross-Modal Knowledge for Domain Adaptive Action Recognition
Yang, L., Huang, Y., Sugano, Y., & Sato, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01431 · Cited by 36
“For solving the domain gap problem which exists widely in various applications such as object recognition [13, 41, 49], image segmentation [3, 14, 17, 60, 64], and natural language understanding [39, 44, 51], domain adaptation has been extensively studied especially in recent years.”
BackgroundA Multi-Stage Hierarchical Relational Graph Neural Network for Multimodal Sentiment Analysis
Gong, P., Liu, J., Zhang, X., & Li, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096644 · Cited by 36
“Tsai et al. [4] proposed a cross-modal Transformer to learn the directional attention between elements across modalities.”
BackgroundAdaMoW: Multimodal Sentiment Analysis Based on Adaptive Modality-Specific Weight Fusion Network
Zhang, J., Wu, X., & Huang, C. (2023) · IEEE Access · doi:10.1109/access.2023.3276932 · Cited by 36
Layer-wise Fusion with Modality Independence Modeling for Multi-modal Emotion Recognition
Sun, J., Han, S., Ruan, Y.-P., Zhang, X., Zheng, S., Liu, Y., Huang, Y., & Li, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.39 · Cited by 38
“Multi-modal transformer (MulT) is proposed in(Tsai et al., 2019), which adopts cross-modal attention to fuse any pair of modalities, and then incorporates all the information.”
BackgroundAdapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34
FDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction
Fu, Y., Huang, B., Wen, Y., & Zhang, P. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111965 · Cited by 35
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
