The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 801–900 of 1,004
MT-TCCT: Multi-task Learning for Multimodal Emotion Recognition
Wang, Y., Chen, Z., Chen, S., & Zhu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15934-3_36 · Cited by 8
Korean Drama Scene Transcript Dataset for Emotion Recognition in Conversations
Pant, S., Lim, E., Yang, H.-J., Lee, G.-S., Kim, S.-H., Kang, Y.-S., & 장혜림 (2022) · IEEE Access · doi:10.1109/access.2022.3221408 · Cited by 9
Sensor Fusion and Multimodal Learning for Robotic Grasp Verification Using Neural Networks
Gohil, P., Thoduka, S., & Plöger, P. G. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9955646 · Cited by 8
“Cross-modal representation learning aims to minimize cross-modal similarity distance or modality reconstruction loss for all the input modalities [5]; where, all modalities are present during training but some of the modalities are absent during testing [15], [16].”
BackgroundCross-modal dynamic sentiment annotation for speech sentiment analysis
Chen, J., Sun, C., Zhang, S., & Zeng, J. (2023) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.108598 · Cited by 8
Towards Temporal Event Detection: A Dataset, Benchmarks and Challenges
Yang, Z., Yang, Z., Guo, Z., Lin, Z., Zhu, H., Li, Q., & Liu, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3276523 · Cited by 9
“For example, Yao et al. [53] modeled interactions between multimodal sequences across distinct time steps, and latently adapted streams from one modality to another.”
BackgroundAdaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Prediction
Nguyen, T., Wu, X., Luu, A. T., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.686 · Cited by 8
A Comprehensive Survey on Multi-Modal Conversational Emotion Recognition with Deep LearningPreprint
Shou, Y., Meng, T., Ai, W., Yin, N., & Li, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5017731 · Cited by 8
CAFA: Cross-Modal Attentive Feature Alignment for Cross-Domain Urban Scene Segmentation
Liu, P., Ge, Y., Duan, L., et al. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3412006 · Cited by 7
“The cross-modal attention (CMA) [31] mechanism enhances one modality a’s feature M a ∈ R l a × d a with the features M b ∈ R l b × d b from another modality b by learning the attention across the two modalities’ features.”
BackgroundRelevance-aware visual entity filter network for multimodal aspect-based sentiment analysis
Chen, Y., Xiong, H., Li, K., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02342-w · Cited by 7
“At the low-level interaction stage, we are inspired by the Cross-Modal Transformer (CMT) layer [32] to design the Aspect-Oriented Transformer (AOT) layer, which is intended to facilitate interaction between the aspect and texts (or images) for the production of aspect-oriented text/image…”
Chen, Q., Xie, S., Fang, X., & Sun, Q. (2024) · The Visual Computer · doi:10.1007/s00371-024-03668-w · Cited by 7
“Different from their studies, we propose a text-centered modal translation network that eliminates the inherent noise in unimodal features To learn intra-modal relationships, some works [9, 17–19] directly concatenate or mix different representations for sentiment prediction.”
APT: Alarm Prediction Transformer
Strem, N., Dhami, D. S., Schmidt, B., Klöpper, B., & Kersting, K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125521 · Cited by 7
CMDAF: Cross-Modality Dual-Attention Fusion Network for Multimodal Sentiment Analysis
Guo, W., Su, K., Jiang, B., Xie, K., & Liu, J. (2024) · Applied Sciences · doi:10.3390/app142412025 · Cited by 10
MulDeF: A Model-Agnostic Debiasing Framework for Robust Multimodal Sentiment Analysis
Huan, R., Zhong, G., Chen, P., & Liang, R. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521836 · Cited by 8
“Since the introduction of transformers [12], studies have explored the possibility of end-to-end sentiment analysis of un-aligned multimodal sequences using attention mechanisms [13], [14], [15].”
Yang, Y., Yang, Y.-Q., Ren, G., & Yu, B. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113164 · Cited by 7
TGFN-SD: A text-guided multimodal fusion network for swine disease diagnosis
Yang, G., Li, Q., Zhao, C., Wang, C., Yan, H., Meng, R., Liu, Y., & Yu, L. (2025) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2025.03.002 · Cited by 9
Emotion-Assisted multi-modal Personality Recognition using adversarial Contrastive learning
Bao, Y., Wang, Y., Qi, Y., et al. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113504 · Cited by 7
HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis
Yang, C., Liang, Z., Yan, D., Hu, Z., & Wu, T. (2025) · IEEE Access · doi:10.1109/access.2025.3560641 · Cited by 7
“Tsai et al. [12] introduces a model known as the Multimodal Transformer (MulT), which fuses information through a cross-modal attention mechanism, making it suitable for unaligned data streams.”
CIEG-Net: Context Information Enhanced Gated Network for multimodal sentiment analysis
Chen, Z., Lu, C., & Wang, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111785 · Cited by 8
Multimodal disentanglement implicit distillation for speech emotion recognition
Qi, X., Wen, Y., Gong, J., Zhang, P., & Zheng, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104213 · Cited by 7
Dual-Branch Network with Hybrid Attention for Multimodal Ophthalmic Diagnosis
Wang, X., Cao, A., Fan, C., Tan, Z., & Wang, Y. (2025) · Bioengineering · doi:10.3390/bioengineering12060565 · Cited by 8
“…learning, the attention mechanism enables deep learning models to extract and fuse feature information from different modalities more efficiently [35]; therefore, for the diagnostic classification of small datasets with category imbalance, a suitable combination of the attention mechanism and a…”
SSMIM: Semi-supervised Multimodal Interaction Model for real-time conversational emotion recognition
Liu, H., & Zhong, K. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07373-w · Cited by 9
“Building on this idea, Tsal et al. [41] enhanced recognition by applying frame-level attention-based alignment methods to better capture nuanced emotional cues from both speech and text channels.”
He, X., Pan, Y., Guo, X., Xu, Z., & Yang, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3580779 · Cited by 8
“For example, Tsai et al. [22] introduced a cross-modal Transformer that enhances the target modal through cross-modal attention, generating unified representations early in the learning process.”
DAM-Faster RCNN: few-shot defect detection method for wood based on dual attention mechanism
Tong, X., Liang, Z., Qin, M., Liu, F., Yang, J., Xiao, H., & Dai, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-05479-y · Cited by 7
Yang, C., Liang, Z., Liu, T., Hu, Z., & Yan, D. (2025) · Electronics · doi:10.3390/electronics14153088 · Cited by 8
CAETFN: Context Adaptively Enhanced Text-Guided Fusion Network for Multimodal Sentiment Analysis
Li, J., Liu, R., Miao, Q., Wang, D., & Liu, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3590246 · Cited by 9
“…two clas-sification ways are calculated according to the prior works, in SectionV,the“-/-”markerinthetablesisleveragedtodemarcate thedistinctclassificationstrategies,wheretheleftvaluecorre-spondstonegative/non-negativeclassification[7],andtheright valuecorrespondstonegative/positiveclassification[78].”
Chen, L., & Fan, X. (2025) · PLoS ONE · doi:10.1371/journal.pone.0333379 · Cited by 8
Sun, N., Zhang, W., Zheng, W., Liu, J., Chai, L., & Wu, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3612369 · Cited by 7
“In addition to the eight baseline methods [9], [17], [18], [19], [20], [21], [24], [25] introduced in [35], we also include DMD , EMT-DLFR , NIAT , CRNet , Meta-NA , TSCL-FHFN , and TSST .”
MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection
Nezakati, N., Reza, M. K., Patil, A. D., Solh, M., & Asif, M. S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401652 · Cited by 7
“CMU-MOSI dataset: For multimodal sentiment analysis, we use multimodal transformer (Tsai et al. 2019) from this publicly available code 5 , and use the default configurations from Yu et al. (2021).”
InfluentialMilam, G., Xie, B., Liu, R., et al. (2022) · Sensors · doi:10.3390/s22207701 · Cited by 7
“[14] explored leveraging MHA mechanism to reinforce a target modality with features from another data modality via learning the cross-modal attention.”
Background天津师范大学, 张, 敬., 李, 彦., & 天津师范大学 (2023) · 计算机科学 · doi:10.55375/cps.2023.3.2 · Cited by 7
Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions
Wörtwein, T., Sheeber, L., Allen, N. B., Cohn, J. F., & Morency, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.344 · Cited by 7
“While multimodal models have led to performance improvements (Zadeh et al., 2017; Tsai et al., 2019; Zellers et al., 2021), they often have the downside of being difficult to interpret: it is unclear whether interactions between two modalities (bimodal) or three modalities (trimodal) are learned,…”
BackgroundSelf-citationVisual-Haptic-Kinesthetic Object Recognition with Multimodal Transformer
Zhou, X., Lan, S., Wang, W., Li, X., Zhou, S., & Yang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44195-0_20 · Cited by 7
Lin, W.-C., Goncalves, L., & Busso, C. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614110 · Cited by 7
“In this study, we adopt the SOTA cross-modal attention model [12, 45] as the backbone architecture, proposing a transformer framework based on the word-chunk concept for audio-text emotion recognition.”
BackgroundMethodInfluentialCANAMRF: An Attention-Based Model for Multimodal Depression Detection
Wei, Y., Zhang, Y., Zhang, S., & Zhang, H. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7022-3_10 · Cited by 10
Wang, P., Liu, S., & Chen, J. (2024) · Applied Sciences · doi:10.3390/app14051934 · Cited by 8
Circular Decomposition and Cross-Modal Recombination for Multimodal Sentiment Analysis
Liang, H., Xie, W., He, X., Song, S., & Shen, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446166 · Cited by 7
“Most prior studies have concentrated on the development of sophisticated fusion networks for effectively harnessing diverse modal information, typically classified as early fusion [4, 5, 2] and late fusion [6, 7, 8].”
BackgroundMethodTransferring Multi-Modal Domain Knowledge to Uni-Modal Domain for Urban Scene Segmentation
Liu, P., Ge, Y., Duan, L., Li, W., Luo, H., & Lv, F. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3382880 · Cited by 8
“The cross-modal attention [42] mechanism enhances one modality a’s feature M a ∈ R l a × d a with the features M b ∈ l b × d b from another modality b by learning the attention across the two modalities’ features.”
BackgroundMethodSC-Net: Multimodal metaphor detection using semantic conflicts
He, X., Yu, L., Tian, S.-W., Yang, Q.-M., & Long, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127825 · Cited by 7
Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture
Song, Y., & Zhou, Q. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2356992 · Cited by 9
“It incorporates a multi-head attention mechanism module (Tsai et al. 2019), a residual connection and a feedforward layer.”
Yang, S., Cai, J., & Liao, L. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00858-9 · Cited by 7
“MulT (Tsai et al., 2019a) It examines the interactions between multimodal sequences at different time steps, integrating multimodal information by focusing directly on low-level features in other modalities.”
Baek, D., Kim, J., & Lee, H.-C. (2024) · IEEE Access · doi:10.1109/access.2024.3447737 · Cited by 9
“The CMA operation employed here utilizes multiple heads to learn and model relationships between different modalities, such as text, video, and audio features [35].”
MethodPerceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers’ Opinion Scores
Goncalves, L., Mathur, P., Lavania, C., Cekic, M., Federico, M., & Han, K. J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72986-7_17 · Cited by 8
Mu, G., Chen, C., Li, X., Li, J., Ju, X., & Dai, J. (2024) · IEEE Access · doi:10.1109/access.2024.3493933 · Cited by 7
A text-speech multimodal Chinese named entity recognition model for crop diseases and pests
Liu, R., Guo, X., Zhu, H., & Wang, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-88874-9 · Cited by 8
IUCL at WASSA 2022 Shared Task: A Text-only Approach to Empathy and Emotion Detection
Chen, Y., Ju, Y., & Kübler, S. (2022) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2022.wassa-1.21 · Cited by 8
“Categorical and numerical features can be treated as additional tokens, or they can be treated as a different modality and handled by co-attention (Tsai et al., 2019).”
BackgroundWhose emotion matters? Speaking activity localisation without prior knowledge
Carneiro, H., Weber, C., & Wermter, S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126271 · Cited by 7
“This approach uses crossmodal transformers (CTs) [38] to enrich the information from one modality by taking into account information from another modality, and this way learn existing correlated information across pairs of modalities.”
MethodSelf-supervised Cross-modal Pretraining for Speech Emotion Recognition and Sentiment Analysis
Chu, I., Chen, Z., Yu, X., Han, M., Xiao, J., & Chang, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.375 · Cited by 7
Shi, W., Zhang, J., & He, S. (2023) · Kybernetes · doi:10.1108/k-04-2023-0723 · Cited by 7
S2E: Towards an End-to-End Entity Resolution Solution from Acoustic Signal
Ruan, K., He, X., Wang, J., Zhou, X., Feng, H., & Kebarighotbi, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446126 · Cited by 7
“…we have two distinct multi-head attention where in one the acoustic embedding serves as the query, utilizing the entity embeddings as both the key and value, and in another, the entity embeddings become the query, and the acoustic embedding become both the key and the value, inspired by [14].”
BackgroundBranch-Fusion-Net for Multi-Modal Continuous Dimensional Emotion Recognition
Li, C., Xie, L., & Pan, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3160373 · Cited by 8
How to Leverage Multimodal EHR Data for Better Medical Predictions?Preprint
Yang, B., & Wu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15763 · Cited by 7
“The second one is attention fusion (Tsai et al., 2019).”
Fu, Y., Fu, J., Xue, H., & Xu, Z. (2024) · Electronics · doi:10.3390/electronics13142835 · Cited by 6
Explainable Multimodal Fusion for Dementia Detection From Text and Speech
Altinok, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70566-3_21 · Cited by 6
Hierarchical Knowledge Stripping for Multimodal Sentiment Analysis
Xiong, A., Zeng, Y., & Hu, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3456117 · Cited by 8
“(cid:2) Attention-based fusion models which utilize attention and transformer modules to improve token representations us-ing non-verbal signals: MulT [70], MAG-BERT [47].”
Querying as Prompt: Parameter-Efficient Learning for Multimodal Language Model
Liang, T., Huang, J., Kong, M. C., Chen, L., & Zhu, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02536 · Cited by 7
Nguyen, D., de Voil, P. G., Potgieter, A. B., Dang, Y. P., Orton, T. G., Nguyen, T. T., Nguyen, T. T., & Chapman, S. C. (2024) · Agricultural Water Management · doi:10.1016/j.agwat.2024.109124 · Cited by 6
MECG: modality-enhanced convolutional graph for unbalanced multimodal representations
Tang, J., Ni, B., Yang, Y., Ding, Y., & Kong, W. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06729-y · Cited by 6
“• MULT [10]: It leverages the cross-modal attention mechanism, allowing stacked transformer networks to mitigate the time alignment problems of multimodal signals.”
InfluentialHu, Z., Yu, Z., & Guo, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01697-6 · Cited by 6
“Tsai et al. [14] proposed the Multi-modal Transformer model, which enables the model to adjust the cross-modal attention module and facilitates the transfer of information between modalities without the need for alignment, effectively addressing the issue of data misalignment in multimodal time…”
Wang, X., Wang, L., Yu, Y., & Jiao, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888606 · Cited by 8
Multi-level Conflict-Aware Network for Multi-modal Sentiment Analysis
Gao, Y., Wu, H., & Zhang, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888597 · Cited by 7
“In recent years, multimodal sentiment analysis (MSA) has attracted increasingly widespread attention [1, 2, 3, 4].”
Exploring EEG and eye movement fusion for multi-class target RSVP-BCI
Li, X., Wei, W., Zhao, K., Mao, J., Lu, Y., Qiu, S., & He, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103135 · Cited by 7
Cai, J., Liu, W., Liu, T., et al. (2025) · Animals · doi:10.3390/ani15071067 · Cited by 6
“Model Image Audio P /% R /% F1 ‐ Score /% FasterR-CNN √ 83.21 79.83 81.47 Advanced multimodal models, including MulT [38] (F1-score: 94.23”
InfluentialWang, M., Xie, L., Li, C., Wang, X., Sun, M., & Liu, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127631 · Cited by 8
Chen, M., Liu, J., Li, X., Zhang, Y., Liu, H., Jiao, J., & He, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3569089 · Cited by 7
“Mult [14]: The Multimodal Transformer (MulT) introduces cross-modal attention, enabling the model to focus on relevant information from one modality while processing another, thus enhancing feature fusion across modalities.”
Multimodal fusion for rumor sleuthing: A comprehensive approach
Farahi, M.-R., & Jafarinejad, F. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128327 · Cited by 6
Harmonizer: A Universal Signal Tokenization Framework for Multimodal Large Language Models
Amiri, A., Ghaffarnia, A., Nia, N. G., Wu, D., & Liang, Y. (2025) · Mathematics · doi:10.3390/math13111819 · Cited by 9
Wardle, G., & Sušnjak, T. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9060149 · Cited by 10
A method for multimodal sentiment analysis: adaptive interaction and multi-scale fusion
Wang, H., Cao, J., Liu, J., & Song, J. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00957-1 · Cited by 6
“Following prior works (Tsai et al., 2019; Zhang et al., 2023; Shi et al., 2024), the evaluation protocol employed distinct accuracy metrics across different datasets to align with their respective annotation schemes.”
InfluentialFoundation models for exploration geophysics
Liu, Q., Chen, Z., & Ma, J. (2025) · Geophysics · doi:10.1190/geo2024-0916.1 · Cited by 9
Aspect-level multimodal sentiment analysis model based on multi-scale feature extraction
Miao, B., & Xu, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-16051-z · Cited by 6
Zhu, X., & Jiang, S. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00975-z · Cited by 7
“Tsai et al. (2019) introduced the multimodal trans-former, which combines the basic modules of the transformer fusion method with a multi-head attention mechanism to achieve Cross-Modality information fusion by using different modalities as query, value, and key.”
Deepfake Detection: A Multimodal Survey
Wang, M. (2025) · ITM Web of Conferences · doi:10.1051/itmconf/20257802027 · Cited by 7
Xi, Z., Yu, B., Zhao, S., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103838 · Cited by 6
He, J., Shi, X., Hu, C.-H., Mi, J., Li, X., & Toda, T. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3614428 · Cited by 7
He, X., Pan, Y., Xu, Z., Li, Z., Guo, X., & Yang, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3614159 · Cited by 7
“In [22], an innovative meta-learning-based paradigm was proposed, retaining the strengths .”
Lin, R., Xiong, A., He, Q., Mai, S., Huang, L., & Hu, H. (2026) · Machine Intelligence Research · doi:10.1007/s11633-025-1630-x · Cited by 6
Grifoni, P., D’Ulizia, A., & Ferri, F. (2021) · IEEE Access · doi:10.1109/access.2021.3061756 · Cited by 6
Transformer-Based Multimodal Infusion Dialogue Systems
Liu, B., He, L., Liu, Y., et al. (2022) · Electronics · doi:10.3390/electronics11203409 · Cited by 6
Local Contrastive Learning for Medical Image Recognition
Rizvi, S. A., Tang, R., Jiang, X., Ma, X., & Hu, X. (2023) · PubMed Central · doi:10.48550/arxiv.2303.14153 · Cited by 9
Zhang, L., Liu, C., & Jia, N. (2023) · Applied Sciences · doi:10.3390/app13179910 · Cited by 6
PSAM: Parameter-Free Spatiotemporal Attention Mechanism for Video Question Answering
Zhang, F., Wang, R., Zhou, F., Luo, Y., & Li, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3333192 · Cited by 6
“With the excellent long-term dependency modelling ability of the self-attention mechanism, improved cross-attention mechanisms based on the self-attention mechanism have also been introduced to the field of VideoQA [17], [35].”
Enhancing Speech Emotion Recognition for Real-World Applications via ASR Integration
Li, Y. (2023) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw59127.2023.10388136 · Cited by 6
“[24] proposed a cross-attention based Transformer to provide latent adaptation that fuses multimodal information by directly attending to features in another modality.”
Semantic Interaction Fusion Framework for Multimodal Sentiment Recognition
Yang, S., Cui, L., & Wang, T. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394433 · Cited by 6
“Multimodal sentiment recognition is confronted with a considerable challenge to extracting valuable semantic information and mining the complementary relationship due to the heterogeneity between the different modalities [3, 21].”
BackgroundMultimodal Emotion Fusion Mechanism and Empathetic Responses in Companion Robots
Liu, X., Lv, Q., Li, J., Song, S., & Cangelosi, A. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3442203 · Cited by 6
“…low-rank multimodal fusion (LMF) [59], tensor fusion network (TFN) [60], multitask tensor fusion network (MTFN) [47], multimodal transformer (MulT) [61], self-supervised multitask learning (Self-MM) [62], multimodal fusion network (MSA-AFN) [63], and fine-grained modal label-based multistage…”
InfluentialLi, Y., Weng, W., Liu, C., & Lin, L. (2024) · Pattern Analysis and Applications · doi:10.1007/s10044-024-01320-w · Cited by 6
Mutual Information of Crossmodal Utterance Representation for Multimodal Sentiment Analysis
Xufei, Yue, D., & Wei, X. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3466968 · Cited by 9
“MulT [29]: Multimodal Transformer solves the problem of data alignment in an end-to-end approach, without the need for explicit alignment of data.”
FGITrans: Cross-City Transformer for Fine-grained Urban Flow Inference
Zheng, Y., Cai, Y., Cai, Z., Fan, C., Wang, S., & Wang, J. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679855 · Cited by 8
“With the success of transformer architecture in natural language processing (NLP) [14–17] and computer vision tasks [18–21], cross-attention in transformer has been proved to work quite well at aligning different distributions, even from different data modalities [22, 23].”
Kumar, D., Madan, S., Singh, P., Dhall, A., & Raman, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3688986 · Cited by 12
Sun, C., Chen, J., Zhao, Y., Han, H., Jing, R., Tan, G., & Wu, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125903 · Cited by 6
Privacy-Preserving Multimodal Sentiment Analysis
Xu, H., Li, W., Takabi, D., Seo, D., & Cai, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3527864 · Cited by 7
“The second one is calculated based on negative/positive ( Neg/Pos ) classes, where the negative and the positive classes are indicated by the negative and the positive scores, respectively [73].”
Cross-Attention Fusion of Visual and Geometric Features for Large-Vocabulary Arabic Lipreading
Daou, S., Hamadou, A. B., Rekik, A., & Kallel, A. (2025) · Technologies · doi:10.3390/technologies13010026 · Cited by 9
Lin, W. W., Li, T., Dang, J., et al. (2025) · Physics of Fluids · doi:10.1063/5.0264450 · Cited by 6
Manifold knowledge-guided feature fusion network for multimodal sentiment analysis
Wang, X., Wang, M., Cui, H. N., & Zhang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127537 · Cited by 6
Wang, C., Zhang, Q., Dong, J., Fang, H., Schaefer, G., Liu, R., & Yi, P. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113638 · Cited by 6
CLIP-driven attention network for multimodal sentiment analysis
Lv, J., Yang, Q., Tian, S., Liu, B., & Yu, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07411-7 · Cited by 6
“To verify the effectiveness of our proposed method, we selected baseline models encompassing classical and recent competitive approaches, including TFN [46], MulT [47], MISA [28], MMIM [48], and MSAN [49].”
Enhancing Sentiment Analysis Through Multimodal Fusion: A BERT-DINOv2 Approach
Zhao, T., Li, M., Chen, K., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97635-3_25 · Cited by 6
Gao, H., Cai, Z., Wang, X., Wu, M., & Liu, C. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3597398 · Cited by 8
“These settings follow configura-tions used in prior multimodal learning works [12], [22], [35], [36], [37].”
Multimodal Weighted Fusion of Transformers for Movie Genre Classification
Bribiesca, I. R., López-Monroy, A. P., & Montes-y-Gómez, M. (2021) · doi:10.18653/v1/2021.maiworkshop-1.1 · Cited by 6
“Some examples include ViLBERT (Lu et al., 2019), MulT (Tsai et al., 2019), VisualBERT (Li et al., 2019), UNITER (Chen et al., 2020), MMBT (Kiela et al., 2019) and LXMERT (Tan and Bansal, 2019).”
MethodInfluentialKnowing Where and What to Write in Automated Live Video Comments: A Unified Multi-Task Approach
Wu, H., Jones, G. J. F., & Pitié, F. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479942 · Cited by 6
“Other older neural architectures such as LSTM are not included in this study, since it is well-established that Transformers are the method of choice for modelling multi-modal signals [8, 13, 19, 22].”
Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data
Kang, Y., Liu, T., Li, H., Hao, Y., & Ding, W. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21334 · Cited by 6
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
