The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 901–1,000 of 1,004
Deep Learning Approaches on Multimodal Sentiment Analysis
Cai, Z., Gao, H., Li, J., & Wang, X. (2022) · IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA) · doi:10.1109/eebda53927.2022.9745018 · Cited by 6
“The result shows that the new method provides a much better performance than most of the previous methods such as EF_LSTM, LF_LSTM, TFN [15], MFN [16], Graph-MFN [14], RMFN [17] and has a slightly higher accuracy than the most recent state-of-the-art method MulT [18].”
ResultLearning modality-fused representation based on transformer for emotion analysis
Shi, P., Hu, M. Q., Ren, F., Shi, X., & Xu, L. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.6.063032 · Cited by 7
“MulT 15 uses a transformer-based model, without the encoder-decoder structure, to capture the directional pairwise crossmodal attention from one modality to another.”
MethodInfluentialGe, C., Xu, J., Hu, J., Liu, P., Tang, R., Wang, J., & Ren, H. (2023) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda56900.2023.10154636 · Cited by 6
“[24] proposed a multimodal transformer for modelling unaligned multimodal language sequences, which attends to interactions between multimodal sequences across C”
BackgroundChen, J., Zhang, S., Li, Z., & Jin, X. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191248 · Cited by 6
“Fα and Fβ are embedded with position in the same way of [20] [22], i.”
BackgroundMethodTopic and Style-aware Transformer for Multimodal Emotion Recognition
Qiu, S., Sekhar, N., & Singhal, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.130 · Cited by 8
“Specific to the area of sentiment analysis, Multimodal Transformer applies pairwise cross-attention to different modalities (Tsai et al., 2019).”
MethodMultiPar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations
Lee, D. W., Kim, Y., Picard, R. W., Breazeal, C., & Park, H. W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/433 · Cited by 6
Building robust Korean speech recognition model by fine-tuning large pretrained model*
Oh, C., Kim, C., & Park, K. (2023) · Phonetics and Speech Sciences · doi:10.13064/ksss.2023.15.3.075 · Cited by 7
“대 표적인 모델로 wav2vec 2.0 과 Hubert 등의 모델이 있으며 , 이렇 게 사전 훈련된 모델은 매주 작은 규모의 음성 데이터로 파인튜 닝된 경우에도 효과적인 성능을 보여주었다 (Baevski et al., 2020; Schneider et al., 2019; Tsai et al., 2019 즉 , 단어 오류 비율을 계산하는 것이다 .”
InfluentialSelf-adaptive attention fusion for multimodal aspect-based sentiment analysis
Wang, Z., & Guo, J. (2023) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2024056 · Cited by 7
Wen, Y., Wan, Z., Zhou, M. Q., Hou, S., Cao, Z., Le, C., Chen, J., Tian, Z., Zhang, W., & Wang, Y.-X. (2023) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150026 · Cited by 15
Lu, C., & Fu, X. (2024) · IEEE Access · doi:10.1109/access.2024.3363028 · Cited by 7
“Tsai et al. [17] introduced a Multimodal Transformer (MulT) and explored cross-modal attention between paired modalities (e.g., t with a ).”
BackgroundND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model
Wang, Q., Fan, C., Jia, T., Han, Y., & Wu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29486 · Cited by 6
“The ND-MRM effectively captures interactions between different senses through neuronal diversity inspiration, demonstrating superiority over the pairwise cross-modal attention approach employed by MulT .”
BackgroundInfluentialFang, C., Liang, F., Li, T., & Guan, F. (2024) · Future Internet · doi:10.3390/fi16060213 · Cited by 6
From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation
Agarwal, D., Agrawal, T., Ferrari, L. M., & Brémond, F. (2021) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss52988.2021.9663793 · Cited by 6
“296 %), performed with state-of-the-art architecture [22], by approximately 3% in accuracy and 4 points in F1score.”
BackgroundMethodInfluentialAffordance embeddings for situated language understanding
Krishnaswamy, N., & Pustejovsky, J. D. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.774752 · Cited by 6
Unleashing Knowledge Potential of Source Hypothesis for Source-Free Domain Adaptation
Hu, B., Liu, J., Zheng, K., & Zha, Z.-J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3333190 · Cited by 6
“Vision Transformer: Transformer [37] was firstly proposed in the natural language processing (NLP) field and has been used to establish state-of-the-art results in many NLP tasks [38], [39], [40], [41].”
MethodGranformer: A granular transformer net with linear complexity
Wang, K., Sun, X., & Shen, T. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128380 · Cited by 5
Multimodal Sentiment Analysis for the Malay Language: New Corpus using CNN-based Framework
Taylor, S., & Fauzi, F. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3703445 · Cited by 5
Prompt-matching synthesis model for missing modalities in sentiment analysis
Liu, J., Wang, Y., Yang, J., Shang, F., & He, F. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113519 · Cited by 7
EAVFormer: an end-to-end audio and visual emotion recognition network based on transformers
Liu, H., Sun, Z., Li, H., Li, Y., Zhang, W., & Song, T. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01845-y · Cited by 5
“These methods combine self-attention and cross-attention mechanisms to construct joint representations that simultaneously preserve modality-specific information and cross-modal interactions [21, 31, 70, 71].”
QD-MSA : A quantum distributed tensor network framework for multimodal sentiment analysis
Li, Y., Li, Y., Qi, Z., Yang, H., Shang, R., & Jiao, L. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103404 · Cited by 7
Multi-Query Cross-Modal Attention Fusion for Cognitive Impairment Recognition
Zhao, M., Gao, H., Qi, X., Yin, H., Song, Y., Bai, Y., Li, J., Zhao, L., & Liu, C. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3583110 · Cited by 6
“The model adaptively adjusts inter-modal attention weights [26], [27], enabling synchronized and synergistic integration of EEG, ECG, and facial expression data within short-time segments.”
Self-attention mechanism prior to modality fusion for multimodal sentiment analysis
Chen, Z., Lu, C., & Wang, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01868-5 · Cited by 5
“Tsai et al. [26] introduced the Multimodal-Transformer (Mult), which achieves modality alignment through a cross-modal attention mechanism, pio-neering this approach.”
A multifaceted approach to drug–drug interaction extraction with fusion strategies
Tran, B.-H., Hoang, H.-M., Nguyen, B.-N., Can, D.-C., & Le, H.-Q. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104874 · Cited by 5
Boosting Modal-Specific Representations for Sentiment Analysis With Incomplete Modalities
Jiang, X., He, L., Gao, F., Zhang, K., Li, J., & Gao, X. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590909 · Cited by 5
“…With Incomplete Modalities Xin Jiang, Lihuo He , Member, IEEE , Fei Gao , Member, IEEE , Kaifan Zhang, Jie Li , and Xinbo Gao , Fellow, IEEE S ENTIMENT analysis has been a popular research topic in machine learning and natural language processing for many years [1], [2], [3], [4], [5], [6].”
He, X., Pan, Y., Chen, Y., Li, Z., Xu, Z., Yang, C., & Wang, K. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3597474 · Cited by 5
“MulT [15]: Multimodal transformer (MulT) employs directional pairwise cross-modal attentions to facilitate interactions between modalities.”
Cross-modal BERT model for enhanced multimodal sentiment analysis in psychological social networks
Feng, J. (2025) · BMC Psychology · doi:10.1186/s40359-025-03443-z · Cited by 5
“Within this section, this paper undertakes a comparative analysis of the proposed CPEF model, juxtaposed against the multimodal model, across the CMU-MOSI and CMU-MOSEI datasets [26], alongside LMF [21], EF-LSTM [23], and MulT [25] models, respectively.”
Affection-Guided Bottleneck Diffusion for Missing Modality Issue in Multimodal Affective Computing
Lin, R., He, Q., Zeng, Y., Mai, S., Huang, L., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3632063 · Cited by 7
“To demonstrate the generalizability of the proposed approach, weadoptthelatentinteractionamongdifferentmodal-ities with the cross-modal attention mechanism as MulT [33] in this paper.”
Text-centric disentangled representation interaction network for Multimodal Sentiment Analysis
Hou, J., Omar, N., Tiun, S. B., Saad, S., & He, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130857 · Cited by 5
SGIR: Star Graph-Based Interaction for Efficient and Robust Multimodal Representation
Ding, Z., Lan, G., Song, Y., & Yang, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321404 · Cited by 5
“MulT [21] uses cross-modal attention to achieve pairwise translation on both unaligned and aligned multimodal sequences.”
BackgroundMethodInfluentialHIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer
Kim, Y., Lee, D. W., Liang, P. P., Alghowinem, S., Breazeal, C., & Park, H. W. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614122 · Cited by 5
“This observation advocates the advantage of crossmodal transformer over conventional alignment; crossmodal attention enables direct capture of potentially long-range signals, including those off diagonals on the attention matrix [29].”
BackgroundMethodInfluentialSelf-citationBae, Y.-H., & Bennett, C. C. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309569 · Cited by 8
“Our investigation revealed that the text encoder considerably outperformed the other encoders, substantiating the notion that a well-designed language network is critical for achieving optimal results [31][33].”
BackgroundMethodInfluentialLiang, S. (2023) · IET Software · doi:10.1049/2023/5566781 · Cited by 8
“Tsai et al. [20] employed multiple pairs of bidirectional crossover encoders to directly focus on and fuse low-level features from each modality.”
BackgroundMultimodal Sentiment Analysis Based on 3D Stereoscopic Attention
Huang, J., Pu, Y., Zhou, D., Shi, H., Zhao, Z., Xu, D., & Cao, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446572 · Cited by 5
“To address this problem, previous works [2] utilized Recurrent Neural Networks (RNNs) and Trans-formers [3, 4, 5].”
BackgroundMethodFusing Modality-Specific Representations and Decisions for Multimodal Emotion Recognition
Ruan, Y.-P., Han, S., Li, T., & Wu, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447035 · Cited by 5
“…: We compare our proposed BCFN model with tesor fusion network (TFN) [18], low-rank multimodal fusion (LMF) [19], multimodal transformer (MulT) [20], progressively modality reinforcement (PMR) [21], and layer-wise fusion with modality independence modeling (LFMIM) [11], in which the PMR and…”
MethodAFMCT: adaptive fusion module based on cross-modal transformer block for 3D object detection
Zhang, B., Wang, Y., Zhang, C., Jiang, J., Pan, Z., Cheng, J., Zhang, Y., Wang, X., Yang, C., & Wang, Y. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01509-3 · Cited by 5
Hwang, Y., & Kim, J.-H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00557 · Cited by 5
“MulT [47] uses cross-modal attention to model interactions between asynchronous modalities and latently adapt one modality to another.”
BackgroundMethodInfluentialMECFNet: Reconstruct Sharp Image for UAV-Based Crack Detection
Liu, C., Zhao, J., Zhu, C., et al. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3401754 · Cited by 5
“Drawing inspiration from the aforementioned works, we introduce the Cross-modal Transformer Module (CTM), which builds upon STM and cross-modal attention [62], aiming to efficiently integrate motion and event information into image features efficiently.”
MethodMTFR: An universal multimodal fusion method through Modality Transfer and Fusion Refinement
Guo, X., Tian, S., Yu, L., He, X., & Wang, Z. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108844 · Cited by 5
Multimodal Transformer Fusion for Emotion Recognition: A Survey
Belaref, A., & Séguier, R. (2024) · doi:10.1109/icnlp60986.2024.10692953 · Cited by 6
“It is based on, and inspired by MulT [22], with several extensions.”
BackgroundMethodInfluentialZhang, Y., Xu, G., Zhao, M. Y., Wang, H., Shi, F., & Chen, S. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3541170 · Cited by 6
“Transformer also plays an important role in multi-modal fusion [33], [34], [35], [36].”
A dynamic weighted fusion model for multimodal sentiment analysis
Yang, L. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04201-9 · Cited by 5
Multimodal Emotion Detection and Analysis from Conversational Data
Jatoth, A., Abri, F., & Nguyen, T. (2025) · Journal of Advances in Information Technology · doi:10.12720/jait.16.8.1127-1141 · Cited by 5
Hypergraph-Based Coordinated Task Allocation and Socially-Aware Navigation for Multi-Robot Systems
Wang, W., Bera, A., & Min, B. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128092 · Cited by 5
“Subsequently, the heterogeneous spatial-temporal features are aligned using the multi-modal transformer [48] to capture the fused spatial-temporal features of the objectives X ST as follows: CMAtten where U ∈ { S, T } represents the unit modality, and U ′ ∈ { S, T } − U denotes the complementary…”
Description Generation for Points of Interest
Zhou, M. R., Zhou, J., Fu, Y., Ren, Z., Wang, X., & Xiong, H. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00226 · Cited by 6
Multimodal Encoders for Food-Oriented Cross-Modal Retrieval
Ying, C., Zhou, D. M., Li, L., & Han, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85899-5_19 · Cited by 5
Modelling Stochastic Context of Audio-Visual Expressive Behaviour With Affective Processes
Tellamekala, M. K., Giesbrecht, T., & Valstar, M. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3157141 · Cited by 5
“Similar to APs, crossmodal self-attention fusion [77] captures long-range temporal context and addresses temporal misalignment between different modalities.”
BackgroundMethodMultimodal Neurophysiological Transformer for Emotion Recognition
Koorathota, S., Khan, Z. A., Lapborisuth, P., & Sajda, P. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871421 · Cited by 5
“While the original transformer architecture has been used for unidirectional translation from source to target texts, Tsai et al [29] were the first to show how human multimodal language time-series can also be well-represented using similar architectures through their MulT architecture.”
MethodER-MRL: Emotion Recognition based on Multimodal Representation Learning
Guo, X., Wang, Y., Miao, Z., Yang, X., Guo, J., Hou, X., & Zao, F. (2022) · International Conference on Information Science and Technology · doi:10.1109/icist55546.2022.9926848 · Cited by 5
Multi-loop graph convolutional network for multimodal conversational emotion recognition
Ren, M., Huang, X., Li, W., & Liu, J. (2023) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2023.103846 · Cited by 5
MTTF: a multimodal transformer for temperature forecasting
Cao, Y. H., Zhai, J., Wei, Z., Zhou, X., & Zhang, F. (2023) · International Journal of Computers and Applications · doi:10.1080/1206212x.2023.2289708 · Cited by 5
“[39]proposedanalgorithmthatusestransformertomodel videoframesandthecorrespondingtitlesininstructionalvideosfor representinglearning.”
The Effectiveness of Self-supervised Pre-training for Multi-modal Endometriosis Classification*†
Butler, D., Wang, H., Zhang, Y., To, M., Condous, G. S., Leonardi, M., Knox, S., Avery, J., Hull, M. L., & Carneiro, G. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340504 · Cited by 5
“Alternatively, cross-modal attention uses query, key, and value combinations from separate modalities [20], [21].”
BackgroundResearch on cross-modal emotion recognition based on multi-layer semantic fusion
Xu, Z., & Gao, Y. (2024) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2024110 · Cited by 5
MHRN: A Multimodal Hierarchical Reasoning Network for Topic Detection
Li, J., Wang, Y., & Li, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3358696 · Cited by 6
“The transformer-based model [48], [49] also shows strong performance in fusing unimodal representations.”
MethodLiang, H., Chen, J., Khan, F., Srivastava, G., & Zeng, J. (2024) · ACM Transactions on Internet Technology · doi:10.1145/3653018 · Cited by 5
Horvath, A.-S., & Pouliou, P. (2024) · CRC Press eBooks · doi:10.1201/9781003425724-46 · Cited by 7
A Review of Multimodal Sentiment Analysis: Modal Fusion and Representation
Fu, H., & Lu, H. (2024) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc61514.2024.10592484 · Cited by 6
“Using the transformer architecture, Tsai et al. [32] transformed the self-attention mechanism into a cross-modal attention structure and presented the MulT model.”
MethodExploring the Role of Audio in Video Captioning
Shen, Y., Yang, L., Wen, L., Yu, H., Elhamifar, E., & Wang, H. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00214 · Cited by 5
“In cross fusion, the two modalities are fed into different Transformer blocks separately, where cross attention Transformers are used to allow cross-modal interaction [31, 47, 48, 51, 55].”
MethodDetail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition
Shi, T., Ge, X., Jose, J. M., Pugeault, N., & Henderson, P. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78305-0_29 · Cited by 5
Luo, Y., Wang, M., & Ye, Q. (2024) · Electronic Commerce Research · doi:10.1007/s10660-024-09937-z · Cited by 5
“To address the challenge of inherent data non-alignments due to different sampling rates for each modal sequence and long-term dependencies between cross-modal elements, The Multimodal Transformer (MULT) [73] adopt an end-to-end manner without explicitly aligning the data to address them.”
Nie, X., Yang, L. T., Li, Z., Deng, X., Fan, F., & Yang, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3459929 · Cited by 6
Dong, Z., Wang, D., Chen, C., Huang, D., & Zhang, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887877 · Cited by 5
“For multimodal fusion, we used mainstream strategies like TFN [39], LMF [40], and MISA [41] at the utterance level, and methods like MFN [42] and MulT [43] at the sequence level.”
Rehman, M. Z. U., Raghuvanshi, D., Jain, U., Bansal, S., & Kumar, N. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103628 · Cited by 6
“These fusion techniques encompass various approaches, including embedding fusion [12], transformer-based fusion [13], and attention-based fusion [14].”
Frame Aggregation and Multi-modal Fusion Framework for Video-Based Person Recognition
Li, F., Wang, W., Liu, Z., Wang, H., Yan, C., & Wu, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67832-6_7 · Cited by 7
“[18] built a multimodal transformer to learn interactions between modalities.”
BackgroundSEOVER: Sentence-Level Emotion Orientation Vector Based Conversation Emotion Recognition Model
Li, Z., Tang, F., Sun, T., Zhu, Y., & Zhao, M. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-92310-5_54 · Cited by 5
“2018), while recent studies consider using multimodal data sets to build emotion analysis models (Tsai et al. 2019), but they only consider the emotion in single utterance and ignore the emotion correlation of context in the conversation process.”
Multimodal Fusion Strategies for Physiological-emotion Analysis
Zhang, T., Huang, Z., Li, R., Zhao, J., & Jin, Q. (2021) · doi:10.1145/3475957.3484452 · Cited by 5
“Tsai et al.[41] proposed MMT x refers to multi-modality features and ˆ y refers to emotion predictions. x A j , x V j , x L j and x P j denote the j th segment of acoustic, visual, textual and physiological features respectively.”
BackgroundPer-frame Sign Language Gloss Recognition
Kim, C. J., & Park, H.-M. (2021) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc52510.2021.9621167 · Cited by 5
An RGB-D Fusion System for Indoor Wheelchair Navigation
Sevastopoulos, C., Acharya, S., & Makedon, F. S. (2023) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3594806.3594851 · Cited by 5
“[25], the multi-head cross-modal attention module is responsible for updating each modality’s sequence(in their case video, audio, and language) via low-level external information.”
BackgroundMOC: Multi-modal Sentiment Analysis via Optimal Transport and Contrastive Interactions
Li, Y., Zhu, Q., He, H., Gu, Z., & Zheng, C. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8082-6_34 · Cited by 5
Stappen, L., Baird, A., Rizos, G., Tzirakis, P., Du, X., Hafner, F., Schumann, L., Mallol-Ragolta, A., Schuller, B. W., Lefter, I., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14858 · Cited by 5
“As baseline for the non-sequential predictions of MuSe-Topic , we choose the Multimodal Transformer (MMT) [41].”
BackgroundMethodInfluentialLee, K., Won, J., Hyun, H., Hahn, S., Choi, E., & Lee, J. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-39539-0_4 · Cited by 5
Li, B., Du, B., & Ye, J. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3744251 · Cited by 4
Li, W., Li, X., Li, Z., Yu, J., & Chen, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130622 · Cited by 4
iAMeta: Advancing multimodal metaphor detection using MLLMs and information debiasing
He, X., Yu, L., & Tian, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103348 · Cited by 5
Cai, T., Wang, S., Wang, J., Xiong, Y., & Liu, L. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0063-x · Cited by 5
Li, Z., Liu, Y., Yang, C., Zhou, Y., & Hu, S. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590929 · Cited by 5
“MulT [15] proposes a cross-modal attention mechanism that fuses modalities pairwise, which has been widely applied in the MER task [9], [13], [21], [22], [23], [24].”
Multimodal Spatiotemporal Collaborative Approach for Network Traffic Anomaly Detection
Sun, W., Li, Z., Cui, J., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3598294 · Cited by 4
Multimodal large language model enhancement network for multimodal sentiment analysis
Zhu, H., Shen, Z., Dai, C., & Yu, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01914-2 · Cited by 5
“Multimodal transformer (MulT) [4] introduced a mechanism called directional pairwise crossmodal attention, which can process time-series data from differ - ent modalities (e.g., textual, visual, and acoustic) without explicitly aligning the data.”
InfluentialAdvanced Algorithm Combinations for Safe Human-Robot Collaboration: A Review and Research Proposal
Lefranc, G., Gatica, G., Vásquez, E., et al. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.08.018 · Cited by 4
Nwaiwu, S., Jongsawat, N., & Tungkasthan, A. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107783 · Cited by 6
Yan, K., Lai, P., Zheng, X., Yang, Y., Ren, Y. C., Badarch, T., & Chen, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112923 · Cited by 8
Yang, Y., Xiao, C., Zou, D., Wang, L., Yang, R., Zhang, Y., Zhang, L., Zhao, Z., Qiu, S., Liu, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02058-3 · Cited by 4
Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review
Khan, U., Nawaz, U., Qayyum, A., Baig, I., Alam, M., Ashraf, S. Q., Xie, Y., Khan, M. H., Bilal, M., & Qadir, J. (2026) · IEEE Access · doi:10.1109/access.2026.3729436 · Cited by 14
FAIR-VID: A Multimodal Pre-Processing Pipeline for Student Application Analysis
Laukaitis, A., Kalibatienė, D., Jodenytė, D., Normantas, K., Jancevičius, J., Jankauskas, M., & Serackis, A. (2025) · Applied Sciences · doi:10.3390/app152413127 · Cited by 6
Bridging modal gaps in multimodal sentiment analysis: a self-supervised text-guided fusion approach
Zhong, H., Yi, J., Hu, R., Chen, A., & Han, G. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.131012 · Cited by 4
Multi-resolution adaptive channel fusion transformer encoder LSTM for accurate streamflow prediction
Apak, S., Kılınç, H. Ç., Yurtsever, A., Haznedar, H., & Özkan, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-40713-1 · Cited by 6
“To ensure comparability across resolutions, these logits are passed through a softmax function to obtain normalized fusion weights: such that α ( r ) ⩾ 0 and α ( r ) = 1 40,41 .”
Lawan, A., Pu, J., Haruna, Y., et al. (2026) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2026.3671067 · Cited by 4
“ATI includes fully connected layers and multi-head cross-modal attention (MHCA) [42] to enhance textual representations by focusing on the aspect.”
BiMNet: A Multimodal Data Fusion Network for continuous circular capsulorhexis Action Segmentation
Bian, G., Zheng, J.-Y., Li, Z., Wang, J., Fu, P., Chen, X., da Silva, D. S., Wu, W., & de Albuquerque, V. H. C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121885 · Cited by 5
Chen, P., & Fu, L. (2024) · International Journal of Information Systems in the Service Sector · doi:10.4018/ijisss.349564 · Cited by 6
Frame-level nonverbal feature enhancement based sentiment analysis
Zheng, C., Peng, J., Wang, L., Zhu, L., Guo, J., & Cai, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125148 · Cited by 4
Zhao, X., Miao, X., Xu, X., Liu, Y., & Cao, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650083 · Cited by 4
“For example, tensor fusion [8]–[10] and attention-based fusion [4], [11], these fusion methods are effective, but they have high complexity and the learned high-dimensional multimodal embeddings are inevitably redundant.”
BackgroundMethodInfluentialChen, F., Tang, X., & Hao, K. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111047 · Cited by 4
MEAS: Multimodal Emotion Analysis System for Short Videos on Social Media Platforms
Wei, Q., Zhou, Y., Xiang, S., Xiao, L., & Zhang, Y. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3490846 · Cited by 9
“…as shown in Table XIV, the experimental results on the IEMOCAP and CMU-MOSEI datasets show our method outperforms all existing open-source multimodal emotion analysis methods [20], [24], [28], [29], [43], [44], [45], [46], [47], [48], achieving the highest of both weighted accuracy and F1 score.”
Wang, F., Zhang, Z., Song, J., et al. (2025) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2025.101484 · Cited by 4
Fu, Y., Liu, Q., Song, Q., Zhang, P., & Liao, G. (2025) · Applied Sciences · doi:10.3390/app15084509 · Cited by 4
Toward Efficient Network Traffic Classifications via Multimodal Learning
Chang, L., & Cao, B. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3563987 · Cited by 4
“Tsai et al. [34] introduced multimodal transformer (MulT), which aligns multimodal information such as vision, language, and audio, through an end-to-end approach and extracts long-distance dependencies between cross-modal elements.”
Bimodal Sentiment Analysis Based on a Pre-Trained Model and Masked Attention Fusion
Cai, L., Li, H., Zhang, N., & He, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3564826 · Cited by 4
“In 2019, Tsai et al. [20] introduced the multimodal transformer (MulT), which attends to interactions between multimodal sequences across distinct time steps and latently adapts streams from one modality to another.”
Hypergraph-based multimodal adaptive fusion for emotion recognition in conversation
Xie, X., Ma, T., Li, J., & Rong, H. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07584-1 · Cited by 4
“To fully leverage the complementarity of multimodal information, Tsai et al. [25] proposed the Cross-modal Transformer (MulT), which enhances target modality features by learning attention relationships between different modality features using pairwise cross-modal attention.”
Kommu, S., Wang, Y., Wang, Y., & Wang, X. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf217 · Cited by 5
Multimodal feature learning for gait recognition of camouflaged individuals
Li, H.-P., Wang, C., & Liu, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129202 · Cited by 4
Knowledge Enhanced and Incongruity Perceiving Network for Multimodal Sarcasm Detection
Liu, M., & Li, Z. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10499-x · Cited by 4
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
