The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Multimodal Transformer for Unaligned Multimodal Language Sequences
Open accessAbstract
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Transformer models [158] have been shown to be useful for temporal multimodal data by automatically aligning and capturing complementary features at different time-steps [154, 174].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [154]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In MulT, multiple pairwise cross-modal transformers were merged for learning attention across modalities and then fused embedding from different modalities [86].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [86]
The cited source says what the claim attributes to it.
In the citing paper
“proposed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇ saitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Previous works have shown that these different modalities are complimentary for emotion expression”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Tsai et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“On the other hand, lightweight natural language processing (NLP) models such as TinyBERT [232] focus on reducing the size of transformer-based models via knowledge distillation, making them suitable for mobile deployment.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [232]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 16×
“We choose the self-attention transformer (Vaswani et al., 2017).”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“Following prior works on transformers (Vaswani et al., 2017; Chen et al., 2018; Devlin et al., 2018; Dai et al., 2018), we add a residual connection to the crossmodal attention computation.”
BackgroundMethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“We convert video transcripts into pre-trained Glove word embeddings (glove.840B.300d) (Pennington et al., 2014).”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“LN means layer normalization (Ba et al., 2016).”
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“We use Facet (iMotions, 2017) to indicate 35 facial action units, which records facial muscle movement (Ekman et al., 1980; Ekman, 1992) for representing per-frame basic and advanced emotions.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 1×
“This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014).”
BackgroundGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Research works printed but not linked: 3
- Degottex Gilles, Kane John, Drugman Thomas, Raitio Tuomo, and Scherer Stefan. 2014 Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
- Dumpala Sri Harsha, Chakraborty Rupayan, and Kopparapu Sunil Kumar. 2019 Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
- Srivastava Nitish and Salakhutdinov Ruslan R. 2012 Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222– 2230.
Cited by
Sorted by citations
Items 401–500 of 1,004
A Multimodal Sentiment Analysis Method Based on Fuzzy Attention Fusion
Zhi, Y., Li, J., Wang, H., Chen, J., & Wei, W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3434614 · Cited by 28
Nguyen, L. D., Nguyen, Q. H., Trinh, Q. H., & Nguyen, B. P. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01240 · Cited by 27
HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
Yi, M.-H., Kwak, K.-C., & Shin, J. (2025) · Applied Sciences · doi:10.3390/app15031053 · Cited by 30
“This method uses ‘query information’ extracted from one modality (e.g., text) to combine the features (key and value) generated by another modality (e.g., speech), enabling the model to learn complex interactions and semantic structures that are difficult to capture with a single modality [21,22].”
Li, J., Wang, X., & Zeng, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3581236 · Cited by 25
“Existing mainstream works focus on enhancing the performance of MSA models by designing complex structures, interaction mechanisms, or fusion paradigms, which can be broadly categorized into tensor-based methods [39], [43], attention-based methods [41], [44], and graph-based methods [45], [46].”
Jiang, P., Ma, Z., & Mei, G. (2026) · Natural hazards and earth system sciences · doi:10.5194/nhess-26-487-2026 · Cited by 35
A multimodal time-series method for gifting prediction in live streaming platforms
Xi, D., Tang, L., Chen, R., & Xu, W. X. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103254 · Cited by 26
Salient Mask-Guided Vision Transformer for Fine-Grained Classification
Demidov, D., Sharif, M., Abdurahimov, A., Cholakkal, H., & Khan, F. S. (2023) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0011611100003417 · Cited by 26
“, 2017), the Transformer architecture with its selfattention mechanism has shown a great success in that field (Devlin et al., 2019; Dai et al., 2019; Tsai et al., 2019) and was later extended by researchers to computer vision (CV) tasks.”
A text guided multi-task learning network for multimodal sentiment analysis
Luo, Y., Wu, R., Liu, J., & Tang, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126836 · Cited by 25
UniSA: Unified Generative Framework for Sentiment Analysis
Li, Z., Lin, T.-E., Wu, Y., Liu, M., Tang, F., Zhao, M., & Li, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612336 · Cited by 32
“Recent research on multimodal sentiment analysis has emphasized the importance of modal consistency and difference through multi-task joint learning [74] or modal translation [39], leveraging cross-modality and multi-scale modality representation to implement the modal alignment [35, 63].”
BackgroundHuang, S., Qin, L., Wang, B., Tu, G., & Xu, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446922 · Cited by 25
“(2) The second layer adopts KQV attention [15] to obtain bi-modal representations of video and audio aligned with text modality.”
BackgroundXiao, L., Zhou, E., Wu, X., Yang, S., Ma, T., & He, L. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9860020 · Cited by 24
“Tsai et al. [10] put forward a Multi-modal Transformer (MulT) which utilizes a directional pairwise cross-modal attention to fuse inter-modality dynamics via directly attending to low-level features in other modalities.”
BackgroundWu, L., Wang, H., Chen, Y., Zhang, X., Zhang, T., Shen, N., Tao, G., Sun, Z., Ding, Y., Wang, W., et al. (2023) · iScience · doi:10.1016/j.isci.2023.108183 · Cited by 24
Gradient-Guided Modality Decoupling for Missing-Modality Robustness
Wang, H., Luo, S., Hu, G., & Zhang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29474 · Cited by 24
Lv, C., Fan, L., Li, H., et al. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106480 · Cited by 23
Multimodal Sentiment Analysis With Mutual Information-Based Disentangled Representation Learning
Sun, H., Niu, Z., Wang, H., Yu, X., Liu, J., Chen, Y.-W., & Lin, L. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3529732 · Cited by 28
“The labels are signed in interval [1, 7], corresponding to [ − 3, 3] in CMU-MOSI and CMU-MOSEI.”
InfluentialHuang, C., Lin, Z., Huang, Q., Huang, X., Jiang, F., & Chen, J. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01806-y · Cited by 26
“MulT [8] introduces transformers into the MSA task, employing cross-modality attention to capture inter-modality interactions.”
InfluentialActual Cause-Guided Adaptive Gradient Scaling for Balanced Multimodal Sentiment Analysis
Chen, J., Huang, Q., Huang, C., & Huang, X. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3736415 · Cited by 24
“Following [13, 14, 20, 36], we use Mean Absolute Error as the loss function for multimodal sentiment analysis and cross-entropy as the loss function for multimodal humor detection.”
InfluentialAudio–Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
Lee, Y.-H., Jang, D., Kim, J.-B., Park, R., & Park, H. (2020) · Applied Sciences · doi:10.3390/app10207263 · Cited by 23
“In addition, the transformer model is under study for various audio–visual tasks (e.g., [29–32]).”
MethodTwo-Level Multimodal Fusion for Sentiment Analysis in Public Security
Sun, J., Yin, H., Tian, Y., Wu, J., Shen, L., & Chen, L. (2021) · Security and Communication Networks · doi:10.1155/2021/6662337 · Cited by 23
Cross-modal dynamic convolution for multi-modal emotion recognition
Wen, H., You, S., & Fu, Y. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103178 · Cited by 23
Climate and Weather: Inspecting Depression Detection via Emotion Recognition
Wu, W., Wu, M., & Yu, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746634 · Cited by 24
“SOTA results on MOSEI vary from 80 to 84% [22, 23].”
BackgroundDeep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 28
“Recently, some literatures adopt attention-based structure to achieve multi-view sequence alignment (Tsai et al. 2019; Yang et al. 2020; Le et al. 2018).”
BackgroundMethodInfluentialYang, J., Li, S., Wang, X., Lu, J., Wu, H., & Wang, X. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3228817 · Cited by 26
“Because of the strong correlation between signal-modal features, a multimodal feature and knowledge fusion module with cross-modal attention is developed to integrate cross-modal interactions and achieve multiview portrayals of manufacturing scenes, inspired by the multimodal transformer [89], [90].”
BackgroundGe, S., Jiang, Z., Cheng, Z., Wang, C., Yin, Y., & Gu, Q. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583258 · Cited by 23
“Previous methods mainly focus on the alignment and fusion of multi-modal data [5, 13, 25, 33, 34].”
MethodMultimodal Reaction: Information Modulation for Cross-Modal Representation Learning
Zeng, Y., Mai, S., Yan, W., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3293335 · Cited by 25
“Furthermore, modality-translation methods [16], [33] propose to learn joint representation by translating the source modality into the target one, which narrows the modality gap and bene-fits multimodal fusion.”
MethodHow to leverage the multimodal EHR data for better medical prediction?
Yang, B., & Wu, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.329 · Cited by 26
“The second one is attention fusion (Tsai et al., 2019).”
Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis
Fan, Q., Han, J., He, Y., Zheng, T., & Zheng, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.821 · Cited by 23
“Therefore, to integrate the non-verbal information into word representations, inspired by Multimodal Transformer (MulT) (Tsai et al., 2019) which provides a latent cross-modal adaptation that fuses multimodal information by directly attending to low-level features in other modalities, we propose a new non-verbal infor…”
BackgroundMethodResultInfluentialA Federated Learning Framework for Brain Tumor Segmentation Without Sharing Patient Data
Zhang, W., Jin, W., Rho, S., Jiang, F., & Yang, C. (2024) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.23147 · Cited by 22
“Tsai et al. [51] 89.6% Vaswani et al. [52] 91.2% DeepMedic + CRF in [53] 90.6% response prediction, or even multimodal data fusion within the confines of data privacy constraints.”
InfluentialWCG-VMamba: A multi-modal classification model for corn disease
Wang, H., He, M.-F., Zhu, M., & Liu, G.-H. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109835 · Cited by 24
OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis
Hu, J., Mathur, L., Liang, P. P., & Morency, L. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099277 · Cited by 40
“Originally popularized in the natural language processing literature [19], [55], it has become common practice to pre-train ViTs [20] or multimodal architectures [40], [46], [62], [41], [66] on vast amounts of unlabeled images using multitask and self-supervised objectives [12], [20].”
Self-citationCotes, R. O., Boazak, M., Griner, E. L., Jiang, Z., Kim, B., Hogg-Bremer, W., Seyedi, S. S., Rad, A. B., & Clifford, G. D. (2022) · JMIR Research Protocols · doi:10.2196/36417 · Cited by 23
Transformer Based Multi-view Network for Mammographic Image Classification
Sun, Z., Jiang, H., Ma, L., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16437-8_5 · Cited by 22
WMNN: Wearables-Based Multi-Column Neural Network for Human Activity Recognition
Tang, C., Chen, X., Gong, J., Occhipinti, L. G., & Gao, S. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3219364 · Cited by 23
“Compared to more complex multimodal architectures such as the Multi-modal Transformer [44], the WMNN proposed in this study focuses more on robustness and generalization on small datasets.”
BackgroundChen, C., Hong, H., Guo, J., & Song, B. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3263801 · Cited by 22
“MFM [30] decomposed the modal features to generate multimodal discriminant factors and specific modal generation factors.”
BackgroundMethodInfluentialTACFN: Transformer-Based Adaptive Cross-Modal Fusion Network for Multimodal Emotion Recognition
Liu, F., Fu, Z., Wang, Y., & Zheng, Q. (2023) · CAAI Artificial Intelligence Research · doi:10.26599/air.2023.9150019 · Cited by 42
A graph convolution-based heterogeneous fusion network for multimodal sentiment analysis
Zhao, T., Peng, J., Huang, Y., Wang, L., Zhang, H., & Cai, Z. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05151-w · Cited by 23
Lin, C., Cheng, H., Rao, Q., & Yang, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3361374 · Cited by 24
“[7]proposedaMultimodal Transformer(MuLT)thatutilizesthecross-modalattention mechanismtofusetwomodalitydataanddoesnotneedmul-timodaldataalignmentoperation.”
Multi-Modal Embeddings Using Multi-Task Learning for Emotion Recognition
Khare, A., Parthasarathy, S., & Sundaram, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1827 · Cited by 24
“More recently, cross-modal transformers have been employed in order to project one modality into another [18], which have been successfully applied to sentiment analysis [18] and speech recognition [19].”
BackgroundMethodInfluentialMulti-Grained Multimodal Interaction Network for Sentiment Analysis
Fang, L., Liu, G., & Zhang, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446351 · Cited by 21
“Following previous work [4, 5] about multimodal fusion, we employ a transformer variant for unaligned multimodal interaction.”
BackgroundMethodResultInfluentialDabbabi, K., & Mars, A. (2024) · Journal of Systems Science and Systems Engineering · doi:10.1007/s11518-024-5607-y · Cited by 24
“…in weighted accuracy (WA) by approximately 2%, outperforming the 24-MFCCs tested for the six-class emotion recognition task on the CMU-MOSEI dataset (Liang et al. 2018, Zadeh et al. 2018, Gideon et al. 2017, Tsai et al. 2019, Vaishnav et al. 2021, Nair and Gomathi 2022, Shukla et al. 2022).”
Steering Angle-Guided Multimodal Fusion Lane Detection for Autonomous Driving
Gong, Y., Zhang, X., Lu, J., Jiang, X., Wang, Z., Liu, H., Li, Z., Wang, L., Yang, Q., & Wu, X. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3507536 · Cited by 23
CorMulT: A Semi-Supervised Modality Correlation-Aware Multimodal Transformer for Sentiment Analysis
Li, Y., Zhu, R., & Li, W. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3559866 · Cited by 21
“Once the dataset is disturbed with random weak-correlation perturbations, the performance drops to 51.7% (Acc@7) and 82.3% (F1), but still substantially outperforms the baseline MULT [18] under the same perturbations (Acc@7: 47.5%, F1: 76.3%).”
InfluentialAlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-Modal Alignment
Li, Y., Xing, Y., Lan, X., Li, X., Chen, H., & Jiang, D. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02307 · Cited by 30
“Similarly, CMA [37], based on cross-modal attention mechanisms, was proposed to fuse features from three modalities.”
InfluentialA versatile multimodal learning framework bridging multiscale knowledge for material design
Wu, Y., Ding, M., He, H., Wu, Q., Jiang, S., Zhang, P., & Ji, J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01767-3 · Cited by 33
“We incorporated a multimodal Transformer with cross-attention 38 to effectively capture interactions between processing conditions and structures to further improve multimodal representation learning.”
Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion
Zhang, S., Li, B., & Yin, C. (2021) · Sensors · doi:10.3390/s22010074 · Cited by 22
“[43], which was able to be directly applied to unaligned sequences.”
BackgroundWu, Y., Zhang, Z., Peng, P., Zhao, Y., & Qin, B. (2022) · doi:10.1145/3551876.3554813 · Cited by 21
“MulT [34] utilizes the cross-modal attention to fuse multi-modal features.”
BackgroundMethodVision Enhanced Generative Pre-trained Language Model for Multimodal Sentence Summarization
Jing, L., Li, Y., Xu, J., Yu, Y., Shen, P., & Song, X. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1372-x · Cited by 22
Visual language integration: A survey and open challenges
Park, S., & Kim, Y. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100548 · Cited by 24
Zheng, Y., Gong, J., Wen, Y., & Zhang, P. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122728 · Cited by 23
Improving conversational recommender systems via multi-preference modelling and knowledge-enhanced
Zhang, C., Huang, X., An, J., & Zou, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111361 · Cited by 23
Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation
Sanyal, S., Asokan, A. R., Bhambri, S., Kulkarni, A., Kundu, J. N., & Babu, R. V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01735 · Cited by 23
“Prior vision transformer works have demonstrated superior feature alignment capabilities within the attention module of a transformer, even across multi-modal scenarios such as text-to-vision [21, 53].”
BackgroundLabel distribution for multimodal machine learning
Ren, Y. C., Xu, N., Ling, M., & Geng, X. (2021) · Frontiers of Computer Science · doi:10.1007/s11704-021-0611-6 · Cited by 22
“[7] introduces the Multimodal Transformer (MulT) for unaligned Multimodal Language Sequences.”
BackgroundLiu, F., Shen, S.-Y., Fu, Z.-W., Wang, H.-Y., Zhou, A.-M., & Qi, J.-Y. (2022) · Entropy · doi:10.3390/e24071010 · Cited by 21
“It is noteworthy that the axis of parameters in Figure 4 omits the range from 1.4 M and 110.4 M, and, thus, the required parameters for BBFN and MISA are huge.”
InfluentialA multimodal framework for pepper diseases and pests detection
Liu, J., & Wang, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-80675-w · Cited by 20
PAMoE-MSA: polarity-aware mixture of experts network for multimodal sentiment analysis
Huang, C., Lin, Z., Han, Z., Huang, Q., Jiang, F., & Huang, X. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00362-y · Cited by 20
“Mult [8].”
Toward Robust Multimodal Sentiment Analysis using multimodal foundational models
Zhao, X., Poria, S., Li, X., Chen, Y., & Tang, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126974 · Cited by 28
Gomaa, A., Huang, Y., Stephan, P., Breininger, K., Frey, B., Doerfler, A., Schnell, O., Delev, D., Coras, R., Donaubauer, A.-J., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-02026-7 · Cited by 20
“For this setup, the optimum value of M was 4.”
Towards robust sentiment analysis with multimodal interaction graph and hybrid contrastive learning
Gong, P., Liu, J., Zhang, X., Li, X., Wei, L., & He, H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111870 · Cited by 22
Artificial Intelligence in Endurance Sports: Metabolic, Recovery, and Nutritional Perspectives
Grivas, G. V., & Safari, K. (2025) · Nutrients · doi:10.3390/nu17203209 · Cited by 31
“In endurance applications, such designs have improved the prediction of overload, illness risk, and recovery patterns by learning interactions among sleep, training load, thermal stress, and glycemic dynamics [21,101,103,104].”
InfluentialDeep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation
Guo, X., Kong, A. W., & Kot, A. C. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142448 · Cited by 21
“In multimodal sentiment prediction, both Tsai et al. [20] and Hazarika et al. [23] employ transformer encoders to fuse unimodal representations.”
BackgroundMethodInfluentialMultimodal Personality Recognition using Cross-attention Transformer and Behaviour Encoding
Agrawal, T., Agarwal, D., Balážia, M., Sinha, N., & Brémond, F. (2022) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010841400003124 · Cited by 30
“Emotion recognition is a closely related problem and has interesting approaches for multimodal data processing (Dai et al., 2021; Tsai et al., 2019).”
Object-Agnostic Transformers for Video Referring Segmentation
Yang, X., Wang, H., Xie, D., Deng, C. D., & Tao, D. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3161832 · Cited by 20
“requirement of object detection like in UNITER [34] and other multi-modal transformers [36], [37], which can operate”
BackgroundCan We Exploit All Datasets? Multimodal Emotion Recognition Using Cross-Modal Translation
Yoon, Y. C. (2022) · IEEE Access · doi:10.1109/access.2022.3183587 · Cited by 23
“To overcome this limitation, the attention-mechanism-based fusion method [22, 23, 24] or transformer algorithm [25, 26, 27, 28] are widely used.”
MethodStyleBERT: Text-audio sentiment analysis with Bi-directional Style Enhancement
Lin, F., Liu, S., Zhang, C., Jin, F., & Wu, Z. (2022) · Information Systems · doi:10.1016/j.is.2022.102147 · Cited by 20
Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross- Modal Denoising Networks
Pan, W., Shi, H., Zhao, Z., Zhu, J., He, X., Pan, Z., Gao, L., Yu, J., Wu, F. Q., & Tian, Q. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00138 · Cited by 19
Exploring Semantic Relations for Social Media Sentiment Analysis
Zeng, J., Zhou, J., & Huang, C. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3285238 · Cited by 22
“Previous image-text learning methods can be generally classified into two categories: 1) feature-based methods [13], [14], [15]; and 2) learning-based methods [16], [17], [18].”
MethodWang, W., Wang, R., Mao, L., & Min, B. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341395 · Cited by 20
Hu, R., Yi, J., Chen, A., & Chen, L. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3388670 · Cited by 22
Zhang, R., Chi, X.-W., Liu, G., Zhang, W., Du, Y., & Wang, F. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3567535 · Cited by 32
“The baselines include: (cid:2) Mult [45] uses pairwise crossmodal attention to address long-range dependencies and data misalignment across modalities, enabling end-to-end interaction between sequences at distinct time steps.”
InfluentialDialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the ConversationPreprint
Mao, Y., Sun, Q. Y., Liu, G., Wang, X., Gao, W., Li, X., & Shen, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07637 · Cited by 35
“MulT (Tsai et al. 2019a) is the SOTA fusion technique for emotion recognition.”
BackgroundMethodInfluentialA Survey on MLLMs in Education: Application and Future Directions
Xing, W., Zhu, T., Wang, J., & Liu, B. (2024) · Future Internet · doi:10.3390/fi16120467 · Cited by 26
Pan, C., Liu, Y., Oh, Y.-J., & Lim, C. (2024) · Energies · doi:10.3390/en17246378 · Cited by 23
Yang, L., Zhong, J., Wen, T., & Liao, Y. J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103230 · Cited by 21
TWEETSPIN: Fine-grained Propaganda Detection in Social Media Using Multi-View Representations
Vijayaraghavan, P., & Vosoughi, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.251 · Cited by 21
“Inspired from the literature in multi-modal learning (Tsai et al., 2019), we pro-pose a transformer-based multi-view propaganda detection model, M V -”
MethodEmotion Recognition with Pre-Trained Transformers Using Multimodal Signals
Vazquez-Rodriguez, J., Lefebvre, G., Cumin, J., & Crowley, J. L. (2022) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii55700.2022.9953852 · Cited by 22
“[3], [4] describes the use of pre-trained models to extract features from visual, audio and text modalities, and followed by a cross-modal Transformer [25] to combine these different features.”
MethodHateful Memes Detection Based on Multi-Task Learning
Ma, Z., Yao, S., Wu, L., Gao, S., & Zhang, Y. (2022) · Mathematics · doi:10.3390/math10234525 · Cited by 19
Han, W., Chen, H., Kan, M., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.717 · Cited by 21
Yeo, J. H., Kim, M., Choi, J., Kim, D. H., & Ro, Y. M. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3352388 · Cited by 22
“Recent study [77] shows that cross-attention can be used for aligning multi-modal language sequences, and receiving information from another modality to supplement one modality.”
BackgroundA multimodal dynamical variational autoencoder for audiovisual speech representation learning
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106120 · Cited by 20
“Finally, we also include in this experiment two state-of-the-art supervised multimodal approaches (Chumachenko et al., 2022; Tsai et al., 2019), which are based on an audiovisual transformer architecture.”
MethodVIEMF: Multimodal metaphor detection via visual information enhancement with multimodal fusion
He, X., Yu, L., Tian, S., Yang, Q., Long, J., & Wang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103652 · Cited by 20
Multi-View Interactive Representations for Multimodal Sentiment Analysis
Tang, Z., Xiao, Q., Qin, Y., Zhou, X., Zhou, J. T., & Li, K. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3357480 · Cited by 20
“In one hand, current methods struggle to capture multi-view sentiment cues across different interaction states, limiting multimodal sentiment representa-tions’ expressiveness [8], [14], [15], [16], [17], [18].”
BackgroundMethodInfluentialPeng, C., Chen, K., Shou, L., & Chen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29374 · Cited by 20
Triple-modality interaction for deepfake detection on zero-shot identity
Yoon, J., Panizo-LLedot, Á., Camacho, D., & Choi, C. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102424 · Cited by 22
Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition
Jin, X., Xiao, J., Jin, L., & Zhang, X. (2024) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12346 · Cited by 19
Chen, R., Zhou, W., Hu, H., Fei, Z., Fei, M., & Zhou, H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112372 · Cited by 19
Generalized Visual-Tactile Transformer Network for Slip Detection
Cui, S., Wei, J., Li, X., Wang, R., Wang, Y., & Wang, S. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2020.12.2430 · Cited by 19
“Different form Tsai et al. (2019), we build modal-specific transformers to extract modal-specific features further.”
BackgroundInfluentialSharma, B., Madhavi, M. C., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413388 · Cited by 19
“We employ a cross-attention method as in [26], [27] to generate the alignment between the two streams of embedding vectors, namely, acoustic embeddings, and ALN linguistic embeddings.”
VAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Wang, Y., Wu, J., Furumai, K., Wada, S., & Kurihara, S. (2022) · IEEE Access · doi:10.1109/access.2022.3174215 · Cited by 19
Deep Learning of radiology-genomics integration for computational oncology: A mini review
Wang, F., Li, Y., & Zeng, T. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.06.019 · Cited by 24
Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Xiong, A., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3553149 · Cited by 21
“…out the noise in nonverbal modalities effectively, which is different from our MGT. Additionally, to avoid the negative impact of modality gap, modality-translation methods [5], [22], [45] reduce the modality gap by translating the features of one modality into the features of another modality.”
Yang, Y., Dong, X., & Qiang, Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34755 · Cited by 27
Adnan, T., Abdelkader, A., Liu, Z., Hossain, E., Park, S., Islam, M. S., & Hoque, M. E. (2025) · npj Parkinson s Disease · doi:10.1038/s41531-025-00956-7 · Cited by 26
“This adaptive, learnable spaceenablesdynamicfeaturefusionduringtraining,promotingrobustand generalizablerepresentations – a critical asset in PD detection where symptoms are subtle and varied 99 .”
EmoVerse: Enhancing Multimodal Large Language Models for Affective Computing via Multitask Learning
Li, A., Xu, L., Ling, C., Zhang, J., & Wang, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130810 · Cited by 26
Yang, Z., He, Q., Yu, M., Du, N., & Lu, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103571 · Cited by 26
CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations
Li, H., Ding, W., Kang, Y., Liu, T., Wu, Z., & Liu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.323 · Cited by 29
“We follow the same experimental protocol as MuIT (Tsai et al., 2019), with the same train/test data split and the same evaluation metrics, which includes two classification metrics: binary accuracy (i.e., Acc 2 : accuracy over positive/negative sentiments classification), and F1 score, two regression…”
MethodWang, H., Yang, M.-C., Li, Z., Liu, Z., Hu, J., Fu, Z.-W., & Liu, F. (2022) · Comput. Animat. Virtual Worlds · doi:10.1002/cav.2090 · Cited by 14
Unaligned Multimodal Sequences for Depression Assessment From Speech
Zhao, Z., & Wang, K. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871556 · Cited by 19
“For example, the audio and video stream may not be completely aligned, face recognition may be conducted independently of the current speech [4], or a subject’s current facial”
BackgroundMai, D. H. A., Nguyen, L. T., & Lee, E. Y. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.1067562 · Cited by 18
“were previously dominated by CNNs, and they are now also used in multimodal learning for content generation (Tsai et al., 2019; Yu et al., 2019; Dzabraev et al., 2021).”
MethodYoon, J., Choi, G. H., & Choi, C. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101922 · Cited by 19
Huang, J., Lu, P., Sun, S., & Wang, F. (2023) · Electronics · doi:10.3390/electronics12163504 · Cited by 19
Metadata
- Authors
- Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Volume
- 2019
- Pages
- 6558-6569
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC7195022
- OpenAlex
- W2947476638
- Semantic Scholar
- 949fef650da4c41afe6049a183b504b3cc91f4bd
- Cited by
- 2,259
- References
- 33
- Citations by year
- 2026: 570 · 2025: 564 · 2024: 426 · 2023: 291 · 2022: 214 · 2021: 138 · 2020: 52 · 2019: 4
- FWCI
- 32.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- National Institute of Mental Health (R01MH096951)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal transformer, cross-modal attention, long-range dependencies, facial gestures, acoustic signals, variable sampling rate, end-to-end modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
