Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Open accessAbstract
There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition.However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption.To address these concerns, we present a regularization method based on tensor rank minimization.Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations.However, the presence of noise or incomplete values breaks these correlations and results in tensor representations of higher rank.We design a model to learn such tensor representations and effectively regularize their rank.Experiments on multimodal language data show that our model achieves good results across various levels of imperfection.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multimodal imperfections capture correlations in imperfections across modalities, such as missing modalities or a chunk of time missing in multimodal time-series data.
“Multimodal imperfections capture correlations in imperfections across modalities (e.g., missing modalities [123], or a chunk of time missing in multimodal time-series data [101]).”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
Section 4.2 (Rank Analysis; page unavailable) describes structured missingness in multimodal time-series data: all features at a selected time step are dropped. It does not describe a contiguous chunk of time missing, and time steps are selected independently for each modality, not as correlated imperfections across modalities. Section 3.3 says imperfections break correlations in the clean data rather than capture correlations among the imperfections. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCurrent benchmarks typically focus on performance without quantifying the potential drawbacks involved with increased time and space complexity and the risk of decreased robustness from imperfect modalities.
“Furthermore, current benchmarks typically focus on performance without quantifying the potential drawbacks involved with increased time and space complexity [148], and the risk of decreased robustness from imperfect modalities [101, 123].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
Real-world multimodal data is often imperfect as a result of missing entries, noise corruption, or missing modalities entirely, which calls for robust models that can still make accurate predictions despite only having access to noisy and missing signals.
“Real-world multimodal data is often imperfect as a result of missing entries, noise corruption, or missing modalities entirely, which calls for robust models that can still make accurate predictions despite only having access to noisy and missing signals [101, 123].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
Following Liang et al., MULTIBENCH implements types of noise and missing values in time-series data.
“Following Liang et al., [101], we implement the following types of noise and missing values in time-series data:”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
Approaches for improving robustness modify the objective function to account for robustness to noisy or missing modalities.
“These approaches modify the objective function to account for robustness to noisy [101] or missing [89, 111, 123] modalities.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
Several efficient variants have been proposed to approximate expensive full tensor products with cheaper variants while maintaining performance.
“Several efficient variants have been proposed to approximate expensive full tensor products with cheaper variants while maintaining performance [71, 101, 106].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [101]
The Related Work section discusses uses of tensor products, and the Discussion section notes a trade-off between computation cost and the tightness of approximating tensor rank. Neither passage says that cheaper variants approximate full tensor products while maintaining performance; searches for efficient tensor-product approximations did not locate that claim in our copy. (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Carroll, J. D., & Chang, J.-J. (1970) · Psychometrika · doi:10.1007/bf02310791 · Cited by 4,783 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
Self-citationMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
Self-citationTensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem
de Silva, V., & Lim, L. (2008) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/06066518x · Cited by 971 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Tensor decompositions for learning latent variable models
(2012) · arXiv (Cornell University) · doi:10.5555/2627435.2697055 · Cited by 779 · see citing works
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · see citing works
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A. B., Urtasun, R., & Fidler, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.501 · Cited by 646 · see citing works
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 600 · see citing works
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works
Self-citationTowards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · see citing works
Self-citationDas, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual GroundingPreprint
Fukui, A., Park, D. H., Yang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01847 · Cited by 388 · see citing works
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350 · see citing works
Self-citationDas, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00279 · Cited by 288 · see citing works
Yang, X., Yumer, E., Asente, P., Kraley, M., Kifer, D., & Giles, C. L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.462 · Cited by 265 · see citing works
Missing Modalities Imputation via Cascaded Residual Autoencoder
Tran, L., Liu, X., Zhou, J., & Jin, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.528 · Cited by 256 · see citing works
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · see citing works
Self-citationWeighted Low-Rank Tensor Recovery for Hyperspectral Image Restoration
Chang, Y., Yan, L., Zhao, X.-L., Fang, H., Zhang, Z., & Zhong, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2983102 · Cited by 224 · see citing works
Hyperspectral Image Restoration Using Low-Rank Tensor Recovery
Fan, H., Chen, Y., Guo, Y., Zhang, H., & Kuang, G. (2017) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2017.2714338 · Cited by 185 · see citing works
Deep Adversarial Learning for Multi-Modality Missing Data Completion
Cai, L., Wang, Z., Gao, H., Shen, D., & Ji, S. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219963 · Cited by 180 · see citing works
Max-Margin Tensor Neural Network for Chinese Word Segmentation
Pei, W., Ge, T., & Chang, B. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1028 · Cited by 165 · see citing works
Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction
Schmerling, E., Leung, K., Vollprecht, W., & Pavone, M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460766 · Cited by 163 · see citing works
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
Low rank tensor completion for multiway visual data
Long, Z., Liu, Y., Chen, L., & Zhu, C. (2018) · Signal Processing · doi:10.1016/j.sigpro.2018.09.039 · Cited by 150 · see citing works
Improved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 142 · see citing works
Low-Rank Tensors for Scoring Dependency Structures
Lei, T., Xin, Y., Zhang, Y., Barzilay, R., & Jaakkola, T. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1130 · Cited by 107 · see citing works
Tensor Regression NetworksPreprint
Kossaifi, J., Lipton, Z. C., Kolbeinsson, A., Khanna, A., Furlanello, T., & Anandkumar, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08308 · Cited by 101 · see citing works
Interactive Multimodal Robot Programming
Iba, S., Paredis, C. J. J., & Khosla, P. K. (2005) · The International Journal of Robotics Research · doi:10.1177/0278364904049250 · Cited by 58 · see citing works
Tensor Rank: Some Lower and Upper Bounds
Alexeev, B., Forbes, M. A., & Tsimerman, J. (2011) · IEEE Conference on Computational Complexity (CCC) · doi:10.1109/ccc.2011.28 · Cited by 58 · see citing works
Yang, X., Yumer, E., Asente, P., Kraley, M., Kifer, D., & Giles, C. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02337 · Cited by 55 · see citing works
A Tucker Deep Computation Model for Mobile Multimedia Feature Learning
Zhang, Q., Yang, L. T., Liu, X., Chen, Z., & Li, P. (2017) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3063593 · Cited by 54 · see citing works
Extracting Low-Dimensional Latent Structure from Time Series in the Presence of Delays
Lakshmanan, K. C., Sadtler, P. T., Tyler‐Kabara, E. C., Batista, A. P., & Yu, B. M. (2015) · Neural Computation · doi:10.1162/neco_a_00759 · Cited by 40 · see citing works
End-to-end Multimodal Speech Recognition
Palaskar, S., Sanabria, R., & Metze, F. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462439 · Cited by 39 · see citing works
Learning Distributed Representations for Structured Output Prediction
Srikumar, V., & Manning, C. D. (2014) · neural information processing systems · Cited by 34 · see citing works
Tensor-Based Transductive Learning for Multimodality Video Semantic Concept Detection
Wu, F. Q., Liu, Y., & Zhuang, Y. (2009) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2009.2021724 · Cited by 32 · see citing works
Weighted Low-rank Tensor Recovery for Hyperspectral Image RestorationPreprint
Chang, Y., Yan, L., Fang, H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00192 · Cited by 31 · see citing works
Bounds on the ranks of some 3-tensors
Atkinson, M. D., & Lloyd, S. (1980) · Linear Algebra and its Applications · doi:10.1016/0024-3795(80)90202-5 · Cited by 27 · see citing works
Multimodal Compact Bilinear Pooling for Multimodal Neural Machine TranslationPreprint
Delbrouck, J.-B., & Dupont, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08084 · Cited by 27 · see citing works
Discriminative optical flow tensor for video semantic analysis
Gao, X., Yang, Y., Tao, D., & Li, X. (2008) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2008.08.007 · Cited by 20 · see citing works
Mihalcea, R. F. (2012) · Cited by 16 · see citing works
Statistical Machine Translation Features with Multitask Tensor Networks
Setiawan, H., Huang, Z., Devlin, J., Lamar, T., Zbib, R. M., Schwartz, R. M., & Makhoul, J. I. (2015) · doi:10.3115/v1/p15-1004 · Cited by 15 · see citing works
Computational Complexity of Tensor Nuclear Norm.Preprint
Friedland, S., & Lim, L. (2014) · arXiv (Cornell University) · Cited by 12 · see citing works
Rudnicky, A. I. (2005) · Text, speech and language technology · doi:10.1007/1-4020-3075-4_1 · Cited by 11 · see citing works
A General Model for Robust Tensor Factorization with Unknown NoisePreprint
Chen, X., Han, Z. J., Wang, Y., Zhao, Q., Meng, D., Lin, L., & Tang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06755 · Cited by 5 · see citing works
Self-citationBallico, E. (2013) · Linear and Multilinear Algebra · doi:10.1080/03081087.2013.839671 · Cited by 4 · see citing works
Analyzing multimodal time series as dynamical systems
Hidaka, S., & Yu, C. (2010) · doi:10.1145/1891903.1891968 · Cited by 3 · see citing works
Low Rank Tensor Completion for Multiway Visual DataPreprint
Long, Z., Liu, Y., Chen, L., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03967 · Cited by 3 · see citing works
Multimodal intent recognition for natural human-robotic interaction
Rossiter, J. (2011) · University of Birmingham Institutional Research Archive (University of Birmingham) · Cited by 2 · see citing works
Relations of the Nuclear Norms of a Tensor and its Matrix FlatteningsPreprint
Hu, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.2443 · Cited by 1 · see citing works
Tensor Product Generation Networks.Preprint
Huang, Q., Smolensky, P., He, X., Li Juan, D. E. N. G., & Wu, D. O. (2017) · arXiv (Cornell University) · Cited by 1 · see citing works
Tensor Rank: Some Lower and Upper BoundsPreprint
Alexeev, B., Forbes, M. A., & Tsimerman, J. (2011) · doi:10.48550/arxiv.1102.0072 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 9
- Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. 2014. Ten- sor decompositions for learning latent variable mod- els. J. Mach. Learn. Res., 15(1):2773-2832.
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In International Conference on Computer Vision (ICCV).
- Abhishek Das, Samyak Datta, Georgia Gkioxari, Ste- fan Lee, Devi Parikh, and Dhruv Batra. 2018. Em- bodied Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep -a col- laborative voice analysis repository for speech tech- nologies. In ICASSP. IEEE.
- Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshmi- narasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low- rank multimodal fusion with modality-specific fac- tors. In ACL.
- Rada Mihalcea. 2012. Multimodal sentiment analy- sis. In Proceedings of the 3rd Workshop in Com- putational Approaches to Subjectivity and Sentiment Analysis, WASSA '12, pages 1-1, Stroudsburg, PA, USA. Association for Computational Linguistics.
- Madhav Nimishakavi, Pratik Kumar Jawanpuria, and Bamdev Mishra. 2018. A dual framework for low- rank tensor completion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5484-5495.
- Xipeng Qiu and Xuanjing Huang. 2015. Convolutional neural tensor network architecture for community- based question answering. In Proceedings of the 24th International Conference on Artificial Intelli- gence, IJCAI'15, pages 1305-1311. AAAI Press.
- Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Ten- sor fusion network for multimodal sentiment analy- sis. In EMNLP, pages 1114-1125.
Cited by
Sorted by citations
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783 · see citing works
Multimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 368 · see citing works
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 283 · see citing works
Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis
Yuan, Z., Li, W., Xu, H., & Yu, W. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475585 · Cited by 211 · see citing works
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodFoundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
Self-citationMai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182 · see citing works
GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation
Lian, Z., Chen, L., Sun, L., Liu, B., & Tao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234553 · Cited by 165 · see citing works
Quantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143 · see citing works
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
M-SENA: An Integrated Platform for Multimodal Sentiment Analysis
Mao, H., Yuan, Z., Xu, H., Yu, W., Liu, Y., & Gao, K. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.20 · Cited by 108 · see citing works
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378570 · Cited by 96 · see citing works
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · see citing works
Gkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 94 · see citing works
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89 · see citing works
Adapted Dynamic Memory Network for Emotion Recognition in Conversation
Xing, S., Mai, S., & Hu, H. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3005660 · Cited by 87 · see citing works
Disentanglement Translation Network for multimodal sentiment analysis
Zeng, Y., Yan, W., Mai, S., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102031 · Cited by 84 · see citing works
Low Rank Fusion based Transformers for Multimodal Sequences
Sahay, S., Okur, E., Kumar, S. H., & Nachman, L. (2020) · doi:10.18653/v1/2020.challengehml-1.4 · Cited by 79 · see citing works
Noise Imitation Based Adversarial Training for Robust Multimodal Sentiment Analysis
Yuan, Z., Liu, Y., Xu, H., & Gao, K. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3267882 · Cited by 78 · see citing works
Quantum-inspired Neural Network for Conversational Emotion Recognition
Li, Q., Gkoumas, D., Sordoni, A., Nie, J., & Melucci, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17567 · Cited by 60 · see citing works
Learning to Learn Better Unimodal Representations via Adaptive Multimodal Meta-Learning
Ya, S., Mai, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3178231 · Cited by 59 · see citing works
Adaptive Fusion Techniques for Multimodal Data
Sahu, G., & Olga Vechtomova, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.275 · Cited by 47 · see citing works
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Zadeh, A. B., Cao, Y., Hessner, S., Liang, P. P., Poria, S., & Morency, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.141 · Cited by 46 · see citing works
Self-citationMultimodal Mutual Attention-Based Sentiment Analysis Framework Adapted to Complicated Contexts
He, L., Wang, Z., Wang, L., & Li, F. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3276075 · Cited by 45 · see citing works
Wang, Q., Wang, X., He, Q., Huang, J. S., Huang, H. L., Wang, P., Yu, T., & Zhang, M. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105342 · Cited by 38 · see citing works
Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542927 · Cited by 61 · see citing works
COM: Contrastive Masked-attention model for incomplete multimodal learning
Qian, S., & Wang, C. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.003 · Cited by 31 · see citing works
Wang, K., Shan, S., Dou, W., Wei, H., & Zhang, K. (2024) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2024.3494266 · Cited by 31 · see citing works
Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors
Wu, Y., Zhao, Y., Yang, H., Chen, S., Qin, B., Cao, X., & Zhao, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.109 · Cited by 33 · see citing works
Multimodal Sentiment Recognition With Multi-Task Learning
Zhang, S., Yin, C., & Yin, Z. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3224929 · Cited by 31 · see citing works
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26 · see citing works
Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling FusionPreprint
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13572 · Cited by 26 · see citing works
Li, Z., Liu, P., Pan, Y., Ding, W., Yu, J., Chen, H., Liu, W., Luo, Y., & Wang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128940 · Cited by 25 · see citing works
Dual Low-Rank Multimodal Fusion
Jin, T., Huang, S., Li, Y., & Zhang, Z. M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.35 · Cited by 24 · see citing works
Multimodal Reaction: Information Modulation for Cross-Modal Representation Learning
Zeng, Y., Mai, S., Yan, W., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3293335 · Cited by 24 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“Robustness: Real-world multimodal data is often imperfect as a result of missing entries, noise corruption, or missing modalities entirely, which calls for robust models that can still make accurate predictions despite only having access to noisy and missing signals [101, 123].”
BackgroundInfluentialSelf-citationSentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis
Fan, Q., Han, J., He, Y., Zheng, T., & Zheng, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.821 · Cited by 23 · see citing works
Towards robust sentiment analysis with multimodal interaction graph and hybrid contrastive learning
Gong, P., Liu, J., Zhang, X., Li, X., Wei, L., & He, H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111870 · Cited by 22 · see citing works
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
Liang, P. P., Wu, P., Ziyin, L., Morency, L.-P., & Salakhutdinov, R. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475247 · Cited by 22 · see citing works
Self-citationLi, H., Li, Z., Li, K., Rellermeyer, J. S., Chen, L. Y., & Li, K. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3047460 · Cited by 20 · see citing works
Injecting Multimodal Information Into Pre-Trained Language Model for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Xiong, A., & Hu, H. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3553149 · Cited by 21 · see citing works
Han, W., Chen, H., Kan, M., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.717 · Cited by 21 · see citing works
Huang, J., Lu, P., Sun, S., & Wang, F. (2023) · Electronics · doi:10.3390/electronics12163504 · Cited by 19 · see citing works
Structural health monitoring data cleaning based on Bayesian robust tensor learning
Yi, L., Ding, Y., Hou, J., Yue, Z., & Zhao, H. (2022) · Structural Health Monitoring · doi:10.1177/14759217221117238 · Cited by 17 · see citing works
Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition
Chen, C., & Zhang, P. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3640343 · Cited by 18 · see citing works
Meta Noise Adaption Framework for Multimodal Sentiment Analysis With Feature Noise
Yuan, Z., Zhang, B., Xu, H., & Gao, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3362600 · Cited by 19 · see citing works
TPFN: Applying Outer Product Along Time to Multimodal Sentiment Analysis Fusion on Incomplete Data
Li, B., Li, C., Duan, F., Zheng, N., & Zhao, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_26 · Cited by 15 · see citing works
Self-citationRethinking Missing Modality Learning from a Decoding Perspective
Jin, T., Cheng, X., Li, L., Wang, L., Wang, Y., & Zhao, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612291 · Cited by 12 · see citing works
Deng, Y., Bian, J., Wu, S., Lai, J., & Xie, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102711 · Cited by 12 · see citing works
Missing Structural Health Monitoring Data Recovery Based on Bayesian Matrix Factorization
Sun, S., Jiao, S., Hu, Q., Wang, Z., Xia, Z., Ding, Y., & Yi, L. (2023) · Sustainability · doi:10.3390/su15042951 · Cited by 10 · see citing works
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning
Mai, S., Ya, S., & Hu, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.209 · Cited by 13 · see citing works
SGD_Tucker: A Novel Stochastic Optimization Strategy for Parallel Sparse Tucker DecompositionPreprint
Li, H., Li, Z., Li, K., Rellermeyer, J. S., Chen, L. Y., & Li, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03550 · Cited by 10 · see citing works
Zhang, Y., Zhong, H., Alhusaini, N., Chen, G., & Wu, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113121 · Cited by 9 · see citing works
Deep Multimodal Fusion with Corrupted Spatio-Temporal Data Using Fuzzy Regularization
Altinses, D., & Schwung, A. (2023) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon51785.2023.10312522 · Cited by 8 · see citing works
Hierarchical Knowledge Stripping for Multimodal Sentiment Analysis
Xiong, A., Zeng, Y., & Hu, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3456117 · Cited by 7 · see citing works
Sun, N., Zhang, W., Zheng, W., Liu, J., Chai, L., & Wu, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3612369 · Cited by 7 · see citing works
Wang, P., Liu, S., & Chen, J. (2024) · Applied Sciences · doi:10.3390/app14051934 · Cited by 8 · see citing works
Low-Rank Tensor Fusion for Enhanced Deep Learning-Based Multimodal Brain Age Estimation
Liu, X., Zheng, G., Beheshti, I., Ji, S., Gou, Z., & Cui, W.-K. (2024) · Brain Sciences · doi:10.3390/brainsci14121252 · Cited by 7 · see citing works
Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor ProjectionsPreprint
Tóth, C., Bonnier, P., & Oberhauser, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07027 · Cited by 6 · see citing works
Yan, K., Lai, P., Zheng, X., Yang, Y., Ren, Y. C., Badarch, T., & Chen, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112923 · Cited by 6 · see citing works
SGIR: Star Graph-Based Interaction for Efficient and Robust Multimodal Representation
Ding, Z., Lan, G., Song, Y., & Yang, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321404 · Cited by 5 · see citing works
MultiViz: Towards User-Centric Visualizations and Interpretations of Multimodal Models
Liang, P. P., Lyu, Y., Chhablani, G., Jain, N., Deng, Z., Wang, X., Morency, L.-P., & Salakhutdinov, R. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585604 · Cited by 4 · see citing works
Self-citationChen, G., Qian, Z., Qiu, S., Zhang, D., & Zhou, R. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105322 · Cited by 5 · see citing works
Towards Multimodal Sentiment Analysis Inspired by the Quantum Theoretical Framework
Li, Q., Stefani, A., Toto, G., Buccio, E. D., & Melucci, M. (2020) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr49039.2020.00044 · Cited by 4 · see citing works
CMFF: A cross-modal multi-layer feature fusion network for multimodal sentiment analysis
Zheng, S., Zhang, J., Deng, Y., & Chen, L. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113868 · Cited by 3 · see citing works
A Cross-Modal Adaptive Masked Autoencoder for Decoding Emotions With Multimodal Data
Cheng, C., Liu, W., Zhang, Y., Feng, L., & Jia, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3415613 · Cited by 3 · see citing works
A Comprehensive Survey on Multimodal Sentiment Analysis
Dhal, D., Mohanty, J. P., & Rout, S. S. (2024) · International Conference on Intelligent Computing and Sustainable Innovations in Technology (IC-SIT) · doi:10.1109/ic-sit63503.2024.10862066 · Cited by 3 · see citing works
Boundary-guided distillation with uncertainty-driven temperature for incomplete multimodal learning
Xu, Y., Chen, Y., & Xie, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113718 · Cited by 3 · see citing works
Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents
Okur, E., Kumar, S. H., Sahay, S., & Nachman, L. (2020) · doi:10.18653/v1/2020.challengehml-1.7 · Cited by 3 · see citing works
Emerging Trends of Multimodal Research in Vision and Language.Preprint
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2020) · arXiv (Cornell University) · Cited by 3 · see citing works
Tan, W., & Zhang, W. (2025) · Sensors · doi:10.3390/s25237109 · Cited by 2 · see citing works
TB-MEN: Text-Centric Bidirectional Modality Enhancement Network for Multimodal Sentiment Analysis
Zhang, J., Yin, M., Wei, G., Shi, D., & Guo, Y. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3660469 · Cited by 2 · see citing works
Recent Advancements and Challenges in Multimodal Sentiment Analysis: A Survey
Dong, Z.-X., & Liu, H. (2023) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc58545.2023.10327944 · Cited by 2 · see citing works
Adversarial Training and Cross-modal Feature Fusion in Multimodal Sentiment Analysis
Li, J., Lin, C., Wang, H., Zhi, Y., Chen, J., & Huang, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890023 · Cited by 2 · see citing works
Li, C., Xie, L., Wang, X., Pan, H., & Wang, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01758-w · Cited by 2 · see citing works
Ren, P., Chen, X., Sun, L., & Sun, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00790 · Cited by 2 · see citing works
Cross-modal Memory Fusion Network for Multimodal Sequential Learning with Missing Values
Lin, C., Ho, J. C., & Agichtein, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72240-1_30 · Cited by 1 · see citing works
Blending of Predictions Boosts Understanding for Multimodal Advertisements
Alekseev, A., Savchenko, A., Tutubalina, E. V., Myasnikov, E. V., & Nikolenko, S. (2024) · Journal of Mathematical Sciences · doi:10.1007/s10958-024-07438-w · Cited by 1 · see citing works
SAFCN: Self-enhancing attention fusion contrastive network for multimodal sentiment analysis
Chen, Y., Yang, L., Wu, T. X., Tan, Y., Li, X., Li, Z., & Zhu, W.-J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131212 · Cited by 1 · see citing works
Embedding physics into deep learning for modeling spatiotemporal systems
Ren, P. (2022) · doi:10.17760/d20467268 · Cited by 1 · see citing works
Tucker Product-Based Dynamical Systems
Chen, C. (2024) · Synthesis lectures on mathematics and statistics · doi:10.1007/978-3-031-54505-4_2
Topologically Consistent Prototype Network for Incomplete Multimodal Learning
Wang, Y., Zhang, X.-Y., & Liu, C. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10448-8
QSMT: Query-Shared Multimodal Transformer for Multimodal Sentiment Analysis
Liu, H., Liu, Y., Ding, J., & Zhang, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8180-8_9
Zi, L., Lv, H., & Cong, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9946-9_31
Qian, B., Ren, Q.-D.-E.-J., & Ji, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0020-8_11
DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment Analysis
Lv, X., Wen, Y., & Yu, H. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755508
MDR-MSA: multi-perspective decoupled representation learning for multimodal sentiment analysis
Du, J., Zhi, Y., Wang, H., Wang, K., & Li, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-02062-3
Dynamic Dominate-Modal-Aware Model for Multimodal Sentiment Analysis
Zhao, J., & Jin, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12657-3_7
Tang, Y.-C., Luo, N., Chen, Q., & Zheng, N. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5679-3_2
Zhao, Z., Zhao, C., Chen, K., Hu, X., Du, Y., & Ji, Y. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423007
Ye, M., Yin, M., & Guo, Y. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3687649
A Deep Multimodal Fusion Framework for Emotion-Aware Adaptive Cinematic Systems: A Systematic Review
Maishanu, M. B., Mariyam, S., & Sadia, H. (2026) · International Conference on Current Trends in Advanced Computing (ICCTAC) · doi:10.1109/icctac68277.2026.11524300 · Cited by 1 · see citing works
Decoupled Cross-Modal Fusion and Knowledge Injection for Multimodal Sentiment Analysis
Zhu, A., Hu, M. Q., Wang, X. J., Yan, X., & Ren, F. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3696459
Rethinking Medical High-Modality Learning Under Missingness—A Long-Tailed Distribution Perspective
Wu, C., Shuai, Z., & Shen, L. (2026) · doi:10.1109/cvprf71909.2026.00542
- Authors
- Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- RIKEN (JP)
- ORCID
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1569-1576
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2950442312
- Cited by
- 95
- References
- 52
- Citations by year
- 2026: 7 · 2025: 19 · 2024: 15 · 2023: 17 · 2022: 11 · 2021: 9 · 2020: 16 · 2019: 1
- FWCI
- 7.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763562, 1750439, 1722822)
- Nvidia
- Defense Advanced Research Projects Agency (FA8750)
- Samsung
- Japan Society for the Promotion of Science (17K00326)
- Air Force Research Laboratory (FA8750-18-C-0014)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Computational Mechanics › Sparse and Compressive Sensing Techniques
- Keywords
- low-rank tensor representation, multimodal time series, imperfect data, missing data, noise corruption, multimodal language processing, cross-modal correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
