Multimodal Language Analysis with Recurrent Multistage Fusion
Open accessAbstract
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities.Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions).In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.Crossmodal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages.Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks.The RMFN displays state-of-the-art performance in modeling human multimodal language across three public datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition.We provide visualizations to show that each stage of fusion focuses on a different subset of multimodal signals, learning increasingly discriminative multimodal representations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos.
“CMU-MOSEI is the largest dataset of sentence-level sentiment analysis and emotion recognition in real-world online videos [102, 183].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [102]
The Multimodal Sentiment Analysis section describes CMU-MOSI for sentiment analysis and IEMOCAP for emotion recognition, but does not describe CMU-MOSEI. A search for CMU-MOSEI found no passage establishing its size, sentence-level annotations, or status as the largest dataset. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,953 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Self-citationIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Statistical Inference for Probabilistic Functions of Finite State Markov Chains
Baum, L. E., & Petrie, T. (1966) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177699147 · Cited by 2,775 · see citing works
(2011) · Cited by 2,752 · see citing works
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567 · see citing works
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · see citing works
Self-citationDomain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · Cited by 1,569 · see citing works
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,445 · see citing works
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · see citing works
Self-citationDeep Convolutional Neural Networks for Sentiment Analysis of Short Texts
dos Santos, C., & de Bayser, M. G. (2014) · International Conference on Computational Linguistics · Cited by 1,162 · see citing works
A Survey on Multi-view LearningPreprint
Xu, C., Tao, D., & Xu, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.5634 · Cited by 983 · see citing works
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · see citing works
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855 · see citing works
Self-citationMultimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · see citing works
Self-citationSpeech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-57 · Cited by 824 · see citing works
A new view of language acquisition
Kuhl, P. K. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.97.22.11850 · Cited by 773 · see citing works
Deep Unordered Composition Rivals Syntactic Methods for Text Classification
Iyyer, M., Manjunatha, V., Boyd-Graber, J. L., & Daumé, H. (2015) · doi:10.3115/v1/p15-1162 · Cited by 729 · see citing works
(1999) · CRC Press eBooks · doi:10.1201/9781420049176 · Cited by 717 · see citing works
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · see citing works
TrueSkill™: A Bayesian Skill Rating System
Herbrich, R., Minka, T., & Graepel, T. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0076 · Cited by 693 · see citing works
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · Proceedings · doi:10.1109/icdm.2016.0055 · Cited by 624 · see citing works
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 600 · see citing works
Poria, S., Cambria, E., & Gelbukh, A. F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1303 · Cited by 553 · see citing works
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.12024 · Cited by 535 · see citing works
Self-citationA Simple Way to Initialize Recurrent Networks of Rectified Linear UnitsPreprint
Le, Q. V., Jaitly, N., & Hinton, G. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00941 · Cited by 521 · see citing works
Axiom: A Householder-Parameterized Pure Unitary RNN for Long-Range Sequence ModelingPreprint
Arjovsky, M., Shah, A., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06464 · Cited by 510 · see citing works
Hidden Conditional Random Fields
Quattoni, A., Wang, S., Morency, L., Collins, M., & Darrell, T. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1124 · Cited by 489 · see citing works
Self-citationTowards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · see citing works
Self-citationRecurrent neural networks: design and applications
Stoianov, I. (1999) · Cited by 442 · see citing works
Latent-Dynamic Discriminative Models for Continuous Gesture Recognition
Morency, L., Quattoni, A., & Darrell, T. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383299 · Cited by 376 · see citing works
Self-citationMultimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · see citing works
Self-citationLearning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works
Self-citationNeural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304 · see citing works
Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification
Chen, X., Sun, Y., Athiwaratkun, B., Cardie, C., & Weinberger, K. Q. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00039 · Cited by 293 · see citing works
Deep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2993176 · Cited by 247 · see citing works
Self-citationRecurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 243 · see citing works
Multimodal Machine Learning: A Survey and TaxonomyPreprint
Baltrušaitis, T., Ahuja, C., & Morency, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09406 · Cited by 152 · see citing works
Self-citationExtending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 138 · see citing works
Self-citationReal-Time Speech Emotion and Sentiment Recognition for Interactive Dialogue Systems
Bertero, D., Siddique, F. B., Wu, C.-S., Wan, Y., Chan, R. H. Y., & Fung, P. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1110 · Cited by 131 · see citing works
Computational Analysis of Persuasiveness in Social Multimedia
Park, S., Shim, H. S., Chatterjee, M., Sagae, K., & Morency, L. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663260 · Cited by 127 · see citing works
Self-citationTaylor, P., Hobbs, J. N., Burroni, J., & Siegelmann, H. T. (2015) · Scientific Reports · doi:10.1038/srep18112 · Cited by 122 · see citing works
Learning Word-Like Units from Joint Audio-Visual Analysis
Harwath, D. F., & Glass, J. R. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1047 · Cited by 102 · see citing works
Action Recognition by Hierarchical Sequence Summarization
Song, Y., Morency, L., & Davis, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.457 · Cited by 97 · see citing works
Self-citationMulti-view latent variable discriminative models for action recognition
Song, Y., Morency, L., & Davis, R. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247918 · Cited by 81 · see citing works
Self-citationSeq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 76 · see citing works
Self-citationAdversarial Deep Averaging Networks for Cross-Lingual Sentiment ClassificationPreprint
Chen, X., Sun, Y., Athiwaratkun, B., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01614 · Cited by 75 · see citing works
Misawa, S., Taniguchi, M., Miura, Y., & Ohkuma, T. (2017) · Workshop on Subword and Character Level Models in NLP (SCLeM) · doi:10.18653/v1/w17-4114 · Cited by 67 · see citing works
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech
Kamper, H., Settle, S., Shakhnarovich, G., & Livescu, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-502 · Cited by 65 · see citing works
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech RecognitionPreprint
Soltau, H., Liao, H., & Sak, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09975 · Cited by 56 · see citing works
Multimodal Local-Global Ranking Fusion for Emotion Recognition
Liang, P. P., Zadeh, A., & Morency, L. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243019 · Cited by 45 · see citing works
Self-citationEmergence of multimodal action representations from neural network self-organization
Parisi, G. I., Tani, J., Weber, C., & Wermter, S. (2016) · Cognitive Systems Research · doi:10.1016/j.cogsys.2016.08.002 · Cited by 41 · see citing works
Visual features for context-aware speech recognition
Gupta, A., Miao, Y., Neves, L., & Metze, F. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953112 · Cited by 37 · see citing works
Reusing Neural Speech Representations for Auditory Emotion RecognitionPreprint
Lakomkin, E., Weber, C., Magg, S., & Wermter, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.11508 · Cited by 32 · see citing works
Multimodal Sentiment Analysis Using Deep Neural Networks
Abburi, H., Prasath, R., Shrivastava, M., & Gangashetty, S. V. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-58130-9_6 · Cited by 31 · see citing works
Wang, H., Meghawat, A., Morency, L., & Xing, E. P. (2016) · arXiv (Cornell University) · Cited by 29 · see citing works
Self-citationLook, listen, and decode: Multimodal speech recognition with images
Sun, F., Harwath, D. F., & Glass, J. (2016) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2016.7846320 · Cited by 29 · see citing works
Select-Additive Learning: Improving Generalization in Multimodal Sentiment AnalysisPreprint
Wang, H., Meghawat, A., Morency, L., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.05244 · Cited by 17 · see citing works
Self-citationAFFEC TIVE COMPUTING AND SENTIMENT ANALYSIS Affective Computing and Sentiment Analysis
Cambria, E., Das, D., Bandyopadhyay, S., & Feraco, A. (2016) · Socio-affective computing · doi:10.1007/978-3-319-55394-8_1 · Cited by 14 · see citing works
(2015) · doi:10.4230/oasics.slate.2021.8 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collabo- rative voice analysis repository for speech technolo- gies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 960-964. IEEE.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997b. Long short-term memory. Neural Comput., 9(8):1735-1780.
- Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional lstm-crf models for sequence tagging. CoRR, abs/1508.01991. · arXiv
- Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multi- modal deep learning. In ICML, pages 689-696.
- ACL.
- In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Informa- tion Processing Systems 25, pages 2222-2230.
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Re- current Highway Networks. arXiv preprint arXiv:1607.03474. · arXiv
Cited by
Sorted by citations
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · see citing works
Self-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073 · see citing works
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783 · see citing works
Integrating Multimodal Information in Large Pretrained Transformers
Rahman, W., Hasan, M. K., Lee, S., Zadeh, A. B., Mao, C., Morency, L., & Hoque, M. E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.214 · Cited by 743 · see citing works
Self-citationWords Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 493 · see citing works
Self-citationSun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410 · see citing works
Multimodal sentiment analysis based on fusion methods: A survey
Zhu, L., Zhu, Z., Zhang, C., Xu, Y., & Kong, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.02.028 · Cited by 368 · see citing works
Mai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237 · see citing works
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224 · see citing works
“Liang et al. [113] presented Recurrent Multistage Fusion Network (RMFN), which decomposes the fusion problem into multiple stages.”
BackgroundA Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
Ma, H., Wang, J., Lin, H., Zhang, B., Zhang, Y., & Xu, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3271019 · Cited by 188 · see citing works
Mai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182 · see citing works
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174 · see citing works
Self-citationTMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis
Huang, J., Zhou, J. W., Tang, Z., Lin, J., & Chen, C. Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111346 · Cited by 162 · see citing works
Towards Debiasing Sentence Representations
Liang, P. P., Li, I., Zheng, E., Lim, Y. C., Salakhutdinov, R., & Morency, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.488 · Cited by 163 · see citing works
Self-citationMISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisPreprint
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03545 · Cited by 154 · see citing works
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · see citing works
Cross-Modal Multitask Transformer for End-to-End Multimodal Aspect-Based Sentiment Analysis
Yang, L., Na, J., & Yu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103038 · Cited by 149 · see citing works
Hierarchical Interactive Multimodal Transformer for Aspect-Based Multimodal Sentiment Analysis
Yu, J., Chen, K., & Xia, R. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3171091 · Cited by 148 · see citing works
AOBERT: All-modalities-in-One BERT for multimodal sentiment analysis
Kim, K.-H., & Park, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.022 · Cited by 148 · see citing works
Multimodal Video Sentiment Analysis Using Deep Learning Approaches, a Survey
Abdu, S. A., Yousef, A. H., & Salem, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.003 · Cited by 147 · see citing works
Quantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143 · see citing works
Yang, K., Xu, H., & Gao, K. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413690 · Cited by 132 · see citing works
Emotion recognition based on brain-like multimodal hierarchical perception
Zhu, X., Huang, Y., Wang, X., & Wang, R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17347-w · Cited by 115 · see citing works
Transformer-based Multimodal Information Fusion for Facial Expression Analysis
Zhang, W., Qiu, F., Wang, S., Zeng, H., Zhang, Z., An, R., Ma, B., & Ding, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00271 · Cited by 109 · see citing works
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
Yang, J., Wang, Y., Yi, R., Zhu, Y., Rehman, A., Zadeh, A., Poria, S., & Morency, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.79 · Cited by 104 · see citing works
Self-citationSocial-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Zadeh, A., Chan, M., Liang, P. P., Tong, E., & Morency, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00901 · Cited by 98 · see citing works
Self-citationLearning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95 · see citing works
Self-citationMultimodal sentiment analysis: A survey
Lai, S., Hu, X., Xu, H., Ren, Z., & Liu, Z. (2023) · Displays · doi:10.1016/j.displa.2023.102563 · Cited by 106 · see citing works
Self-citationMai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · see citing works
Gkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 94 · see citing works
SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition
Lian, Z., Liu, B., & Tao, J. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3141237 · Cited by 89 · see citing works
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89 · see citing works
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 91 · see citing works
BAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis
Tang, J., Liu, D., Jin, X., Peng, Y. K., Zhao, Q., Ding, Y., & Kong, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3218018 · Cited by 90 · see citing works
PS-Mixer: A Polar-Vector and Strength-Vector Mixer Model for Multimodal Sentiment Analysis
Lin, H., Zhang, P., Ling, J., Yang, Z., Lee, L., & Liu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103229 · Cited by 84 · see citing works
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Chen, X., & Zhou, M. (2020) · arXiv (Cornell University) · Cited by 82 · see citing works
Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition
Zhang, F., Li, X., Lim, C. P., Hua, Q., Dong, C.-R., & Zhai, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.07.006 · Cited by 81 · see citing works
Al-Saadawi, H. F. T., Daş, B., & Daş, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124852 · Cited by 81 · see citing works
Multimodal emotion recognition based on audio and text by using hybrid attention networks
Zhang, S., Yang, Y., Chen, C., Liu, R., Tao, X., Guo, W., Xu, Y., & Zhao, X. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105052 · Cited by 76 · see citing works
Neural Networks Fail to Learn Periodic Functions and How to Fix ItPreprint
Ziyin, L., Hartwig, T., & Ueda, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08195 · Cited by 75 · see citing works
Humor Knowledge Enriched Transformer for Understanding Multimodal Humor
Hasan, M. K., Lee, S., Rahman, W., Zadeh, A., Mihalcea, R. F., Morency, L., & Hoque, M. E. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17534 · Cited by 76 · see citing works
Self-citationZeng, Y., Li, Z., Tang, Z., Chen, Z., & Ma, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119240 · Cited by 74 · see citing works
Multi-Task Momentum Distillation for Multimodal Sentiment Analysis
Lin, R., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3282410 · Cited by 71 · see citing works
Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension
Mai, S., Hu, H., Jia, X., & Xing, S. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3000510 · Cited by 66 · see citing works
Luo, X., Jia, N., Ouyang, E., & Fang, Z. (2024) · Strategic Management Journal · doi:10.1002/smj.3597 · Cited by 68 · see citing works
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
Guo, Z., Jin, T., & Zhao, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.94 · Cited by 77 · see citing works
Learning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking Factor
Ya, S., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3101421 · Cited by 65 · see citing works
Quantum-inspired Neural Network for Conversational Emotion Recognition
Li, Q., Gkoumas, D., Sordoni, A., Nie, J., & Melucci, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17567 · Cited by 60 · see citing works
MEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis
Qi, Q., Lin, L., Zhang, R., & Xue, C. (2022) · IEEE Access · doi:10.1109/access.2022.3157712 · Cited by 63 · see citing works
TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis
Wang, F., Tian, S., Yu, L., Liu, J., Wang, J., Li, K., & Wang, Y. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10073-9 · Cited by 60 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
Excavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57 · see citing works
Lin, R., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3303711 · Cited by 56 · see citing works
SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis
Chen, M., & Li, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.93 · Cited by 54 · see citing works
Huan, R., Zhong, G., Chen, P., & Liang, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3338769 · Cited by 53 · see citing works
A comprehensive survey on deep learning-based approaches for multimodal sentiment analysis
Ghorbanali, A., & Sohrabi, M. K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10555-8 · Cited by 51 · see citing works
DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50 · see citing works
Xiao, L., Wu, X., Wu, W., Yang, J., & He, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747542 · Cited by 49 · see citing works
Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation
Chen, F., Shao, J., Zhu, A., Ouyang, D., Liu, X., & Shen, H. T. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3185119 · Cited by 49 · see citing works
Li, X., Liu, J., Xie, Y., Gong, P., Zhang, X., & He, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111126 · Cited by 49 · see citing works
Yang, D., Liu, Y., Huang, C., Li, M., Zhao, X. X., Wang, Y., Yang, K., Wang, Y., Zhai, P., & Zhang, L. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110370 · Cited by 48 · see citing works
Recent advances in deep learning based sentiment analysis
Yuan, J., Wu, Y., Lu, X., Zhao, Y., Qin, B., & Liu, T. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1634-3 · Cited by 48 · see citing works
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French
Zadeh, A. B., Cao, Y., Hessner, S., Liang, P. P., Poria, S., & Morency, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.141 · Cited by 46 · see citing works
Self-citationMultimodal Local-Global Ranking Fusion for Emotion Recognition
Liang, P. P., Zadeh, A., & Morency, L. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243019 · Cited by 45 · see citing works
Self-citationGraph Capsule Aggregation for Unaligned Multimodal Sequences
Wu, J., Mai, S., & Hu, H. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479931 · Cited by 44 · see citing works
Co-space Representation Interaction Network for multimodal sentiment analysis
Shi, H., Pu, Y., Zhao, Z., Huang, J., Zhou, D., Xu, D., & Cao, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111149 · Cited by 45 · see citing works
Zhang, Q., Shi, L., Liu, P., Zhu, Z., & Xu, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03343-4 · Cited by 43 · see citing works
RetractedA feature-based restoration dynamic interaction network for multimodal sentiment analysis
Zeng, Y., Li, Z., Chen, Z., & Ma, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107335 · Cited by 42 · see citing works
Foundations of Multimodal Co-learning
Zadeh, A., Liang, P. P., & Morency, L.-P. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.001 · Cited by 40 · see citing works
Self-citationHierarchically stacked graph convolution for emotion recognition in conversation
Wang, B., Dong, G., Zhao, Y., Li, R., Cao, Q., Hu, K., & Jiang, D. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110285 · Cited by 40 · see citing works
Factorized Multimodal Transformer for Multimodal Sequential LearningPreprint
Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., & Morency, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09826 · Cited by 45 · see citing works
Self-citationLearning speaker-independent multimodal representation for sentiment analysis
Wang, J., Wang, S., Lin, M., Xu, Z., & Guo, W. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.116 · Cited by 40 · see citing works
Learning from the global view: Supervised contrastive learning of multimodal representation
Mai, S., Zeng, Y., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101920 · Cited by 41 · see citing works
Token-disentangling Mutual Transformer for multimodal emotion recognition
Yin, G., Liu, Y., Liu, T., Zhang, H., Fang, F., Tang, C., & Jiang, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108348 · Cited by 37 · see citing works
Deep Gamblers: Learning to Abstain with Portfolio TheoryPreprint
Ziyin, L., Wang, Z., Liang, P. P., Salakhutdinov, R., Morency, L., & Ueda, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00208 · Cited by 40 · see citing works
Self-citationQuantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis
Gkoumas, D., Li, Q., Dehdashti, S., Melucci, M., Yu, Y., & Song, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16165 · Cited by 34 · see citing works
Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3542927 · Cited by 61 · see citing works
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
Dai, W., Liu, Z., Yu, T., & Fung, P. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.30 · Cited by 33 · see citing works
Self-citationDeng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3211197 · Cited by 31 · see citing works
Multimodal Sentiment Recognition With Multi-Task Learning
Zhang, S., Yin, C., & Yin, Z. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3224929 · Cited by 31 · see citing works
Coupled data/physics-driven framework for accurate and efficient structural response simulation
Zhai, G., Spencer, B. F., Yan, J., Liao, W., Gu, D., Contiguglia, C. P., Demartino, C., & Xu, Y. (2025) · Engineering Structures · doi:10.1016/j.engstruct.2025.119636 · Cited by 30 · see citing works
Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition
Gu, Y., Lyu, X., Sun, W., Li, W., Chen, S., Li, X., & Marsic, I. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351039 · Cited by 30 · see citing works
Hierarchical self-adaptation network for multimodal named entity recognition in social media
Tian, Y., Sun, X., Yu, H., Li, Y., & Fu, K. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.060 · Cited by 30 · see citing works
Hwang, Y., & Kim, J.-H. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.2 · Cited by 31 · see citing works
Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 29 · see citing works
Strong and Simple Baselines for Multimodal Utterance Embeddings
Liang, P. P., Lim, Y. C., Tsai, Y.-H. H., Salakhutdinov, R., & Morency, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1267 · Cited by 29 · see citing works
Self-citationMultistage linguistic conditioning of convolutional layers for speech emotion recognition
Triantafyllopoulos, A., Reichel, U. D., Liu, S., Huber, S., Eyben, F., & Schuller, B. W. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1072479 · Cited by 28 · see citing works
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26 · see citing works
GLoMo: Global-Local Modal Fusion for Multimodal Sentiment Analysis
Yan, Z., Zhang, Y., Hu, Z., Zhang, X., Deng, J., & Ren, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681527 · Cited by 27 · see citing works
Zeng, Y., Mai, S., & Hu, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.109 · Cited by 31 · see citing works
A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism
Qiu, K., Zhang, Y., Zhao, J., Zhang, S., Wang, Q., & Chen, F. (2024) · Electronics · doi:10.3390/electronics13101922 · Cited by 26 · see citing works
A Optimized BERT for Multimodal Sentiment Analysis
Wu, J., Zhu, T., Zhu, J., Li, T., & Wang, C. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3566126 · Cited by 26 · see citing works
Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling FusionPreprint
Mai, S., Xing, S., He, J., Zeng, Y., & Hu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13572 · Cited by 26 · see citing works
Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis
He, J., Yanga, H., Zhang, C., Chen, H., & Xua, Y. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/2105593 · Cited by 25 · see citing works
Multimodal Learning for Multi-omics: A Survey
Tabakhi, S., Suvon, M. N. I., Ahadian, P., & Lu, H. (2022) · World Scientific Annual Review of Artificial Intelligence · doi:10.1142/s2811032322500047 · Cited by 25 · see citing works
Deep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 28 · see citing works
Multimodal Reaction: Information Modulation for Cross-Modal Representation Learning
Zeng, Y., Mai, S., Yan, W., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3293335 · Cited by 24 · see citing works
Incorporating End-to-End Speech Recognition Models for Sentiment Analysis
Lakomkin, E., Zamani, M. A., Weber, C., Magg, S., & Wermter, S. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794468 · Cited by 24 · see citing works
Cross-modal dynamic convolution for multi-modal emotion recognition
Wen, H., You, S., & Fu, Y. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103178 · Cited by 23 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“…speech recognition [48] to the recent explosion of interest in language, vision, and video understanding [48] for applications such as multimedia [102, 116], affective computing [101, 127], robotics [84, 91], finance [70], dialogue [126], human-computer interaction [47, 117], and healthcare [51,…”
BackgroundSelf-citation
- Authors
- Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 150-161
- OpenAlex
- W2886193235
- Cited by
- 230
- References
- 69
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 43 · 2023: 37 · 2022: 39 · 2021: 31 · 2020: 32 · 2019: 15 · 2018: 2
- FWCI
- 6.03
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Samsung
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- cross-modal interaction, recurrent neural networks, multimodal sentiment analysis, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review