The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Abstract
Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Mining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Feature-rich part-of-speech tagging with a cyclic dependency network
Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073478 · Cited by 2,844
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,386
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,204
Self-citation
Cited by
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Self-citationDeep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Self-citationA survey on sentiment analysis methods, applications, and challenges
Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10144-1 · Cited by 1,528
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433
Self-citationAffective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,204
Self-citationEfficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Self-citationSentiment analysis using deep learning architectures: a review
Yadav, A., & Vishwakarma, D. K. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09794-5 · Cited by 841
Deep Learning-Based Document Modeling for Personality Detection from Text
Majumder, N., Poria, S., Gelbukh, A., & Cambria, E. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.23 · Cited by 675
Self-citationA survey of sentiment analysis in social media
Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-018-1236-4 · Cited by 577
Sentiment Analysis and Opinion Mining
Zhang, L., & Liu, B. (2023) · Encyclopedia of Machine Learning and Data Science · doi:10.1007/978-1-4899-7502-7_907-2 · Cited by 518
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505
Self-citationSentiment Analysis Is a Big Suitcase
Cambria, E., Poria, S., Gelbukh, A., & Thelwall, M. (2017) · IEEE Intelligent Systems · doi:10.1109/mis.2017.4531228 · Cited by 439
Self-citationSun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
Natural language based financial forecasting: a survey
Xing, F. Z., Cambria, E., & Welsch, R. E. (2018) · Artificial Intelligence Review · doi:10.1007/s10462-017-9588-9 · Cited by 398
Self-citationHow to get pavement distress detection ready for deep learning? A systematic approach
Eisenbach, M., Stricker, R., Seichter, D., Amende, K., Debes, K., Sesselmann, M., Ebersbach, D., Stoeckert, U., & Gross, H.-M. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966101 · Cited by 391
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 383
Self-citationAffective Computing and Sentiment Analysis
Cambria, E., Das, D., Bandyopadhyay, S., & Feraco, A. (2017) · Socio-Affective Computing · doi:10.1007/978-3-319-55394-8_1 · Cited by 342
Self-citationCARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1404 · Cited by 308
SMIL: Multimodal Learning with Severely Missing Modality
Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., & Peng, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16330 · Cited by 332
“Early fusion is a common method which fuses different modalities by feature concatenation, and it has been widely adopted in previous studies (Wang et al. 2017; Poria et al. 2016).”
MethodImage–text sentiment analysis via deep multimodal attentive fusion
Huang, F., Zhang, X., Zhao, Z., Xu, J., & Li, Z. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.01.019 · Cited by 296
Transformer based Deep Intelligent Contextual Embedding for Twitter sentiment analysis
Naseem, U., Razzak, I., Musial, K., & Imran, M. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.06.050 · Cited by 296
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
Poria, S., Cambria, E., Hazarika, D., & Vij, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.08815 · Cited by 292
Self-citationHybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Zheng, S., & Hu, H. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3172360 · Cited by 277
Emotion detection from text and speech: a survey
Sailunaz, K., Dhaliwal, M., Rokne, J., & Alhajj, R. (2018) · Social Network Analysis and Mining · doi:10.1007/s13278-018-0505-2 · Cited by 260
Mai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237
Multimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230
Are Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198
“Simple methods like concatenation have been widely studied in [32,42].”
MethodMai, S., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171679 · Cited by 182
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Mai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
LIGHT-SERNET: A Lightweight Fully Convolutional Neural Network for Speech Emotion Recognition
Aftab, A., Morsali, A., Ghaemmaghami, S., & Champagne, B. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746679 · Cited by 130
A survey on sentiment analysis of scientific citations
Yousif, A., Niu, Z., Tarus, J. K., & Ahmad, A. (2019) · Artificial Intelligence Review · doi:10.1007/s10462-017-9597-8 · Cited by 104
Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89
Mai, S., Xing, S., & Hu, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3068598 · Cited by 89
Dilated Residual Network with Multi-head Self-attention for Speech Emotion Recognition
Li, R., Wu, Z., Jia, J., Zhao, S., & Meng, H. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682154 · Cited by 82
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58
Chen, L., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3236320 · Cited by 53
(2023) · doi:10.1109/cvpr52729.2023.00627 · Cited by 28
Mai, S., Hu, H., & Xing, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3082398 · Cited by 26
EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge
Palash, M., & Bhargava, B. (2023) · arXiv · doi:10.48550/arxiv.2306.08657 · Cited by 25
“Similarly, in [31], [32], [33], [34], [35], [36] authors proposed various methods to identify the emotion based on multimodal data.”
Background
Metadata
- Authors
- Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A.
- Venue
- 2016 IEEE 16th International Conference on Data Mining (ICDM)
- Publisher
- IEEE
- Published
- 2016-12
- Pages
- 439-448
- ISSN
- 1550-4786 (ISSN-L)
- OpenAlex
- W2583643061
- Cited by
- 595
- References
- 36
- FWCI
- 36.12
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.