The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Image-Text Multimodal Emotion Classification via Multi-View Attentional Network
Abstract
Compared with single-modal content, multimodal data can express users’ feelings and sentiments more vividly and interestingly. Therefore, multimodal sentiment analysis has become a popular research topic. However, most existing methods either learn modal sentiment feature independently, without considering their correlations, or they simply integrate multimodal features. In addition, most publicly available multimodal datasets are labeled by sentiment polarities, while the emotions expressed by users are specific. Based on this observation, in this paper, we build a large-scale image-text emotion dataset (i.e., labeled by different emotions), called TumEmo, with more than190,000instances from Tumblr.1We further propose a novel multimodal emotion analysis model based on the Multi-view Attentional Network (MVAN), which utilizes a memory network that is continually updated to obtain the deep semantic features of image-text. The model includes three stages: feature mapping, interactive learning, and feature fusion. In the feature mapping stage, we leverage image features from an object viewpoint and a scene viewpoint to capture effective information for multimodal emotion analysis. Then, an interactive learning mechanism is adopted that uses the memory network; this mechanism extracts single-modal emotion features and interactively models the cross-view dependencies between the image and text. In the feature fusion stage, multiple features are deeply fused using a multilayer perceptron and a stacking-pooling module. The experimental results on the MVSA-Single, MVSA-Multiple, and TumEmo datasets show that the proposed MVAN outperforms strong baseline models by large margins.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Lexicon-based methods for sentiment analysis
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011) · Computational Linguistics · doi:10.1162/coli_a_00049 · Cited by 3,301
Recurrent Convolutional Neural Networks for Text Classification
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9513 · Cited by 2,280
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification
Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., & Xu, B. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2034 · Cited by 2,012
Deep learning for sentiment analysis: A survey
Zhang, L., Wang, S., & Liu, B. (2018) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1253 · Cited by 1,928
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification
Tang, D., Qin, B., & Liu, T. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1167 · Cited by 1,519
A Survey on Multi-view Learning
Xu, C., Tao, D., & Xu, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.5634 · Cited by 942
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.41 · Cited by 871
Large-scale visual sentiment ontology and detectors using adjective noun pairs
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S.-F. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502282 · Cited by 851
Hadamard Product for Low-rank Bilinear PoolingPreprint
Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04325 · Cited by 780
A survey of multimodal sentiment analysis
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.-F., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 601
A survey of sentiment analysis in social media
Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-018-1236-4 · Cited by 577
Poria, S., Cambria, E., & Gelbukh, A. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1303 · Cited by 533
Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach
Wang, X., Wei, F., Liu, X., Zhou, M., & Zhang, M. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063726 · Cited by 393
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 383
Image–text sentiment analysis via deep multimodal attentive fusion
Huang, F., Zhang, X., Zhao, Z., Xu, J., & Li, Z. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.01.019 · Cited by 296
Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion
Abdi, A., Shamsuddin, S. M., Hasan, S., & Piran, J. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.02.018 · Cited by 256
(2013) · Cited by 236
Sentiment Analysis on Multi-View Social Data
Niu, T., Zhu, S., Pang, L., & El Saddik, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-27674-8_2 · Cited by 231
Analyzing and predicting sentiment of images on the social web
Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874060 · Cited by 218
Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis
Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., & Morency, L.-P. (2017) · 2017 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2017.134 · Cited by 193
You, Q., Luo, J., Jin, H., & Yang, J. (2016) · Proceedings of the Ninth ACM International Conference on Web Search and Data Mining · doi:10.1145/2835776.2835779 · Cited by 187
MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis
Xu, N., & Mao, W. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3133142 · Cited by 184
A Co-Memory Network for Multimodal Sentiment Analysis
Xu, N., Mao, W., & Chen, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210093 · Cited by 181
Boosting image sentiment analysis with visual attention
Song, K., Yao, T., Ling, Q., & Mei, T. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.104 · Cited by 162
Visual Sentiment Prediction with Deep Convolutional Neural NetworksPreprint
Xu, C., Cetintas, S., Lee, K.-C., & Li, L.-J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5731 · Cited by 159
Visual Sentiment Analysis by Attending on Local Image Regions
You, Q., Jin, H., & Luo, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10501 · Cited by 158
Multi-level region-based Convolutional Neural Network for image emotion classification
Rao, T., Li, X., Zhang, H., & Xu, M. G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.12.053 · Cited by 148
Multimodal Sentiment Analysis To Explore the Structure of Emotions
Hu, A., & Flaxman, S. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219853 · Cited by 137
You, Q., Cao, L., Jin, H., & Luo, J. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964288 · Cited by 129
Analyzing multimodal public sentiment based on hierarchical semantic attentional network
Xu, N. (2017) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi.2017.8004895 · Cited by 109
Multimodal Sentiment Analysis: A Survey and Comparison
Kaur, R., & Kautish, S. (2019) · International Journal of Service Science Management Engineering and Technology · doi:10.4018/ijssmet.2019040103 · Cited by 93
Visual-textual sentiment classification with bi-directional multi-level attention networks
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.04.018 · Cited by 75
Cross-Modality Microblog Sentiment Prediction via Bi-Layer Multimodal Hypergraph Learning
Ji, R., Chen, F., Cao, L., & Gao, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2867718 · Cited by 74
Towards building a high-quality microblog-specific Chinese sentiment lexicon
Wu, F., Huang, Y., Song, Y., & Liu, S. (2016) · Decision Support Systems · doi:10.1016/j.dss.2016.04.007 · Cited by 72
Sun, S., Mao, L., Dong, Z., & Wu, L. (2019) · Springer eBooks · doi:10.1007/978-981-13-3029-2 · Cited by 61
Learning visual and textual representations for multimodal matching and classification
Liu, Y., Liu, L., Guo, Y., & Lew, M. S. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.001 · Cited by 44
Multimodal Learning of Social Image Representation by Exploiting Social Relations
Huang, F., Zhang, X., Xu, J. J., Zhao, Z., & Li, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2896100 · Cited by 39
Sentiment Classification Using Negative and Intensive Sentiment Supplement Information
Chen, X., Rao, Y., Xie, H., Wang, F. L., Zhao, Y., & Yin, J. (2019) · Data Science and Engineering · doi:10.1007/s41019-019-0094-8 · Cited by 33
Learning Joint Multimodal Representation with Adversarial Attention Networks
Huang, F., Zhang, X., & Li, Z. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240614 · Cited by 22
Generalized Hadamard-Product Fusion Operators for Visual Question Answering
Duke, B., & Taylor, G. W. (2018) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2018.00016 · Cited by 15
Cited by
Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks
Yang, X., Feng, S., Zhang, Y., & Wang, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.28 · Cited by 145
“, 2016), and TumEmo (Yang et al., 2020), and compare our MGNNS model with a number of unimodal and multimodal approaches.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Yang, X., Feng, S., Wang, D., & Zhang, Y.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2021
- Volume
- 23
- Pages
- 4014-4026
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W3095024162
- Cited by
- 267
- References
- 49
- FWCI
- 8.33
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.