The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Image–text sentiment analysis via deep multimodal attentive fusion
Abstract
Abstract Sentiment analysis of social media data is crucial to understand people’s position, attitude, and opinion toward a certain event, which has many applications such as election prediction and product evaluation. Though great effort has been devoted to the single modality (image or text), less effort is paid to the joint analysis of multimodal data in social media. Most of the existing methods for multimodal sentiment analysis simply combine different data modalities, which results in dissatisfying performance on sentiment classification. In this paper, we propose a novel image–text sentiment analysis model, i.e., Deep Multimodal Attentive Fusion (DMAF), to exploit the discriminative features and the internal correlation between visual and semantic contents with a mixed fusion framework for sentiment analysis. Specifically, to automatically focus on discriminative regions and important words which are most related to the sentiment, two separate unimodal attention models are proposed to learn effective emotion classifiers for visual and textual modality respectively. Then, an intermediate fusion-based multimodal attention model is proposed to exploit the internal correlation between visual and textual features for joint sentiment classification. Finally, a late fusion scheme is applied to combine the three attention models for sentiment prediction. Extensive experiments are conducted to demonstrate the effectiveness of our approach on both weakly labeled and manually labeled datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,961
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Lexicon-based methods for sentiment analysis
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011) · Computational Linguistics · doi:10.1162/coli_a_00049 · Cited by 3,301
SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining
Baccianella, S., Esuli, A., & Sebastiani, F. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2jktnrykui9s · Cited by 2,666
Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · HAL · Cited by 2,020
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.345 · Cited by 1,670
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Sentiment Analysis using Support Vector Machines with Diverse Information Sources
Mullen, T., & Collier, N. (2004) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 861
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Large-scale visual sentiment ontology and detectors using adjective noun pairs
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S.-F. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502282 · Cited by 851
Twitter Sentiment Analysis with Deep Convolutional Neural Networks
Severyn, A., & Moschitti, A. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767830 · Cited by 659
Effective Approaches to Attention-based Neural Machine Translation
Luong, M.-T., Pham, H., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04025 · Cited by 655
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0055 · Cited by 595
Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
You, Q., Luo, J., Jin, H., & Yang, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9179 · Cited by 548
A System for Real-time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle
Wang, H., Can, D., Kazemzadeh, A., Bar, F., & Narayanan, S. S. (2012) · Cited by 542
News impact on stock price return via sentiment analysis
Li, X., Xie, H., Chen, L., Wang, J., & Deng, X. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.04.022 · Cited by 503
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470
Fully automatic lexicon expansion for domain-oriented sentiment analysis
Kanayama, H., & Nasukawa, T. (2006) · Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 · doi:10.3115/1610075.1610125 · Cited by 408
Aspect-based sentiment analysis of movie reviews on discussion boards
Thet, T. T., Na, J.-C., & Khoo, C. S. G. (2010) · Journal of Information Science · doi:10.1177/0165551510388123 · Cited by 406
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403
Comparative experiments on sentiment classification for online product reviews
Cui, H., Mittal, V. O., & Datar, M. (2006) · National Conference on Artificial Intelligence · Cited by 393
Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach
Wang, X., Wei, F., Liu, X., Zhou, M., & Zhang, M. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063726 · Cited by 393
Learning Semantic Representations of Users and Products for Document Level Sentiment Classification
Tang, D., Qin, B., & Liu, T. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1098 · Cited by 365
Neural Sentiment Classification with User and Product Attention
Chen, H., Sun, M., Tu, C., Lin, Y., & Liu, Z. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1171 · Cited by 339
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval
Song, J., Yu, Q., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.592 · Cited by 261
Topic Modeling based Sentiment Analysis on Social Media for Stock Market Prediction
Nguyen, T. H., & Shirai, K. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1131 · Cited by 246
Coooolll: A Deep Learning System for Twitter Sentiment Classification
Tang, D., Wei, F., Qin, B., Liu, T., & Zhou, M. (2014) · Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) · doi:10.3115/v1/s14-2033 · Cited by 243
(2013) · Cited by 236
Analyzing and predicting sentiment of images on the social web
Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874060 · Cited by 218
Sentribute: image sentiment analysis from a mid-level perspective
Yuan, J., Mcdonough, S., You, Q., & Luo, J. (2013) · Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining · doi:10.1145/2502069.2502079 · Cited by 199
Can Collective Sentiment Expressed on Twitter Predict Political Elections?
Chung, J., & Mustafaraj, E. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.8065 · Cited by 197
Building emotional dictionary for sentiment analysis of online news
Rao, Y., Lei, J., Wenyin, L., Li, Q., & Chen, M. (2014) · World Wide Web · doi:10.1007/s11280-013-0221-9 · Cited by 192
You, Q., Luo, J., Jin, H., & Yang, J. (2016) · Proceedings of the Ninth ACM International Conference on Web Search and Data Mining · doi:10.1145/2835776.2835779 · Cited by 187
Visual Sentiment Prediction with Deep Convolutional Neural NetworksPreprint
Xu, C., Cetintas, S., Lee, K.-C., & Li, L.-J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5731 · Cited by 159
Visual Sentiment Analysis by Attending on Local Image Regions
You, Q., Jin, H., & Luo, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10501 · Cited by 158
The Stock Sonar — Sentiment Analysis of Stocks Based on a Hybrid Approach
Feldman, R., Rosenfeld, B. L., Bar-Haim, R., & Fresko, M. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i2.18854 · Cited by 149
You, Q., Cao, L., Jin, H., & Luo, J. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964288 · Cited by 129
Microblog Sentiment Analysis Based on Cross-media Bag-of-words Model
Wang, M., Cao, D., Li, L., Li, S., & Ji, R. (2014) · International Conference on Internet Multimedia Computing and Service · doi:10.1145/2632856.2632912 · Cited by 128
A cross-media public sentiment analysis system for microblog
Cao, D., Ji, R., Lin, D., & Li, S. (2014) · Multimedia Systems · doi:10.1007/s00530-014-0407-8 · Cited by 112
How Do Your Friends on Social Media Disclose Your Emotions?
Yang, Y., Jia, J., Zhang, S., Wu, B., Chen, Q., Li, J., Xing, C., & Tang, J. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8740 · Cited by 97
Muise, C. J., Belle, V., Felli, P., McIlraith, S. A., Miller, T., Pearce, A. R., & Sonenberg, L. (2015) · National Conference on Artificial Intelligence · Cited by 60
Junczys-Dowmunt, M., & Grundkiewicz, R. (2016) · The Association for Computational Linguistics · Cited by 48
Multimodal Network Embedding via Attention based Multi-view Variational Autoencoder
Huang, F., Zhang, X., Li, C., Li, Z., He, Y., & Zhao, Z. (2018) · International Conference on Multimedia Retrieval · doi:10.1145/3206025.3206035 · Cited by 39
Self-citationASVUniOfLeipzig: Sentiment Analysis in Twitter using Data-driven Machine Learning Techniques
Remus, R. (2013) · Cited by 9
Cited by
Sentiment analysis using deep learning architectures: a review
Yadav, A., & Vishwakarma, D. K. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09794-5 · Cited by 841
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges
Das, R., & Singh, T. D. (2023) · ACM Computing Surveys · doi:10.1145/3586075 · Cited by 363
Recent advancements and challenges of NLP-based sentiment analysis: A state-of-the-art review
Jim, J. R., Talukder, M. A. R., Malakar, P., Kabir, M. M., Nur, K., & Mridha, M. F. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100059 · Cited by 297
Image-Text Multimodal Emotion Classification via Multi-View Attentional Network
Yang, X., Feng, S., Wang, D., & Zhang, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3035277 · Cited by 267
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Metadata
- Authors
- Huang, F., Zhang, X., Zhao, Z., Xu, J., & Li, Z.
- Venue
- Knowledge-Based Systems
- Publisher
- Elsevier BV
- Published
- 2019-03
- Volume
- 167
- Pages
- 26-37
- ISSN
- 0950-7051 (ISSN-L)
- OpenAlex
- W2910191085
- Cited by
- 296
- References
- 55
- FWCI
- 14.62
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.