Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,720
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
A Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852
Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T., Sandner, P., & Welpe, I. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,649
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · cited 1×
“Even though deep neural networks may be trapped in local optima (Hinton 2010; Bengio 2012), using different optimization techniques, one can achieve the state-of-the-art performance on many challenging tasks mentioned above.”
BackgroundConvolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213
Predicting the future with social media
Asur, S., & Huberman, B. A. (2010) · 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology · doi:10.1109/wi-iat.2010.63 · Cited by 2,081
From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series
O'Connor, B., Balasubramanyan, R., Routledge, B., & Smith, N. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14031 · Cited by 1,923
(2011) · doi:10.5591/978-1-57735-516-8/ijcai11-210 · Cited by 1,229
Modeling Public Mood and Emotion: Twitter Sentiment and Socio-Economic Phenomena
Bollen, J., Mao, H., & Pepe, A. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14171 · Cited by 936
Enhanced Sentiment Learning Using Twitter Hashtags and Smileys
Davidov, D., Tsur, O., & Rappoport, A. (2010) · International Conference on Computational Linguistics · Cited by 884
Large-scale visual sentiment ontology and detectors using adjective noun pairs
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S.-F. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502282 · Cited by 851
SUN attribute database: Discovering, annotating, and recognizing scene attributes
Patterson, G., & Hays, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247998 · Cited by 836
Building text classifiers using positive and unlabeled examples
Liu, B., Dai, Y., Li, X., Lee, W. S., & Yu, P. S. (2004) · Third IEEE International Conference on Data Mining · doi:10.1109/icdm.2003.1250918 · Cited by 667
Predicting Stock Market Indicators Through Twitter “I hope it is not as bad as I fear”
Zhang, X., Fuehres, H., & Gloor, P. A. (2011) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2011.10.562 · Cited by 640
Learning Convolutional Feature Hierarchies for Visual Recognition
Kavukcuoglu, K., Sermanet, P., Boureau, Y.-L., Gregor, K., Mathieu, M., & LeCun, Y. (2010) · neural information processing systems · Cited by 604
Towards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470
Visual Event Recognition in Videos by Learning from Web Data
Duan, L., Xu, D., Tsang, I. W.-H., & Luo, J. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.265 · Cited by 409
Self-citationAesthetics and Emotions in Images
Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q.-T., Wang, J., Li, J., & Luo, J. (2011) · IEEE Signal Processing Magazine · doi:10.1109/msp.2011.941851 · Cited by 408
Self-citation(1984) · Medical Entomology and Zoology · Cited by 385
Unsupervised sentiment analysis with emotional signals
Hu, X., Tang, J., Gao, H., & Liu, H. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488442 · Cited by 384
Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena
Bollen, J., Pepe, A., & Mao, H. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0911.1583 · Cited by 316
Learning Features From Music Audio With Deep Belief Networks.
Hamel, P., & Eck, D. (2010) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1414969 · Cited by 254
Analyzing and predicting sentiment of images on the social web
Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874060 · Cited by 218
Sentribute: image sentiment analysis from a mid-level perspective
Yuan, J., Mcdonough, S., You, Q., & Luo, J. (2013) · Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining · doi:10.1145/2502069.2502079 · Cited by 199
Self-citationBorth, D., Chen, T., Ji, R., & Chang, S.-F. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502268 · Cited by 188
The wisdom of social multimedia: using flickr for prediction and forecast
Jin, X. Y., Gallagher, A., Cao, L., Luo, J., & Han, J. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874196 · Cited by 171
Self-citationMicro-blogging Sentiment Detection by Collaborative Online Learning
Li, G., Hoi, S. C. H., Chang, K., & Jain, R. (2010) · Proceedings · doi:10.1109/icdm.2010.139 · Cited by 51
Learned-norm pooling for deep neural networks.Preprint
Gülçehre, Ç., Cho, K., Pascanu, R., & Bengio, Y. (2013) · arXiv (Cornell University) · Cited by 10
Cited by
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Sentiment analysis using deep learning architectures: a review
Yadav, A., & Vishwakarma, D. K. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09794-5 · Cited by 841
A survey of multimodal sentiment analysis
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.-F., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 601
Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark
You, Q., Luo, J., Jin, H., & Yang, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9987 · Cited by 348
Self-citation
Metadata
- Authors
- You, Q., Luo, J., Jin, H., & Yang, J.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2015-02-09
- Volume
- 29
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2963992782
- Cited by
- 548
- References
- 36
- FWCI
- 29.35
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.