Towards multimodal sentiment analysis: harvesting opinions from the web
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
With more than 10,000 new videos posted online every day on social websites such as YouTube and Facebook, the internet is becoming an almost infinite source of information. One crucial challenge for the coming decade is to be able to harvest relevant information from this constant flow of multimodal data. This paper addresses the task of multimodal sentiment analysis, and conducts proof-of-concept experiments that demonstrate that a joint model that integrates visual, audio, and textual features can be effectively used to identify sentiment in Web videos. This paper makes three important contributions. First, it addresses for the first time the task of tri-modal sentiment analysis, and shows that it is a feasible task that can benefit from the joint exploitation of visual, audio and textual modalities. Second, it identifies a subset of audio-visual features relevant to sentiment analysis and present guidelines on how to integrate these features. Finally, it introduces a new dataset consisting of real online data, which will be useful for future research in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The YouTube dataset was first introduced by Morency et al. in 2011 [25].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [25]
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 1×
“Human communication is a dynamic process and explicitly modeling this dynamic has been shown to improve tasks such as gestures recognition [17] and speech recognition [21].”
BackgroundMining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Lexicon-based methods for sentiment analysis
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011) · Computational Linguistics · doi:10.1162/coli_a_00049 · Cited by 3,301
A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts
Pang, B., & Lee, L. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1218990 · Cited by 3,301
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining
Esuli, A., & Sebastiani, F. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/54vo5wjya4cw · Cited by 2,480
(2007) · Cited by 2,018
Annotating Expressions of Opinions and Emotions in Language
Wiebe, J., Wilson, T., & Cardie, C. (2005) · Language Resources and Evaluation · doi:10.1007/s10579-005-7880-9 · Cited by 1,755
Yu, H., & Hatzivassiloglou, V. (2003) · Proceedings of the 2003 conference on Empirical methods in natural language processing - · doi:10.3115/1119355.1119372 · Cited by 1,053
The General Inquirer: A Computer Approach to Content Analysis
Green, B. F., Stone, P. J., Dunphy, D. C., Smith, M. S., & Ogilvie, D. M. (1967) · American Educational Research Journal · doi:10.2307/1161774 · Cited by 1,013
Emotions from text: machine learning for text-based emotion prediction
Alm, C. O., Roth, D., & Sproat, R. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220648 · Cited by 781
Large-Scale Sentiment Analysis for News and Blogs
Godbole, N., Srinivasaiah, M., & Skiena, S. (2007) · International Conference on Weblogs and Social Media · Cited by 687
SemEval-2007 task 14: affective text
Strapparava, C., & Mihalcea, R. (2007) · Proceedings of the 4th International Workshop on Semantic Evaluations - SemEval '07 · doi:10.3115/1621474.1621487 · Cited by 602
Self-citationProceedings of the 2008 Conference on Empirical Methods in Natural Language Processing
Lapata, M., & Ng, H. T. (2008) · Cited by 567
Creating Subjective and Objective Sentence Classifiers from Unannotated Texts
Wiebe, J., & Riloff, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_53 · Cited by 545
Keltner, D., Ekman, P., Gonzaga, & Beer, J. (2002) · Handbook of Affective Sciences · doi:10.1093/oso/9780195126013.003.0022 · Cited by 513
Learning Multilingual Subjective Language via Cross-Lingual Projections
Mihalcea, R. F., Banea, C., & Wiebe, J. (2007) · University of North Texas Digital Library (University of North Texas) · Cited by 479
Self-citationOpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 3×
“We used the open source software OpenEAR [22] to automatically compute the pitch and voice intensity.”
MethodInfluentialLatent-Dynamic Discriminative Models for Continuous Gesture Recognition
Morency, L.-P., Quattoni, A., & Darrell, T. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383299 · Cited by 374
Self-citationThe General Inquirer: a computer approach to content analysis
Stone, P. J., Dunphy, D. C., & Smith, M. S. (1966) · CERN Document Server (European Organization for Nuclear Research) · Cited by 344
Building Lexicon for Sentiment Analysis from Massive Collection of HTML Documents
Kaji, N., & Kitsuregawa, M. (2007) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 315
Wiebe, J., & Mihalcea, R. (2006) · Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 · doi:10.3115/1220175.1220309 · Cited by 296
Self-citationThe SEMAINE corpus of emotionally coloured character interactions
McKeown, G., Valstar, M. F., Cowie, R., & Pantic, M. (2010) · 2010 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2010.5583006 · Cited by 278
Proceedings of the Association for Computational Linguistics 2014
Heafield, K., Kayser, M., & Manning, C. D. (2014) · Cited by 232
Markov models for offline handwriting recognition: a survey
Plötz, T., & Fink, G. A. (2009) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-009-0098-4 · Cited by 220
Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) corpus
Bänziger, & Scherer, T. A. (2010) · Cited by 209
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · cited 1×
“Also related to our work is the research done on multimodal emotion analysis [7, 23, 32].”
MethodWollmer, M., Schuller, B., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 165
Interrelation Between Speech and Facial Gestures in Emotional Utterances: A Single Subject Study
Busso, C., & Narayanan, S. S. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.905145 · Cited by 149
Rapport expressed through nonverbal behavior
Harrigan, J. A., Oxman, T. E., & Rosenthal, R. (1985) · Journal of Nonverbal Behavior · doi:10.1007/bf00987141 · Cited by 136
Why are they excited?: identifying and explaining spikes in blog mood levels
Balog, K., Mishne, G., & de Rijke, M. (2006) · Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations on - EACL '06 · doi:10.3115/1608974.1609010 · Cited by 114
Multimodal subjectivity analysis of multiparty conversation
Raaijmakers, S., Truong, K., & Wilson, T. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613774 · Cited by 103
Summarizing Emails with Conversational Cohesion and Subjectivity
Carenini, G., Ng, R. T., & Zhou, X. (2008) · Cited by 79
Topic segmentation of dialogue
Arguello, J., & Rosé, C. (2006) · Proceedings of the HLT-NAACL 2006 Workshop on Analyzing Conversations in Text and Speech - ACTS '09 · doi:10.3115/1564535.1564542 · Cited by 58
Why Are They Excited? Identifying and Explaining Spikes in Blog Mood Levels
Balog, K., Mishne, G., & de Rijke, M. ;. I. 0.-0.-1.-0. (2006) · UvA-DARE (University of Amsterdam) · Cited by 19
Research works printed but not linked: 3
- 10.5555/1613715.1613774 · doi:10.5555/1613715.1613774
- P. Stone . General Inquirer: Computer Approach to Content Analysis . MIT Press , 1968 . P. Stone. General Inquirer: Computer Approach to Content Analysis. MIT Press, 1968.
- 10.5555/1621474.1621487 · doi:10.5555/1621474.1621487
Cited by
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
“Creative and novel applications of using multiple modalities have been among successful recent research directions in machine learning (You et al., 2016; Donahue et al., 2015; Antol et al., 2015; Specia et al., 2016; Tong et al., 2017). opinions from YouTube movie reviews (Zadeh et al., 2016a).”
BackgroundMethodInfluentialSelf-citationA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433
Self-citationNew Avenues in Opinion Mining and Sentiment Analysis
Cambria, E., Schuller, B., Xia, Y., & Havasi, C. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.30 · Cited by 1,285
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,204
Metadata
- Authors
- Morency, L.-P., Mihalcea, R., & Doshi, P.
- Affiliations
- University of Southern California (US)
- University of North Texas (US)
- Venue
- Proceedings of the 13th international conference on multimodal interfaces
- Publisher
- ACM
- Published
- 2011-11-14
- Pages
- 169-176
- OpenAlex
- W2122563357
- Cited by
- 470
- References
- 37
- Citations by year
- 2026: 24 · 2025: 49 · 2024: 48 · 2023: 53 · 2022: 39 · 2021: 46 · 2020: 31 · 2019: 42 · 2018: 37 · 2017: 27 · 2016: 23 · 2015: 21 · 2014: 12 · 2013: 14 · 2012: 4
- FWCI
- 7.51
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, online video, audio-visual features, feature integration, social media videos, sentiment datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.