Video Emotion Recognition with Transferred Deep Feature Encodings
Abstract
Despite growing research interest, emotion understanding for user-generated videos remains a challenging problem. Major obstacles include the diversity and complexity of video content, as well as the sparsity of expressed emotions. For the first time, we systematically study large-scale video emotion recognition by transferring deep feature encodings. In addition to the traditional, supervised recognition, we study the problem of zero-shot emotion recognition, where emotions in the test set are unseen during training. To cope with this task, we utilize knowledge transferred from auxiliary image and text corpora. A novel auxiliary Image Transfer Encoding (ITE) process is proposed to efficiently encode and generate video representation. We also thoroughly investigate different configurations of convolutional neural networks. Comprehensive experiments on multiple datasets demonstrate the effectiveness of our framework.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523 · see citing works
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · see citing works
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335 · see citing works
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680 · see citing works
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · see citing works
The brain basis of emotion: a meta-analytic review
Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett, L. F. (2012) · Behavioral and Brain Sciences · doi:10.1017/s0140525x11000446 · Cited by 2,320 · cited 4× · see citing works
“In contrast, some very recent research [2, 25] questioned the validity of basic emotional categories and implied high variances of emotions far beyond several fixed basic categories.”
BackgroundResultInfluentialUniversals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273 · see citing works
Barrett, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00003.x · Cited by 1,859 · see citing works
Attribute-Based Classification for Zero-Shot Visual Object Categorization
Lampert, C. H., Nickisch, H., & Stefan Harmeling, B. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.140 · Cited by 1,639 · see citing works
Multisensory integration: current issues from the perspective of the single neuron
Stein, B. E., & Stanford, T. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2331 · Cited by 1,598 · see citing works
Hypercolumns for object segmentation and fine-grained localization
Hariharan, B., Arbelaez, P., Girshick, R., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298642 · Cited by 1,511 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Support Vector Machines for Multiple-Instance Learning
Andrews, S. J., Tsochantaridis, I., & Hofmann, T. F. (2002) · Cited by 1,370 · see citing works
Hughes, C. W. (1982) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-198205000-00017 · Cited by 975 · see citing works
Large-scale visual sentiment ontology and detectors using adjective noun pairs
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S.-F. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502282 · Cited by 851 · see citing works
Introduction to Information Retrieval
Larson, R. R. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21234 · Cited by 845 · see citing works
MILES: Multiple-Instance Learning via Embedded Instance Selection
Chen, Y., Bi, J., & Wang, J. Z. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.248 · Cited by 748 · see citing works
Solving Multiple-Instance Problem: A Lazy Learning Approach
Wang, J., Zucker, E. J.-D., & Zucker, J. (2000) · CogPrints (University of Southampton) · Cited by 570 · see citing works
Transductive Multi-View Zero-Shot Learning
Fu, Y., Hospedales, T. M., Xiang, T., & Gong, S. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2408354 · Cited by 514 · see citing works
Self-citationGärtner, T., Flach, P. A., Kowalczyk, A., & Smola, A. J. (2002) · Cited by 460 · see citing works
EMA: A process model of appraisal dynamics
Marsella, S. C., & Gratch, J. (2009) · Cognitive Systems Research · doi:10.1016/j.cogsys.2008.03.005 · Cited by 452 · see citing works
Do facial expressions signal specific emotions? Judging emotion from the face in context.
Carroll, J. M., & Russell, J. A. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.2.205 · Cited by 439 · see citing works
A discriminative CNN video representation for event detection
Xu, Z., Yang, Y., & Hauptmann, A. G. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298789 · Cited by 438 · see citing works
Do facial expressions signal specific emotions? Judging emotion from the face in context.
Carroll, J. M., & Russell, J. A. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.70.2.205 · Cited by 422 · see citing works
EMOTION: Theory, Research, and Experience
(1980) · Elsevier eBooks · doi:10.1016/b978-0-12-558701-3.50001-6 · Cited by 422 · see citing works
Affective understanding in film
Wang, H. L., & Cheong, L. (2006) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2006.873781 · Cited by 267 · see citing works
DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural NetworksPreprint
Chen, T., Borth, D., Darrell, T., & Chang, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.8586 · Cited by 263 · see citing works
Emotion: Theory, Research, and Experience. Vol. 1. Theories of Emotion
Camras, L., Plutchik, R., & Kellerman, H. (1981) · The American Journal of Psychology · doi:10.2307/1422757 · Cited by 241 · see citing works
Logistic Regression and Boosting for Labeled Bags of Instances
Xu, X., & Frank, E. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24775-3_35 · Cited by 220 · see citing works
Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep NetworksPreprint
You, Q., Luo, J., Jin, H., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06041 · Cited by 209 · see citing works
Solving multi-instance problems with classifier ensemble based on constructive clustering
Zhou, Z., & Zhang, M.-L. (2006) · Knowledge and Information Systems · doi:10.1007/s10115-006-0029-3 · Cited by 150 · see citing works
Affective content detection using HMMs
Kang, H.-B. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957066 · Cited by 149 · see citing works
Learning Multimodal Latent Attributes
Fu, Y., Hospedales, T. M., Xiang, T., et al. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.128 · Cited by 148 · see citing works
Self-citationVisual Sentiment Prediction with Deep Convolutional Neural NetworksPreprint
Xu, C., Cetintas, S., Lee, K.-C., & Li, L.-J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5731 · Cited by 139 · see citing works
Predicting Emotions in User-Generated Videos
Jiang, Y., Xu, B., & Xue, X. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8724 · Cited by 124 · see citing works
Self-citationPredicting Viewer Perceived Emotions in Animated GIFs
Jou, B., Bhattacharya, S., & Chang, S. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2656408 · Cited by 81 · see citing works
Campos, V., Salvador, A., Giró-i-Nieto, X., & Jou, B. (2015) · doi:10.1145/2813524.2813530 · Cited by 78 · see citing works
Scalable Multi-instance Learning
Wei, X.-S., Wu, J., & Zhou, Z. (2014) · Proceedings · doi:10.1109/icdm.2014.16 · Cited by 67 · see citing works
Ontological supervision for fine grained classification of Street View storefronts
Movshovitz-Attias, Y., Yu, Q., Stumpe, M. C., Shet, V. D., Arnoud, S., & Yatziv, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298778 · Cited by 66 · see citing works
Key Instance Detection in Multi-Instance Learning
Liu, G., Wu, J., & Zhou, Z. (2012) · DR-NTU (Nanyang Technological University) · Cited by 66 · see citing works
Emotion Theory, research, and experience
PLUTCHIK, R., & Kellerman, H. (1989) · Cited by 65 · see citing works
Weakly supervised pain localization using multiple instance learning
Sikka, K., Dhall, A., & Bartlett, M. S. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553762 · Cited by 62 · see citing works
Is 2D Information Enough For Viewpoint Estimation?
Ghodrati, A., Pedersoli, M., & Tuytelaars, T. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.19 · Cited by 43 · see citing works
A Discriminative CNN Video Representation for Event DetectionPreprint
Xu, Z., Yang, Y. P., & Hauptmann, A. G. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4006 · Cited by 29 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Zero-Shot Learning: Settings, Methods, and Applications
Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips
Nguyen, T.-L., Kavuri, S., & Lee, M. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.06.010 · Cited by 84 · see citing works
Multimodal Emotion Recognition in Deep Learning:a Survey
Gu, X., Shen, Y., & Xu, J. (2021) · 2021 International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/iccst53801.2021.00027 · Cited by 36 · see citing works
- Authors
- Xu, B., Fu, Y., Jiang, Y.-G., Li, B., & Sigal, L.
- Affiliations
- Fudan University (CN)
- Walt Disney (United States) (US)
- Venue
- Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval
- Publisher
- ACM
- Published
- 2016-06-06
- Pages
- 15-22
- OpenAlex
- W2414501075
- Cited by
- 53
- References
- 51
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 4 · 2022: 6 · 2021: 12 · 2020: 5 · 2019: 13 · 2018: 4 · 2017: 5 · 2016: 2
- FWCI
- 9.25
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Multimedia Retrieval (publishedVersion)
- Funders
- National 863 Program (2014AA015101)
- National Natural Science Foundation of China (61572134)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video-based emotion recognition, user-generated video, cross-modal knowledge transfer, video representation learning, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review