Group emotion recognition with individual facial emotion CNNs and global image based CNNs
Abstract
This paper presents our approach for group-level emotion recognition in the Emotion Recognition in the Wild Challenge 2017. The task is to classify an image into one of the group emotion such as positive, neutral or negative. Our approach is based on two types of Convolutional Neural Networks (CNNs), namely individual facial emotion CNNs and global image based CNNs. For the individual facial emotion CNNs, we first extract all the faces in an image, and assign the image label to all faces for training. In particular, we utilize a large-margin softmax loss for discriminative learning and we train two CNNs on both aligned and non-aligned faces. For the global image based CNNs, we compare several recent state-of-the-art network structures and data augmentation strategies to boost performance. For a test image, we average the scores from all faces and the image to predict the final group emotion category. We win the challenge with accuracies 83.9% and 80.9% on the validation set and testing set respectively, which improve the baseline results by about 30%.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214 · see citing works
Self-citationDeep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Learning Face Representation from ScratchPreprint
Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7923 · Cited by 1,629 · see citing works
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926 · see citing works
SphereFace: Deep Hypersphere Embedding for Face RecognitionPreprint
Liu, W., Wen, Y., Yu, Z., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08063 · Cited by 409 · see citing works
From individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218 · see citing works
EmotiW 2016: video and group-level emotion recognition challenges
Dhall, A., Goecke, R., Joshi, J., Hoey, J., & Gedeon, T. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2997638 · Cited by 147 · see citing works
The more the merrier: Analysing the affect of a group of people in images
Dhall, A., Joshi, J., Sikka, K., Goecke, R., & Sebe, N. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7163151 · Cited by 93 · see citing works
LSTM for dynamic emotion and group emotion recognition in the wild
Sun, B., Wei, Q., Li, L., Xu, Q., He, J. Y. D., & Yu, L. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2997640 · Cited by 69 · see citing works
Group-level arousal and valence recognition in static images: Face, body and context
Mou, W., Çeliktutan, O., & Güneş, H. (2015) · IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7284862 · Cited by 49 · see citing works
Happiness level prediction with sequential inputs via multiple regressions
Li, J., Roy, S., Feng, J., & Sim, T. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997636 · Cited by 47 · see citing works
Group happiness assessment using geometric features and dataset balancing
Vonikakis, V., Yazıcı, Y., Nguyễn, V. D., & WINKLER, S. S. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2997633 · Cited by 45 · see citing works
Huang, X., Dhall, A., Zhao, G., Goecke, R., & Pietikäinen, M. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.34 · Cited by 38 · see citing works
Towards using visual attributes to infer image sentiment of social events
Ahsan, U., Choudhury, M. D., & Essa, I. A. (2017) · doi:10.1109/ijcnn.2017.7966013 · Cited by 25 · see citing works
From Facial Expression Recognition to Interpersonal Relation PredictionPreprint
Zhang, Z., Luo, P., Loy, C. C., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.06426 · Cited by 16 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084827 · Cited by 5,273 · see citing works
“Large-margin softmax loss was used in face recognition [81], emotion recognition [82], speaker verification [83], and so on.”
MethodDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
- Authors
- Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., & Qiao, Y.
- Affiliations
- Chinese Academy of Sciences (CN)
- National Taiwan University (TW)
- Venue
- Proceedings of the 19th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2017-11-03
- Pages
- 549-552
- OpenAlex
- W2767359575
- Cited by
- 73
- References
- 18
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 1 · 2023: 10 · 2022: 9 · 2021: 13 · 2020: 12 · 2019: 14 · 2018: 10
- FWCI
- 11.32
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- National Natural Science Foundation of China
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- group emotion recognition, facial emotion recognition, convolutional neural networks, face detection, data augmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
