The more the merrier: Analysing the affect of a group of people in images
Abstract
The recent advancement of social media has given users a platform to socially engage and interact with a global population. With millions of images being uploaded onto social media platforms, there is an increasing interest in inferring the emotion and mood display of a group of people in images. Automatic affect analysis research has come a long way but has traditionally focussed on a single subject in a scene. In this paper, we study the problem of inferring the emotion of a group of people in an image. This group affect has wide applications in retrieval, advertisement, content recommendation and security. The contributions of the paper are: 1) a novel emotion labelled database of groups of people in images; 2) a Multiple Kernel Learning based hybrid affect inference model; 3) a scene context based affect inference model; 4) a user survey to better understand the attributes that affect the perception of affect of a group of people in an image. The detailed experimentation validation provides a rich baseline for the proposed database.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most approaches target primitive facial expressions of affect, but others recognize cognitive states, positive and negative affect from groups of people, and liking/not-liking of online media for predicting buying behaviour for marketing purposes.
“Most approaches target primitive FEs of affect, but others recognize cognitive states [266], postive and negative affect from groups of people [267] and liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [267]
Methods for recognizing other affective states have been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes.
“Methods for recognizing other affective states have also been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [267]
In naturalistic environments, most approaches target primitive facial expressions of affect, but methods for recognizing cognitive states, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes are also common.
“In naturalistic environments, most approaches target primitive FEs of affect. Methods for recognizing cognitive states [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268] are also common.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [267]
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. B. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,321 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Face detection, pose estimation, and landmark localization in the wild
Zhu, X., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248014 · Cited by 2,171 · see citing works
Bar, M. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1476 · Cited by 1,613 · see citing works
Representing shape with a spatial pyramid kernel
Bosch, A., Zisserman, A., & Muñoz, X. (2007) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1282280.1282340 · Cited by 1,407 · see citing works
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,099 · see citing works
Blur Insensitive Texture Classification Using Local Phase Quantization
Ojansivu, V., & Heikkilä, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69905-7_27 · Cited by 1,043 · see citing works
Mood and emotions in small groups and work teams
Kelly, J. R., & Barsade, S. G. (2001) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2001.2974 · Cited by 1,011 · see citing works
CENTRIST: A Visual Descriptor for Scene Categorization
Wu, J., & Rehg, J. M. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.224 · Cited by 710 · see citing works
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 664 · see citing works
Self-citationAffective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi-Berthouze, N. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 646 · see citing works
The computer expression recognition toolbox (CERT)
Littlewort, G. C., Whitehill, J., Wu, T., Fasel, I., Frank, M. G., Movellan, J. R., & Bartlett, M. S. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771414 · Cited by 552 · see citing works
What do we perceive in a glance of a real-world scene?
Fei-Fei, L., Iyer, A., Christof Koch, A., & Perona, P. (2007) · Journal of Vision · doi:10.1167/7.1.10 · Cited by 469 · see citing works
Neighborhood Repulsed Metric Learning for Kinship Verification
Lu, J., Zhou, X., Tan, Y.-P., Shang, Y., & Zhou, J. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.134 · Cited by 442 · see citing works
Understanding images of groups of people
Gallagher, A., & Chen, T. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206828 · Cited by 357 · see citing works
Vision-Based Analysis of Small Groups in Pedestrian Crowds
Ge, W., Collins, R. T., & Ruback, R. B. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.176 · Cited by 354 · see citing works
Toward Practical Smile Detection
Whitehill, J., Littlewort, G. C., Fasel, I., Bartlett, M. S., & Movellan, J. R. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.42 · Cited by 310 · see citing works
Automatic nonverbal analysis of social interaction in small groups: A review
Gática-Pérez, D. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.01.004 · Cited by 309 · see citing works
Group emotion: A view from top and bottom.
Barsade, S. G., & Gibson, D. E. (1998) · Cited by 299 · see citing works
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234 · see citing works
Hipster Wars: Discovering Elements of Fashion Styles
Kiapour, M. H., Yamaguchi, K., Berg, A. C., & Berg, T. L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_31 · Cited by 230 · see citing works
Emotion recognition using PHOG and LPQ features
Dhall, A., Asthana, A., Goecke, R., & Gedeon, T. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771366 · Cited by 221 · see citing works
Self-citationAutotagging Facebook: Social network context improves photo annotation
Stone, Z., Zickler, T. E., & Darrell, T. J. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562956 · Cited by 175 · see citing works
Statistical context priming for object detection
Torralba, A. B., & Sinha, P. (2002) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2001.937604 · Cited by 172 · see citing works
Valstar, M., Patras, I., & Pantić, M. (2006) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvpr.2005.457 · Cited by 158 · see citing works
Seeing People in Social Context: Recognizing People and Social Relationships
Wang, G., Gallagher, A., Luo, J., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15555-0_13 · Cited by 132 · see citing works
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119 · see citing works
Self-citationAutomatic Group Happiness Intensity Analysis
Dhall, A., Goecke, R., & Gedeon, T. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2397456 · Cited by 117 · see citing works
Self-citationJoho, H., Staiano, J., Sebe, N., & Jose, J. (2010) · Multimedia tools and applications · doi:10.1007/s11042-010-0632-x · Cited by 104 · see citing works
Self-citationHernandez, J., Hoque, M. E., Drevo, W., & Picard, R. W. (2012) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2370216.2370264 · Cited by 98 · see citing works
Unsupervised discovery of facial events
Zhou, F., la Torre, F. D., & Cohn, J. F. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539966 · Cited by 81 · see citing works
Real-time indoor scene understanding using Bayesian filtering with motion cues
Tsai, G., Xu, C., Liu, J.-G., & Kuipers, B. (2011) · Vision · doi:10.1109/iccv.2011.6126233 · Cited by 78 · see citing works
Neighborhood repulsed metric learning for kinship verification
Lu, J., Hu, J., Zhou, X., et al. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247978 · Cited by 71 · see citing works
In the Pursuit of Effective Affective Computing: The Relationship Between Features and Registration
Chew, S. W., Lucey, P., Lucey, S., Saragih, J., Cohn, J. F., Matthews, I. A., & Sridharan, S. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2194485 · Cited by 64 · see citing works
Finding Happiest Moments in a Social Context
Dhall, A., Joshi, J., Radwan, I., & Goecke, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37444-9_48 · Cited by 50 · see citing works
Self-citationUrban tribes: Analyzing group photos from a social perspective
Murillo, A. C., Kwak, I. S., Bourdev, L., et al. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239352 · Cited by 44 · see citing works
Bucak, S. S., Jin, R., & Jain, A. K. (2010) · Cited by 33 · see citing works
Facial action unit detection using kernel partial least squares
Gehrig, T., & Ekenel, H. K. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130506 · Cited by 21 · see citing works
Dhall, A., Sikka, K., Littlewort, G. C., Goecke, R., & Bartlett, M. S. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835991 · Cited by 6 · see citing works
Self-citationDhall, A., Sikka, K., Littlewort, G. C., et al. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835729 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
From individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218 · see citing works
Self-citationGroup emotion recognition with individual facial emotion CNNs and global image based CNNs
Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., & Qiao, Y. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143008 · Cited by 73 · see citing works
- Authors
- Dhall, A., Joshi, J., Sikka, K., Goecke, R., & Sebe, N.
- Affiliations
- Australian National University (AU)
- University of Canberra (AU)
- University of California San Diego (US)
- University of Trento (IT)
- Venue
- 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)
- Publisher
- IEEE
- Published
- 2015-05
- Pages
- 1-8
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W1603712378
- Cited by
- 93
- References
- 42
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 4 · 2023: 7 · 2022: 6 · 2021: 5 · 2020: 9 · 2019: 10 · 2018: 21 · 2017: 12 · 2016: 9 · 2015: 2
- FWCI
- 19.59
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Automatic Face and Gesture Recognition (publishedVersion)
- University of Canberra Research Portal (submittedVersion)
- Iris (University of Trento) (submittedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Funders
- Ministero dell’Istruzione, dell’Università e della Ricerca
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- emotion inference, social media images, multiple kernel learning, scene context, user survey
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
