Affective Image Content Analysis: Two Decades Review and New Perspectives
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges - the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“Based on the triplet loss [125], Yang et al.”
Salovey, P., & Mayer, J. D. (1990) · Imagination, Cognition and Personality · doi:10.2190/dugg-p24e-52wk-6cdg · Cited by 10,484
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 3×
“Ekman’s six emotions (anger, disgust, fear, happiness, sadness, surprise) [62] and Mikels’s eight emotions (amuse-”
BackgroundPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1×
“To enrich the generalization ability, one possible solution is to randomize the labeled source images to a sufficient number of domains in the training stage using domain randomization [183], and then, the target domain belongs to the randomized domains to a large extent and thus, the models trained on the randomized…”
MethodTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 1×
“Furthermore, if we have sufficient labeled data in one domain, such as abstract paintings, how can we effectively transfer the well-trained models to another unlabeled or sparsely labeled domain? Because of the presence of domain shift or dataset bias [20], [21], direct transfer often results in poor performance, as s…”
BackgroundAppraisal Processes in Emotion: Theory, Methods, Research
(2001) · doi:10.1093/oso/9780195130072.001.0001 · Cited by 2,384
Norms of valence, arousal, and dominance for 13,915 English lemmas
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0314-x · Cited by 1,959
Feng, Y., You, H., Zhang, Z., Ji, R., & Gao, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013558 · Cited by 1,895
Semantic Understanding of Scenes Through the ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1140-0 · Cited by 1,831
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · cited 1×
“Affective content analysis has also been widely studied in other modalities, such as text [39], [40], speech acoustics [41], [42] and linguistics [43], music [44], [45], sound [46], facial expression [47], [48], [49], [50], video [51], [52], physiological signals [53], [54], [55], and multi-modal data [56], [57], [58],”
BackgroundEmotions Recognition Using EEG Signals: A Survey
Alarcao, S. M., & Fonseca, M. J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2714671 · Cited by 1,201 · cited 1×
“Affective content analysis has also been widely studied in other modalities, such as text [39], [40], speech acoustics [41], [42] and linguistics [43], music [44], [45], sound [46], facial expression [47], [48], [49], [50], video [51], [52], physiological signals [53], [54], [55], and multi-modal data [56], [57], [58],”
BackgroundDomain Generalization via Invariant Feature Representation
Muandet, K., Balduzzi, D., & Schölkopf, B. (2013) · arXiv · doi:10.48550/arxiv.1301.2115 · Cited by 771
Are they different? Affect, feeling, emotion, sentiment, and opinion detection in text
Munezero, M., Montero, C. S., Sutinen, E., & Pajunen, J. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2317187 · Cited by 374 · cited 1×
“Interested readers can refer to [67] for more details.”
BackgroundGunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · cited 1×
“VAD is the most widely used DES model [72], where valence represents the pleasantness ranging from positive to negative, arousal”
MethodSelf-citationContext Based Emotion Recognition Using EMOTIC Dataset
Kosti, R., Alvarez, J., Recasens, A., & Lapedriza, A. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2916866 · Cited by 211
Learning deep features for image emotion classification
Chen, M., Zhang, L., & Allebach, J. P. (2015) · 2015 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2015.7351656 · Cited by 53
(2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3338002 · Cited by 3
Metadata
- Authors
- Zhao, S., Yao, X., Yang, J., Jia, G., Ding, G., Chua, T.-S., Schuller, B. W., & Keutzer, K.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 6729-6751
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3174134652
- Cited by
- 198
- References
- 22
- FWCI
- 24.84
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.