The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display
Abstract
Psychologists have long explored mechanisms with which humans recognize other humans' affective states from modalities, such as voice and face display. This exploration has led to the identification of the main mechanisms, including the important role played in the recognition process by the modalities' dynamics. Constrained by the human physiology, the temporal evolution of a modality appears to be well approximated by a sequence of temporal segments called onset, apex, and offset. Stemming from these findings, computer scientists, over the past 15 years, have proposed various methodologies to automate the recognition process. We note, however, two main limitations to date. The first is that much of the past research has focused on affect recognition from single modalities. The second is that even the few multimodal systems have not paid sufficient attention to the modalities' dynamics: The automatic determination of their temporal segments, their synchronization to the purpose of modality fusion, and their role in affect recognition are yet to be adequately explored. To address this issue, this paper focuses on affective face and body display, proposes a method to automatically detect their temporal segments or phases, explores whether the detection of the temporal phases can effectively support recognition of affective states, and recognizes affective states based on phase synchronization/alignment. The experimental results obtained show the following: 1) affective face and body displays are simultaneous but not strictly synchronous; 2) explicit detection of the temporal phases can improve the accuracy of affect recognition; 3) recognition from fused face and body modalities performs better than that from the face or the body modality alone; and 4) synchronized feature-level fusion achieves better performance than decision-level fusion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 2×
“to converge to a local optimum in the parameter space [59].”
BackgroundMethodRussell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
An extended set of Haar-like features for rapid object detection
Lienhart, R., & Maydt, J. (2003) · Proceedings. International Conference on Image Processing · doi:10.1109/icip.2002.1038171 · Cited by 2,977 · cited 1×
“the art is based on the robust and well-known method proposed by Viola and Jones [74] and improved by Lienhart and Maydt based on a set of rotated Haarlike features [42].”
MethodToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Human facial expressions as adaptations: Evolutionary questions in facial expression research
Schmidt, K. L., & Cohn, J. F. (2001) · American Journal of Physical Anthropology · doi:10.1002/ajpa.20001 · Cited by 560
Multisensory perception: Beyond modularity and convergence
Driver, J., & Spence, C. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00740-5 · Cited by 465
Body expressions influence recognition of emotions in the face and voice.
Van den Stock, J., Righart, R., & de Gelder, B. (2007) · Emotion · doi:10.1037/1528-3542.7.3.487 · Cited by 457 · cited 1×
“the influence of whole-body expressions of emotions on the recognition of facial and vocal expressions of emotion [73].”
BackgroundBi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382
Self-citationEmotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · cited 1×
“Examples of the geometric feature-based methods are auxiliary particle filtering (e.g., [53] and [71]) and the piecewise Bezier volume deformation tracking, which use an explicit 3-D wireframe face model to track the geometric facial features defined on the model (e.g., [ 63 ] and [64]).”
MethodMultimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190
Cited by
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916
“One possible way to handle such hard examples is to take the context [53] or human body gestures [54] within images into consideration, which we will explore in future work.”
BackgroundA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Affective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi-Berthouze, N. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 607
Recognizing actions using depth motion maps-based histograms of oriented gradients
Yang, X., Zhang, C., & Tian, Y. (2012) · Proceedings of the 20th ACM international conference on Multimedia · doi:10.1145/2393347.2396382 · Cited by 564
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 426
Self-citationGunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360
Self-citationEffective 3D action recognition using EigenJoints
Yang, X., & Tian, Y. (2014) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2013.03.001 · Cited by 340
Tian, Y., Kanade, T., & Cohn, J. F. (2011) · Handbook of Face Recognition · doi:10.1007/978-0-85729-932-1_19 · Cited by 255
Automatic analysis of affective postures and body motion to detect engagement with a game companion,
Sanghvi, J., Castellano, G., Leite, I., Pereira, A., McOwan, P. W., & Paiva, A. (2011) · Proceedings of the 6th international conference on Human-robot interaction · doi:10.1145/1957656.1957781 · Cited by 250
Towards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236
Super Normal Vector for Human Activity Recognition with Depth Cameras
Yang, X., & Tian, Y. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2565479 · Cited by 162
Multimodal emotional state recognition using sequence-dependent deep hierarchical features
Barros, P., Jirak, D., Weber, C., & Wermter, S. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.09.009 · Cited by 86
“…is analyzed, the filters tend to take more time until they can learn that (Chen et al., 2013) 66.50% Bag of words (Chen et al., 2013) 59.00% SVM (Gunes & Piccardi, 2009) 32.49% Adaboost (Gunes & Piccardi, 2009) 35.22% the structures of the face are important for the final classification, and…”
BackgroundMethodResultInfluentialChen, S., Tian, Y., Liu, Q., & Metaxas, D. N. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.014 · Cited by 72
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53
Strength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53
Learning Gait Emotions Using Affective and Deep Features
Randhavane, T., Bhattacharya, U., Kabra, P., Kapsaskis, K., Gray, K., Manocha, D., & Bera, A. (2022) · Proceedings of the 15th ACM SIGGRAPH Conference on Motion, Interaction and Games · doi:10.1145/3561975.3562957 · Cited by 13
Towards Analyzing and Predicting the Experience of Live Performances with Wearable Sensing
Gedik, E., Cabrera-Quiros, L., Martella, C., Englebienne, G., & Hung, H. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2875987 · Cited by 10
“turing bodily expression such as computer vision [13], [14], motion capture [15] and pressure sensors [16], and generally aim to automatically map bodily expressions into well-known affective states.”
Background
Metadata
- Authors
- Gunes, H., & Piccardi, M.
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
- Published
- 2009-02
- Volume
- 39
- Issue
- 1
- Pages
- 64-84
- ISSN
- 1083-4419 (ISSN-L)
- OpenAlex
- W2132680170
- Cited by
- 198
- References
- 11
- FWCI
- 14.33
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.