The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display
Abstract
Psychologists have long explored mechanisms with which humans recognize other humans' affective states from modalities, such as voice and face display. This exploration has led to the identification of the main mechanisms, including the important role played in the recognition process by the modalities' dynamics. Constrained by the human physiology, the temporal evolution of a modality appears to be well approximated by a sequence of temporal segments called onset, apex, and offset. Stemming from these findings, computer scientists, over the past 15 years, have proposed various methodologies to automate the recognition process. We note, however, two main limitations to date. The first is that much of the past research has focused on affect recognition from single modalities. The second is that even the few multimodal systems have not paid sufficient attention to the modalities' dynamics: The automatic determination of their temporal segments, their synchronization to the purpose of modality fusion, and their role in affect recognition are yet to be adequately explored. To address this issue, this paper focuses on affective face and body display, proposes a method to automatically detect their temporal segments or phases, explores whether the detection of the temporal phases can effectively support recognition of affective states, and recognizes affective states based on phase synchronization/alignment. The experimental results obtained show the following: 1) affective face and body displays are simultaneous but not strictly synchronous; 2) explicit detection of the temporal phases can improve the accuracy of affect recognition; 3) recognition from fused face and body modalities performs better than that from the face or the body modality alone; and 4) synchronized feature-level fusion achieves better performance than decision-level fusion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 2×
“to converge to a local optimum in the parameter space [59].”
BackgroundMethodRussell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
An extended set of Haar-like features for rapid object detection
Lienhart, R., & Maydt, J. (2003) · Proceedings. International Conference on Image Processing · doi:10.1109/icip.2002.1038171 · Cited by 2,977 · cited 1×
“the art is based on the robust and well-known method proposed by Viola and Jones [74] and improved by Lienhart and Maydt based on a set of rotated Haarlike features [42].”
MethodHuman facial expressions as adaptations: Evolutionary questions in facial expression research
Schmidt, K. L., & Cohn, J. F. (2001) · American Journal of Physical Anthropology · doi:10.1002/ajpa.20001 · Cited by 560
Multisensory perception: Beyond modularity and convergence
Driver, J., & Spence, C. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00740-5 · Cited by 465
Body expressions influence recognition of emotions in the face and voice.
Van den Stock, J., Righart, R., & de Gelder, B. (2007) · Emotion · doi:10.1037/1528-3542.7.3.487 · Cited by 457 · cited 1×
“the influence of whole-body expressions of emotions on the recognition of facial and vocal expressions of emotion [73].”
BackgroundBi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382
Self-citationEmotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · cited 1×
“Examples of the geometric feature-based methods are auxiliary particle filtering (e.g., [53] and [71]) and the piecewise Bezier volume deformation tracking, which use an explicit 3-D wireframe face model to track the geometric facial features defined on the model (e.g., [ 63 ] and [64]).”
MethodMultimodal integration-a statistical view
Wu, L., Oviatt, S. L., & Cohen, P. R. (1999) · IEEE Transactions on Multimedia · doi:10.1109/6046.807953 · Cited by 190
Cited by
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916
“One possible way to handle such hard examples is to take the context [53] or human body gestures [54] within images into consideration, which we will explore in future work.”
BackgroundA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Affective Body Expression Perception and Recognition: A Survey
Kleinsmith, A., & Bianchi-Berthouze, N. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.16 · Cited by 607
Recognizing actions using depth motion maps-based histograms of oriented gradients
Yang, X., Zhang, C., & Tian, Y. (2012) · Proceedings of the 20th ACM international conference on Multimedia · doi:10.1145/2393347.2396382 · Cited by 564
Metadata
- Authors
- Gunes, H., & Piccardi, M.
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
- Published
- 2009-02
- Volume
- 39
- Issue
- 1
- Pages
- 64-84
- ISSN
- 1083-4419 (ISSN-L)
- OpenAlex
- W2132680170
- Cited by
- 198
- References
- 9
- FWCI
- 14.33
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.