Audio-Visual Affect Recognition
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
The ability of a computer to detect and appropriately respond to changes in a user's affective state has significant implications to human-computer interaction (HCI). In this paper, we present our efforts toward audio-visual affect recognition on 11 affective states customized for HCI application (four cognitive/motivational and seven basic affective states) of 20 nonactor subjects. A smoothing method is proposed to reduce the detrimental influence of speech on facial expression recognition. The feature selection analysis shows that subjects are prone to use brow movement in face, pitch and energy in prosody to express their affects while speaking. For person-dependent recognition, we apply the voting method to combine the frame-based classification results from both audio and visual channels. The result shows 7.5% improvement over the best unimodal performance. For person-independent test, we apply multistream HMM to combine the information from multiple component streams. This test shows 6.1% improvement over the best component performance
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · cited 1×
“However, most current automatic emotion recognition approaches are unimodal: information processed by the computer system is limited to either face images [8], [9], [17] or speech signals [19], [23]–[26].”
BackgroundMehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874
Self-citationToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Emotion recognition by speech signals
Kwon, O.-W., Chan, K., Hao, J., & Lee, T.-W. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-80 · Cited by 366
De Silva, L. C., & Ng, P. C. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840655 · Cited by 196
Multimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183
Self-citationYoshitomi, Y., Kim, S.-I., Kawano, T., & Kilazoe, T. (2002) · Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499) · doi:10.1109/roman.2000.892491 · Cited by 144
Cited by
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Self-citationA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 758
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568
“The increasing applications of SER, especially those in human computer interaction and affective computing [27], [45], make it a core component in the”
BackgroundFusing audio, visual and textual clues for sentiment analysis from multimodal content
Poria, S., Cambria, E., Howard, N., Huang, G.-B., & Hussain, A. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.01.095 · Cited by 534
Kessous, L., Castellano, G., & Caridakis, G. (2010) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0025-5 · Cited by 256
Towards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144
Self-citationError weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121
Metallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89
DeepVANet: A Deep End-to-End Network for Multi-modal Emotion Recognition
Zhang, Y., Hossain, M. Z., & Rahman, S. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-85613-7_16 · Cited by 29
Metadata
- Authors
- Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2007-02
- Volume
- 9
- Issue
- 2
- Pages
- 424-428
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2168053878
- Cited by
- 136
- References
- 10
- FWCI
- 14.18
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.