A survey of affect recognition methods: audio, visual and spontaneous expressions
The DOI matches its Crossref record; 159 of the 160 references it cites were found in a registry or an index.
Abstract
Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The array of prosodic features contributing to the depiction of emotions in speech encompasses, among others, fundamental frequency (F0), energy, and speech rate [39,40,41,42].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [40]
Citations
Cites
Development and validation of brief measures of positive and negative affect: The PANAS scales.
Watson, D., Clark, L. A., & Tellegen, A. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.6.1063 · Cited by 32,433
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
Emotion: A Psychoevolutionary Synthesis
Camras, L., & Plutchik, R. (1980) · The American Journal of Psychology · doi:10.2307/1422394 · Cited by 2,487
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,941
Evidence for a three-factor theory of emotions
Russell, J. A., & Mehrabian, A. (1977) · Journal of Research in Personality · doi:10.1016/0092-6566(77)90037-x · Cited by 1,940
Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.
Ambady, N., & Rosenthal, R. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.111.2.256 · Cited by 1,922
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778
Acoustic profiles in vocal emotion expression
Banse, R., & Scherer, K. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.3.614 · Cited by 1,546
The vocabulary problem in human-system communication
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987) · Communications of the ACM · doi:10.1145/32206.32212 · Cited by 1,504
(1972) · Elsevier eBooks · doi:10.1016/c2013-0-02458-9 · Cited by 1,446
Ekman, P., & Oster, H. (1979) · Annual Review of Psychology · doi:10.1146/annurev.ps.30.020179.002523 · Cited by 1,178
A 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique.
Ekman, P. (1994) · Psychological Bulletin · doi:10.1037//0033-2909.115.2.268 · Cited by 1,130
The Handbook of emotion elicitation and assessment
(2008) · Choice Reviews Online · doi:10.5860/choice.46-1769 · Cited by 1,119
Web-Based Database for Facial Expression Analysis
Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005) · 2005 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2005.1521424 · Cited by 1,039
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988
Handbook of Pattern Recognition and Computer Vision
Chen, C. H., & Wang, P. S. P. (2005) · WORLD SCIENTIFIC eBooks · doi:10.1142/5711 · Cited by 968
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965
Automatic recognition and analysis of human faces and facial expressions: a survey
Samal, A., & Iyengar, P. A. (1992) · Pattern Recognition · doi:10.1016/0031-3203(92)90007-6 · Cited by 910
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874
Self-citationFacial and vocal expressions of emotion
Russell, J. A., Bachorowski, J.-A., & Fernández-Dols, J.-M. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145102 · Cited by 873
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840
Toward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame.
Keltner, D. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.68.3.441 · Cited by 718
A systems approach to appraisal mechanisms in emotion
Sander, D., Grandjean, D., & Scherer, K. R. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.001 · Cited by 700
Kotsia, I., & Pitas, I. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.884954 · Cited by 687
Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior
Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.297 · Cited by 624
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Facial expression of pain: An evolutionary account
Williams, A. C. D. C. (2002) · Behavioral and Brain Sciences · doi:10.1017/s0140525x02000080 · Cited by 583
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
FEELTRACE: an instrument for recording perceived emotion in real time
Cowie, R., Douglas‐Cowie, E., Savvidou, S. M., & McMahon, E. R. (2000) · Research Portal (Queen's University Belfast) · Cited by 539
THE DICTIONARY OF AFFECT IN LANGUAGE
WHISSELL, C. M. (1989) · The Measurement of Emotions · doi:10.1016/b978-0-12-558704-4.50011-6 · Cited by 536
(1989) · Journal of Psychophysiology · Cited by 489
Recognition of Facial Expression from Optical Flow
Mase, K. (1991) · IEICE Transactions on Information and Systems · Cited by 480
Emotional speech: Towards a new generation of databases
Douglas-Cowie, E., Campbell, N., Cowie, R., & Roach, P. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00070-5 · Cited by 461
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
The production and recognition of emotions in speech: features and algorithms
Pierre-Yves, O. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(02)00141-6 · Cited by 439
Facial Action Recognition for Facial Expression Analysis From Static Face Images
Pantic, M., & Rothkrantz, L. J. M. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825931 · Cited by 417
Emotion recognition in human–computer interaction
Fragopanagos, N., & Taylor, J. G. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.006 · Cited by 414
Prosody-based automatic detection of annoyance and frustration in human-computer dialog
Ang, J., Dhillon, R., Krupski, A., Shriberg, E., & Stolcke, A. (2002) · 7th International Conference on Spoken Language Processing (ICSLP 2002) · doi:10.21437/icslp.2002-559 · Cited by 386
Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
Tong, Y., Liao, W., & Ji, Q. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1094 · Cited by 378
Mother-infant face-to-face interaction: The sequence of dyadic states at 3, 6, and 9 months.
Cohn, J. F., & Tronick, E. Z. (1987) · Developmental Psychology · doi:10.1037//0012-1649.23.1.68 · Cited by 373
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 355
Tian, Y.-L., Kanade, T., & Cohn, J. F. (2005) · Handbook of Face Recognition · doi:10.1007/0-387-27257-7_12 · Cited by 347
Challenges in real-life emotion annotation and machine learning based detection
Devillers, L., Vidrascu, L., & Lamel, L. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.007 · Cited by 342
Multimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101300 · Cited by 339
EMOTION RECOGNITION BY SPEECH SIGNAL
Kwon, O., Chan, K.-H., & Hao, J. (2003) · Interspeech (USB) · Cited by 338
Human Computing and Machine Understanding of Human Behavior: A Survey
Pantic, M., Pentland, A., Nijholt, A., & Huang, T. S. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72348-6_3 · Cited by 320
Self-citationJuslin, P. N., & Scherer, K. R. (2005) · The New Handbook of Methods in Nonverbal Behavior Research · doi:10.1093/oso/9780198529613.003.0003 · Cited by 318
3D Facial Expression Recognition Based on Primitive Surface Feature Distribution
Wang, J., Yin, L., Wei, X., & Sun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.14 · Cited by 307
THE TIMING OF FACIAL MOTION IN POSED AND SPONTANEOUS SMILES
COHN, J. F., & SCHMIDT, K. L. (2004) · International Journal of Wavelets, Multiresolution and Information Processing · doi:10.1142/s021969130400041x · Cited by 298
Fully Automatic Facial Action Recognition in Spontaneous Behavior
Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.55 · Cited by 290
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
El Kaliouby, R., & Robinson, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.427 · Cited by 282
How to find trouble in communication
Batliner, A., Fischer, K., Huber, R., Spilker, J., & Nöth, E. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00079-1 · Cited by 280
Socially aware, computation and communication
Pentlan, A. (2005) · Computer · doi:10.1109/mc.2005.104 · Cited by 253
Cohen, I., Cozman, F. G., Sebe, N., Cirelo, M. C., & Huang, T. S. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.127 · Cited by 251
Self-citationMachine Analysis of Facial Expressions
Pantic, M., & Stewart, M. (2007) · Face Recognition · doi:10.5772/4847 · Cited by 248
Emotion recognition in spontaneous speech using GMMs
Neiberg, D., Elenius, K., & Laskowski, K. (2006) · Interspeech 2006 · doi:10.21437/interspeech.2006-277 · Cited by 247
A probabilistic framework for modeling and real-time monitoring human fatigue
Ji, Q., Lan, P., & Looney, C. (2006) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2005.855922 · Cited by 245
Emotion recognition through facial expression analysis based on a neurofuzzy network
Ioannou, S. V., Raouzaiou, A. T., Tzouvaras, V. A., Mailis, T. P., Karpouzis, K. C., & Kollias, S. D. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.004 · Cited by 242
Ekman, P. (1982) · Cited by 236
Recognition of facial expressions and measurement of levels of interest from video
Yeasin, M., Bullot, B., & Sharma, R. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.870737 · Cited by 235
Affective multimodal human-computer interaction
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101299 · Cited by 234
Self-citationKanade, T. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564386_1 · Cited by 223
Detection of Cough Signals in Continuous Audio Recordings Using Hidden Markov Models
Matos, S., Birring, S. S., Pavord, I. D., & Evans, D. H. (2006) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2006.873548 · Cited by 216
Manifold based analysis of facial expression
Chang, Y., Hu, C., Feris, R., & Turk, M. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.08.006 · Cited by 215
A video database of moving faces and people
O'Toole, A. J., Ha, J., Snow, S. L., Hurst, D. R., Pappas, M. R., Ayyad, J. H., & Abdi, H. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.90 · Cited by 212
How to distinguish posed from spontaneous smiles using geometric features
Valstar, M. F., Gunes, H., & Pantic, M. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322202 · Cited by 203
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202
Self-citationMULTIMODAL EMOTION RECOGNITION
Sebe, N., Cohen, I., & Huang, T. S. (2005) · Handbook of Pattern Recognition and Computer Vision · doi:10.1142/9789812775320_0021 · Cited by 201
Self-citationGunes, H., & Piccardi, M. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.39 · Cited by 199
Roisman, G. I., Tsai, J. L., & Chiang, K.-H. S. (2004) · Developmental Psychology · doi:10.1037/0012-1649.40.5.776 · Cited by 196
Self-citationFacial expression decomposition
Wang, H., & Ahuja (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238452 · Cited by 195
Gibson, R. (2002) · Cited by 195
Learning From Examples in the Small Sample Case: Face Expression Recognition
Guo, G., & Dyer, C. R. (2005) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.846658 · Cited by 194
Affect Recognition from Face and Body: Early Fusion vs. Late Fusion
Gunes, H., & Piccardi, M. (2006) · 2005 IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/icsmc.2005.1571679 · Cited by 191
Duric, Z., Gray, W. D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M. J., Schunn, C., & Wechsler, H. (2002) · Proceedings of the IEEE · doi:10.1109/jproc.2002.801449 · Cited by 185
Automatic discrimination between laughter and speech
Truong, K. P., & van Leeuwen, D. A. (2007) · Speech Communication · doi:10.1016/j.specom.2007.01.001 · Cited by 184
Multimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183
Self-citationRobust full‐motion recovery of head by dynamic templates and re‐registration techniques
Xiao, J., Moriyama, T., Kanade, T., & Cohn, J. F. (2003) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.10048 · Cited by 180
Evaluation of natural emotions using self assessment manikins
Grimm, M., & Kroschel, K. (2005) · IEEE Workshop on Automatic Speech Recognition and Understanding, 2005. · doi:10.1109/asru.2005.1566530 · Cited by 179
Predicting student emotions in computer-human tutoring dialogues
Litman, D. J., & Forbes-Riley, K. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219000 · Cited by 179
The New Handbook of Methods in Nonverbal Behavior Research
(2005) · doi:10.1093/oso/9780198529613.001.0001 · Cited by 170
Spontaneous vs. posed facial behavior: automatic analysis of brow actions
Valstar, M. F., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181031 · Cited by 168
Haar Features for FACS AU Recognition
Whitehill, J., & Omlin, C. W. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.61 · Cited by 162
Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face
Lucey, S., Bilal, A., & F., J. (2007) · Face Recognition · doi:10.5772/4841 · Cited by 161
Real-life emotions detection with lexical and paralinguistic cues on human-human call center dialogs
Devillers, L., & Vidrascu, L. (2006) · Interspeech 2006 · doi:10.21437/interspeech.2006-275 · Cited by 160
Enhancing relevance feedback in image retrieval using unlabeled data
Zhou, Z., Chen, K., & Dai, H.-B. (2006) · ACM Transactions on Information Systems · doi:10.1145/1148020.1148023 · Cited by 159
Dellaert, F., Polzin, T., & Waibel, A. (1996) · 4th International Conference on Spoken Language Processing (ICSLP 1996) · doi:10.21437/icslp.1996-462 · Cited by 158
The ISL meeting corpus: the impact of meeting type on speech style
Burger, S., MacLaren, V., & Yu, H. (2002) · 7th International Conference on Spoken Language Processing (ICSLP 2002) · doi:10.21437/icslp.2002-140 · Cited by 156
Beyond emotion archetypes: Databases for emotion modelling using neural networks
Cowie, R., Douglas‐Cowie, E., & Cox, C. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.002 · Cited by 155
Human computing and machine understanding of human behavior: a survey
Pantic, M., Pentland, A., Nijholt, A., & Huang, T. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181044 · Cited by 151
Self-citationFaces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322198 · Cited by 150
Emotion Analysis in Man-Machine Interaction Systems
Balomenos, T., Raouzaiou, A. T., Ioannou, S. V., Drosopoulos, A., Karpouzis, K. C., & Kollias, S. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-30568-2_27 · Cited by 144
Batliner, A. M., Hacker, C., Steidl, S., Nöth, E., D’Arcy, S., Russell, M. J., & Wong, M. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2g6tkyi6gw2r · Cited by 140
Audio-Visual Affect Recognition
Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.886310 · Cited by 136
Self-citationModeling naturalistic affective states via facial and vocal expressions recognition
Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A., & Karpouzis, K. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181029 · Cited by 131
Audio-visual based emotion recognition-a new approach
Song, M., Bu, J., Chen, C., & Li, N. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315276 · Cited by 126
Probabilistic expression analysis on manifolds
Chang, Y., Hu, C., & Turk, M. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315208 · Cited by 123
Huang, T. S., & Chen, L. S.-H. (2000) · Cited by 109
Self-citationMotion history for facial action detection in video
Valstar, M., Pantic, M., & Patras, I. (2005) · 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583) · doi:10.1109/icsmc.2004.1398371 · Cited by 109
Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources
Forbes-Riley, K., & Litman, D. J. (2004) · North American Chapter of the Association for Computational Linguistics · Cited by 107
The painful face: pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K., Solomon, P., & Theobald, B. J. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322197 · Cited by 106
Distinguishing Deceptive from Non-Deceptive Speech
Hirschberg, J., Stefan Benus, A., Brenier, J., Enos, F., Friedman, S., Gilman, S., Girand, C., Graciarena, M., Kathol, A., Michaelis, L. A., et al. (2005) · doi:10.7916/d8697c06 · Cited by 106
Foundations of human computing: facial expression and emotion
Cohn, J. F. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181043 · Cited by 102
Authentic facial expression analysis
Sebe, N., Lew, M. S., Cohen, I., Sun, Y., Gevers, T., & Huang, T. S. (2004) · Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings. · doi:10.1109/afgr.2004.1301585 · Cited by 101
Self-citationExplanation-based facial motion tracking using a piecewise Bezier volume deformation model
Tao, H., & Huang, T. S. (2003) · Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) · doi:10.1109/cvpr.1999.787002 · Cited by 100
Self-citationDetecting certainness in spoken tutorial dialogues
Liscombe, J., Hirschberg, J., & Venditti, J. J. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-581 · Cited by 99
The recognition of basic facial expressions by neural network
Kobayashi, H., & Hara, F. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170444 · Cited by 98
Spontaneous Emotional Facial Expression Detection
Zeng, Z., Fu, Y., Roisman, G. I., Wen, Z., Hu, Y., & Huang, T. S. (2006) · Journal of Multimedia · doi:10.4304/jmm.1.5.1-8 · Cited by 98
Self-citationCapturing subtle facial motions in 3D face tracking
Wen, Z., & Huang (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238646 · Cited by 97
Detection of real-life emotions in call centers
Vidrascu, L., & Devillers, L. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-582 · Cited by 96
MAUI: a multimodal affective user interface
Lisetti, C. L., & Nasoz, F. (2002) · Proceedings of the tenth ACM international conference on Multimedia · doi:10.1145/641007.641038 · Cited by 96
Steidl, S., Levit, M., Batliner, A., Noth, E., & Niemann, H. (2006) · Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. · doi:10.1109/icassp.2005.1415114 · Cited by 94
ASR for emotional speech: Clarifying the issues and enhancing performance
Athanaselis, T., Bakamidis, S., Dologlou, I., Cowie, R., Douglas‐Cowie, E., & Cox, C. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.008 · Cited by 88
Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior
Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J., & Moriyama, T. (2005) · 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583) · doi:10.1109/icsmc.2004.1398367 · Cited by 88
Bimodal Fusion of Emotional Data in an Automotive Environment
Hoch, S., Althoff, F., McGlaun, G., & Rigoll, G. (2006) · Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. · doi:10.1109/icassp.2005.1415597 · Cited by 87
Foundations of Human Computing: Facial Expression and Emotion
Cohn, J. F. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_1 · Cited by 87
Audiovisual recognition of spontaneous interest within conversations
Schuller, B., Müeller, R., Höernler, B., Höethker, A., Konosu, H., & Rigoll, G. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322201 · Cited by 85
A Prototype for Automatic Recognition of Spontaneous Facial Actions
Bartlett, Littlewort, G. C., B, T. J. S. A., & Movellan, J. R. (2002) · neural information processing systems · Cited by 85
Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition
Karpouzis, K. C., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A. T., Malatesta, L., & Kollias, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_5 · Cited by 84
Emotion Detection From Infant Facial Expressions And Cries
Pal, P., Iyer, A. N., & Yantorno, R. E. (2006) · 2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings · doi:10.1109/icassp.2006.1660444 · Cited by 84
An automated face reader for fatigue detection
Gu, H., & Ji, Q. (2004) · Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings. · doi:10.1109/afgr.2004.1301517 · Cited by 83
Audiovisual discrimination between laughter and speech
Petridis, S., & Pantic, M. (2008) · 2008 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2008.4518810 · Cited by 82
Combining Prosodic Lexical and Cepstral Systems for Deceptive Speech Detection
Graciarena, M., Shriberg, E., Stolcke, A., Enos, F., Hirschberg, J., & Kajarekar, S. (2006) · 2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings · doi:10.1109/icassp.2006.1660200 · Cited by 76
Emotion recognition from the facial image and speech signal
Go, H.-J., Kwak, K.-C., Lee, D.-J., & Chun, M.-G. (2003) · Cited by 76
Automatic 3D Facial Expression Analysis in Videos
Chang, Y., Vieira, M., Turk, M., & Velho, L. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564386_23 · Cited by 74
Efficient reinforcement learning
Fiechter, C.-N. (1994) · Proceedings of the seventh annual conference on Computational learning theory - COLT '94 · doi:10.1145/180139.181019 · Cited by 72
Face Recognition Across Pose and Illumination
Ralph Gross, A., Baker, S., Matthews, I., & Takeo Kanade, A. (2005) · Springer eBooks · doi:10.1007/0-387-27257-7_10 · Cited by 64
Automatic detection of laughter
Truong, K. P., & Leeuwen, D. A. V. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-322 · Cited by 63
Audio-Visual Affect Recognition through Multi-Stream Fused HMM for HCI
Zeng, Z., Tu, J., Pianfetti, B., Liu, M., Zhang, T., Zhang, Z., Huang, T. S., & Levinson, S. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.77 · Cited by 62
Self-citationMeta-Classifiers in Acoustic and Linguistic Feature Fusion-Based Affect Recognition
Schuller, B., Villar, R. J., Rigoll, G., & Lang, M. (2006) · Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. · doi:10.1109/icassp.2005.1415116 · Cited by 61
Robust Real-Time Face Pose and Facial Expression Recovery
Zhu, Z., & Ji, Q. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.259 · Cited by 60
Audio-Visual Spontaneous Emotion Recognition
Zeng, Z., Hu, Y., Roisman, G. I., Wen, Z., Fu, Y., & Huang, T. S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_4 · Cited by 59
Self-citationRecognizing Human Emotion from Audiovisual Informaiton
Wang, Y., & Guan, L. (2006) · Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. · doi:10.1109/icassp.2005.1415607 · Cited by 55
Facial Expression in Affective Disorders
Ekman, P., Matsumoto, D., & Friesen, W. V. (2005) · What the Face RevealsBasic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) · doi:10.1093/acprof:oso/9780195179644.003.0020 · Cited by 54
Facial Expression Analysis Using Nonlinear Decomposable Generative Models
Lee, C.-S., & Elgammal, A. (2005) · Lecture notes in computer science · doi:10.1007/11564386_3 · Cited by 52
Latent semantic analysis of facial action codes for automatic facial expression recognition
Fasel, B., Monay, F., & Gatica-Perez, D. (2004) · Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval · doi:10.1145/1026711.1026742 · Cited by 51
Zeng, Z., Hu, Y., Liu, M., Fu, Y., & Huang, T. S. (2006) · Proceedings of the 14th ACM international conference on Multimedia · doi:10.1145/1180639.1180661 · Cited by 48
Self-citationAudio-visual emotion recognition in adult attachment interview
Zeng, Z., Hu, Y., Fu, Y., Huang, T. S., Roisman, G. I., & Wen, Z. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181028 · Cited by 46
Self-citationMozziconacci, S. J. L. (2002) · Speech prosody · doi:10.21437/speechprosody.2002-1 · Cited by 45
Gaze-X: Adaptive, Affective, Multimodal Interface for Single-User Office Scenarios
Maat, L., & Pantic, M. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72348-6_13 · Cited by 44
Güneş, H., & Piccardi, M. (2005) · Lecture notes in computer science · doi:10.1007/11573548_14 · Cited by 35
Audio-Visual Affect Recognition in Activation-Evaluation Space
Zeng, Z., Zhang, Z., Pianfetti, B., Tu, J., & Huang, T. S. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521551 · Cited by 32
Self-citationGaze-X: adaptive affective multimodal interface for single-user office scenarios
Maat, L., & Pantic, M. (2006) · Proceedings of the 8th international conference on Multimodal interfaces · doi:10.1145/1180995.1181032 · Cited by 31
Reliability of Lexical and Prosodic Cues in Two Real-life Spoken Dialog Corpora
Devillers, L. Y., & Vasilescu, I. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3cfc3utmj4u6 · Cited by 30
Teeters, A., el Kaliouby, R., & Picard, R. W. (2006) · ACM SIGGRAPH Posters · doi:10.1145/1179622.1179782 · Cited by 29
Affect-robust speech recognition by dynamic emotional adaptation
Schuller, B., Stadermann, J., & Rigoll, G. (2006) · Speech Prosody 2006 · doi:10.21437/speechprosody.2006-221 · Cited by 28
Prosody based emotion recognition for MEXI
Austermann, A., Esau, N., Kleinjohann, L., & Kleinjohann, B. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2005.1545341 · Cited by 27
Children's emotion recognition in an intelligent tutoring scenario
Hasegawa‐Johnson, M., Levinson, S. E., & Zhang, T. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-552 · Cited by 25
Annotation and Analysis of Emotionally Relevant Behavior in the ISL Meeting Corpus
Laskowski, K., & Burger, S. (2006) · International Conference on Language Resources and Evaluation · doi:10.63317/3hu9rtu8whw3 · Cited by 24
Case-based reasoning for user-profiled recognition of emotions from face images
Pantic, M., & Rothkrantz, L. (2005) · 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763) · doi:10.1109/icme.2004.1394211 · Cited by 23
A study on the automatic detection and characterization of emotion in a voice service context
Blouin, C. K., & Maffiolo, V. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-318 · Cited by 16
Sherwin, D. (1989) · PubMed · Cited by 2
Research works printed but not linked: 5
- 10.5555/1032641.1033044 · doi:10.5555/1032641.1033044
- 10.5555/795661.796155 · doi:10.5555/795661.796155
- 10.5555/1051814 · doi:10.5555/1051814
- Stein , B. , Meredith , M. A. ( 1993 ). The Merging of Senses . MIT Press , Cambridge, USA Stein, B., Meredith, M. A. (1993). The Merging of Senses. MIT Press, Cambridge, USA
- 10.5555/1126250.1126340 · doi:10.5555/1126250.1126340
Cited by
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Self-citationFacing Airport Security: Affect, Biopolitics, and the Preemptive Securitisation of the Mobile Body
Adey, P. (2009) · Environment and Planning D: Society and Space · doi:10.1068/d0208 · Cited by 271
A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models
Koelstra, S., Pantic, M., & Patras, I. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.50 · Cited by 242
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
Brain-Computer Interfacing and Games
Bos, D. P.-O., Reuderink, B., van de Laar, B., Gürkök, H., Mühl, C., Poel, M., Nijholt, A., & Heylen, D. (2010) · Human-computer interaction series · doi:10.1007/978-1-84996-272-8_10 · Cited by 194
Metadata
- Authors
- Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Imperial College London, London, United Kingdom
- Venue
- Proceedings of the 9th international conference on Multimodal interfaces
- Publisher
- ACM
- Published
- 2007-11-12
- Pages
- 126-133
- OpenAlex
- W2155841434
- Cited by
- 144
- References
- 160
- Citations by year
- 2026: 11 · 2025: 29 · 2024: 24 · 2023: 9 · 2022: 4 · 2021: 1 · 2020: 1 · 2019: 3 · 2018: 5 · 2017: 4 · 2016: 2 · 2015: 6 · 2013: 11 · 2012: 4
- FWCI
- 15.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- University of Twente Research Information (publishedVersion)
- University of Twente Research Information (submittedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- affect recognition, emotion recognition, facial expression recognition, vocal emotion recognition, emotion perception, affective computing, naturalistic behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.