The DOI matches its Crossref record; 136 of the 136 references it cites were found in a registry or an index.
A Review and Meta-Analysis of Multimodal Affect Detection Systems
Abstract
Affect detection is an important pattern recognition problem that has inspired researchers from several areas. The field is in need of a systematic review due to the recent influx of Multimodal (MM) affect detection systems that differ in several respects and sometimes yield incompatible results. This article provides such a survey via a quantitative review and meta-analysis of 90 peer-reviewed MM systems. The review indicated that the state of the art mainly consists of person-dependent models (62.2% of systems) that fuse audio and visual (55.6%) information to detect acted (52.2%) expressions of basic emotions and simple dimensions of arousal and valence (64.5%) with feature- (38.9%) and decision-level (35.6%) fusion techniques. However, there were also person-independent systems that considered additional modalities to detect nonbasic emotions and complex dimensions using model-level fusion techniques. The meta-analysis revealed that MM systems were consistently (85% of systems) more accurate than their best unimodal counterparts, with an average improvement of 9.83% (median of 6.60%). However, improvements were three times lower when systems were trained on natural (4.59%) versus acted data (12.7%). Importantly, MM accuracy could be accurately predicted (cross-validated R 2 of 0.803) from unimodal accuracies and two system-level factors. Theoretical and applied implications and recommendations are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646
(2000) · Cited by 9,198
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208
Core affect and the psychological construction of emotion.
Russell, J. A. (2003) · Psychological Review · doi:10.1037/0033-295x.110.1.145 · Cited by 6,113 · cited 4×
“modalities (since prototypical emotions are invoked), and increased specificity (since there is lower likelihood of multiple emotions being experienced) [Barrett 2006; Russell 2003].”
BackgroundInfluentialDEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 4×
“…of the affective phenomenon being modeled (e.g., moods vs. emotions, categorical vs. dimensional representations, partly due to a difficulty in defining affect [Izard 2010]); and (g) many others as articulated in previous reviews [Calvo and D’Mello 2010; Pantic and Rothkrantz 2003; Zeng et al. 2009].”
BackgroundResultInfluentialSelf-citationBarrett, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00003.x · Cited by 1,859 · cited 2×
“Hence other than the rare cases of prototypical emotions, or in artificial experimental contexts involving 43:28 S. K. D’mello and J. Kory”
BackgroundRussell, J. A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.102 · Cited by 1,640
Barrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085709 · Cited by 1,425 · cited 3×
“The third hypothesized advantage of MM systems stems from the fact that UM affect detectors are inherently noisy since the link between specific signals and affective states is tenuous at best [Barrett et al. 2007; Russell et al. 2003].”
BackgroundMethodPlutchik, R. (2001) · American Scientist · doi:10.1511/2001.4.344 · Cited by 1,419
Proceedings of the British Machine Vision Conference 2014
(2014) · doi:10.5244/c.28 · Cited by 1,337
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179
Basic Emotions, Natural Kinds, Emotion Schemas, and a New Paradigm
Izard, C. E. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00044.x · Cited by 1,110
Multimodal human–computer interaction: A survey
Jaimes, A., & Sebe, N. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2006.10.019 · Cited by 965
Modeling naturalistic affective states via facial and vocal expressions recognition
Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A., & Karpouzis, K. (2006) · International Conference on Multimodal Interaction · doi:10.1145/1180995.1181029 · Cited by 932
Facial and vocal expressions of emotion
Russell, J. A., Bachorowski, J.-A., & Fernández-Dols, J.-M. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145102 · Cited by 873
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 2×
“There was also considerable variability (SD = 37.8) in the number of participants used model building, ranging from a single participant [Busso et al. 2004; Haq et al. 2008] to 343 participants [W¨ollmer et al. 2013b].”
BackgroundInfluentialToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.47 · Cited by 790
(2003) · Choice Reviews Online · doi:10.5860/choice.40-4648 · Cited by 779
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722
McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.20 · Cited by 665
Bridging emotion theory and neurobiology through dynamic systems modeling
Lewis, M. D. (2005) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0500004x · Cited by 618
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
The many meanings/aspects of emotion: definitions, functions, activation, and regulation
Izard, C. E. (2010) · Emotion Review · doi:10.1177/1754073910374661 · Cited by 574
Levels of Analysis and the Organization of Affect
Rosenberg, E. L. (1998) · Review of General Psychology · doi:10.1037//1089-2680.2.3.247 · Cited by 563
The Oxford Handbook of Affective Computing
(2015) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199942237.001.0001 · Cited by 482
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450
Youtube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403
Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2011.2116000 · Cited by 398
D'mello, S., & Graesser, A. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2395123.2395128 · Cited by 388
Self-citationMultimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101300 · Cited by 339
Empirically Building and Evaluating a Probabilistic Model of User Affect
Conati, C., & Maclaren, H. (2009) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-009-9062-8 · Cited by 328
D’Mello, S. K., & Graesser, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9074-4 · Cited by 323
Self-citationDouglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., et al. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74889-2_43 · Cited by 321
Coherence between expressive and experiential systems in emotion
Rosenberg, E. L., & Ekman, P. (1994) · Cognition & Emotion · doi:10.1080/02699939408408938 · Cited by 300
D'Mello, S. (2013) · Journal of Educational Psychology · doi:10.1037/a0032674 · Cited by 300
Self-citationMeta-Analysis of the First Facial Expression Recognition Challenge
Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2200675 · Cited by 297
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288 · cited 3×
“, [Bailenson et al. 2008; Glodek et al. 2013; Koelstra et al. 2012; Soleymani et al. 2012; Wöllmer et al. 2013a]) or (c) naturalistic displays of affect (i.”
BackgroundInfluentialBailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A. C., Nass, C., & John, O. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.10.011 · Cited by 276
Wu, C.-H., & Liang, W.-B. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 266
Affective Computing: From Laughter to IEEE
Picard, R. W. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.10 · Cited by 265
Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech
Castellano, G., Kessous, L., & Caridakis, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85099-1_8 · Cited by 259
Kessous, L., Castellano, G., & Caridakis, G. (2010) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0025-5 · Cited by 256
Multimodal Sentiment Analysis of Spanish Online Videos
Perez Rosas, V., Mihalcea, R., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.9 · Cited by 227
Recognizing Human Emotional State From Audiovisual Signals
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921734 · Cited by 214
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202
Recognizing Human Emotional State From Audiovisual Signals*
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.927665 · Cited by 202
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · cited 2×
“…was considerable variation in model-level fusion methods, but bidirectional long short-term memories [Eyben et al. 2010; Metallinou et al. 2012; W¨ollmer et al. 2010, 2013a], various HMM-based approaches (error-weighted semicoupled HMMs [Lin et al. (e.g., Jiang et al. [2011], Paleari et al.…”
BackgroundAutomatic Temporal Segment Detection and Affect Recognition From Face and Body Display
Gunes, H., & Piccardi, M. (2009) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.927269 · Cited by 198
The half-life of cognitive-affective states during complex learning
D'Mello, S., & Graesser, A. (2011) · Cognition and Emotion · doi:10.1080/02699931.2011.613668 · Cited by 195
Self-citationPredicting student emotions in computer-human tutoring dialogues
Litman, D. J., & Forbes-Riley, K. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219000 · Cited by 179
Population validity for educational data mining models: A case study in affect detection
Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C. (2014) · British Journal of Educational Technology · doi:10.1111/bjet.12156 · Cited by 168
Multimodal information fusion application to human emotion recognition from face and speech
Mansoorizadeh, M., & Moghaddam Charkari, N. (2010) · Multimedia Tools and Applications · doi:10.1007/s11042-009-0344-2 · Cited by 166
Detecting user engagement with a robot companion using task and social interaction-based features
Castellano, G., Pereira, A., Leite, I., Paiva, A., & McOwan, P. W. (2009) · Proceedings of the 2009 international conference on Multimodal interfaces · doi:10.1145/1647314.1647336 · Cited by 162
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156 · cited 2×
“Some of these include metadecision trees [Wu and Liang 2011], cascading specialists [Kim and Lingenfelser 2010; Wagner et al. 2011], Kalman filters [Glodek et al. 2013], Bayesian belief integration [Chanel et al. 2011”
BackgroundContext-Sensitive Learning for Enhanced Audiovisual Emotion Classification
Metallinou, A., Wöllmer, M., Katsamanis, A., Eyben, F., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.40 · Cited by 153
Khalili, Z., & Moradi, M. H. (2009) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2009.5178854 · Cited by 147
Bimodal Emotion Recognition using Speech and Physiological Changes
Kim, J. (2007) · doi:10.5772/4754 · Cited by 146
Eyben, F., Wöllmer, M., Graves, A., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0032-6 · Cited by 137
Audio-Visual Affect Recognition
Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.886310 · Cited by 137
Beyond emotion archetypes: Databases for emotion modelling using neural networks
Cowie, R., Douglas‐Cowie, E., & Cox, C. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.002 · Cited by 136
Speaker-dependent audio-visual emotion recognition
Haq, S., & Jackson, P. J. B. (2009) · Open Research (University of Surrey) · Cited by 134
Forbes-Riley, K., & Litman, D. J. (2011) · Speech Communication · doi:10.1016/j.specom.2011.02.006 · Cited by 133
Crowdsourcing Facial Responses to Online Videos
McDuff, D., el Kaliouby, R., & Picard, R. W. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.19 · Cited by 128
Emergent Ghosts of the Emotion Machine
Coan, J. A. (2010) · Emotion Review · doi:10.1177/1754073910361978 · Cited by 127
Beyond Facial Expressions: Learning Human Emotion from Body Gestures
Shan, C., Gong, S., & McOwan, P. W. (2007) · British Machine Vision Conference (BMVC) · doi:10.5244/c.21.43 · Cited by 127
Tukey, J. W., & McLaughlin, D. H. (2016) · Cited by 127
Error weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121
Audio-visual feature selection and reduction for emotion classification
Haq, S., Jackson, P. J. B., & Edge, J. D. (2008) · Surrey Research Insight Open Access (The University of Surrey) · Cited by 121
Detection of Emotions during Learning with AutoTutor
Chipman, P., D’Mello, S. K., Gholson, B., Graesser, A. C., McDaniel, B., & Witherspoon, A. M. (2006) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 120
Self-citationLitman, D. J., & Forbes-Riley, K. (2005) · Speech Communication · doi:10.1016/j.specom.2005.09.008 · Cited by 118
Multi-Modal Emotion Recognition from Speech and Text
Chuang, Z.-J., & Wu, C. (2004) · Cited by 117
Ensemble of SVM trees for multimodal emotion recognition
Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., & Prasad, R. (2012) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · Cited by 111
Dimensional affect recognition using Continuous Conditional Random Fields
Baltrušaitis, T., Banda, N., & Robinson, P. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553785 · Cited by 110
D’Mello, S. K., & Kory-Westlund, J. M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388686 · Cited by 103
Self-citationAudio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice
Metallinou, A., Lee, S., & Narayanan, S. S. (2008) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2008.40 · Cited by 98
Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources
Forbes-Riley, K., & Litman, D. J. (2004) · Cited by 94
Emotion Detection From Infant Facial Expressions And Cries
Pal, P., Iyer, A. N., & Yantorno, R. E. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2006.1660444 · Cited by 87
Bimodal Fusion of Emotional Data in an Automotive Environment
Hoch, S., Althoff, F., McGlaun, G., & Rigoll, G. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415597 · Cited by 87
Audiovisual recognition of spontaneous interest within conversations
Schuller, B. W., Müeller, R., Hörnler, B., Hoethker, A., Konosu, H., & Rigoll, G. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322201 · Cited by 85
Towards Efficient Multi-Modal Emotion Recognition
Dobrišek, S., Gajšek, R., Mihelič, F., Pavešić, N., & Štruc, V. (2013) · International Journal of Advanced Robotic Systems · doi:10.5772/54002 · Cited by 81
Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition
Karpouzis, K. C., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A. T., Malatesta, L., & Kollias, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_5 · Cited by 75
Introduction to Affective Computing
Calvo, R. A., D’Mello, S. K., Gratch, J. S., & Kappas, A. (2014) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199942237.013.040 · Cited by 74
Self-citationActed vs. natural frustration and delight: Many people smile in natural frustration
Hoque, M. E., & Picard, R. W. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771425 · Cited by 70
Recognizing Affect from Linguistic Information in 3D Continuous Space
Schuller, B. W. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.17 · Cited by 69
Audio-visual emotion recognition using an emotion space concept
Kanluan, I., Grimm, M., & Kroschel, K. (2008) · Cited by 66
Integrating information from speech and physiological signals to achieve emotional sensitivity
Kim, J., André, E., Rehm, M., Vogt, T., & Wagner, J. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-380 · Cited by 65
Savran, A., Cao, H., Shah, M., Nenkova, A., & Verma, R. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388781 · Cited by 62
Proceedings of the 8th European Conference on Speech Communication and Technology
Wrigley, S. N., Brown, G. J., Wan, V., & Renals, S. J. (2003) · Interspeech (USB) · Cited by 62
Hybrid video emotional tagging using users’ EEG and video content
Wang, S., Zhu, Y., Wu, G., & Ji, Q. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-013-1450-8 · Cited by 60
Directing Physiology and Mood through Music: Validation of an Affective Music Player
van der Zwaag, M. D., Janssen, J. H., & Westerink, J. H. D. M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.28 · Cited by 57
Conati, C., Marsella, S., & Paiva, A. (2005) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1040830.1040838 · Cited by 57
String-based audiovisual fusion of behavioural events for the assessment of dimensional affect
Eyben, F., Wöllmer, M., Valstar, M., Güneş, H., Schuller, B. W., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771417 · Cited by 55
Recognizing Human Emotion from Audiovisual Informaiton
Wang, Y., & Guan, L. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415607 · Cited by 55
Multimodal Emotion Classification in Naturalistic User Behavior
Walter, S., Scherer, S. J., Schels, M., Glodek, M., Hrabal, D., Schmidt, M., Böck, R., Limbrecht, K., Traue, H. C., & Schwenker, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21616-9_68 · Cited by 55
LOW-LEVEL FUSION OF AUDIO AND VIDEO FEATURE FOR MULTI-MODAL EMOTION RECOGNITION
Wimmer, M., Schuller, B. W., Arsić, D., Rigoll, G., & Radig, B. (2008) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001082801450151 · Cited by 52
Audio-visual emotion recognition in adult attachment interview
Zeng, Z., Hu, Y., Fu, Y., Huang, T. S., Roisman, G. I., & Wen, Z. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181028 · Cited by 51
Mind and Body: Dialogue and Posture for Affect Detection in Learning Environments
D’Mello, S. K., & Graesser, A. C. (2007) · Cited by 49
Self-citationHuman emotion recognition from videos using spatio-temporal and audio features
Rashid, M., Abu–Bakar, S. A. R., & Mokji, M. M. (2012) · The Visual Computer · doi:10.1007/s00371-012-0768-y · Cited by 43
Patterns, prototypes, performance: classifying emotional user states
Seppi, D., Batliner, A. M., Schuller, B. W., Steidl, S., Vogt, T., Wagner, J., Devillers, L. Y., Vidrascu, L., Amir, N., & Aharonson, V. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-193 · Cited by 41
Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models
Jiang, D., Cui, Y., Zhang, X., Fan, P., Ganzalez, I., & Hichem Sahli, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_64 · Cited by 40
Kalman Filter Based Classifier Fusion for Affective State Recognition
Glodek, M., Reuter, S., Schels, M., Dietmayer, K., & Schwenker, F. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38067-9_8 · Cited by 39
Evidence Theory-Based Multimodal Emotion Recognition
Paleari, M., Benmokhtar, R., & Huet, B. (2009) · Lecture notes in computer science · doi:10.1007/978-3-540-92892-8_44 · Cited by 39
Emotion recognition using bimodal data fusion
Datcu, D., & Rothkrantz, L. J. M. (2011) · International Conference on Computer Systems and Technologies (CompSysTech) · doi:10.1145/2023607.2023629 · Cited by 37
Multi-modal Emotion Recognition Using Canonical Correlations and Acoustic Features
Gajšek, R., Štruc, V., & Mihelič, F. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.1005 · Cited by 35
Güneş, H., & Piccardi, M. (2005) · Lecture notes in computer science · doi:10.1007/11573548_14 · Cited by 32
A systematic discussion of fusion techniques for multi-modal affect recognition tasks
Lingenfelser, F., Wagner, J., & André, E. (2011) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2070481.2070487 · Cited by 32
Emotion recognition based on human gesture and speech information using RT middleware
Vu, H. A., Yamazaki, Y., Dong, F., & Hirota, K. (2011) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzzy.2011.6007557 · Cited by 31
Visual/Acoustic Emotion Recognition
Chen, C., Huang, Y.-K., & Cook, P. R. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521709 · Cited by 28
Natural Affect Data: Collection and Annotation
Afzal, S., & Robinson, P. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1_5 · Cited by 26
Classification of affects using head movement, skin color features and physiological signals
Monkaresi, H., Hussain, M. S., & Calvo, R. A. (2012) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2012.6378149 · Cited by 26
A Multilevel Fusion Approach for Audiovisual Emotion Recognition
Chetty, G., Wagner, M., & Goecke, R. (2015) · Wiley eBooks · doi:10.1002/9781118910566.ch17 · Cited by 21
Fusion of Fragmentary Classifier Decisions for Affective State Recognition
Krell, G., Glodek, M., Panning, A., Siegert, I., Michaelis, B., Wendemuth, A., & Schwenker, F. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37081-6_13 · Cited by 19
Evaluation and Discussion of Multi-modal Emotion Recognition
Rabie, A., Wrede, B., Vogt, T., & Hanheide, M. (2009) · International Conference on Computer and Electrical Engineering · doi:10.1109/iccee.2009.192 · Cited by 19
A new information fusion method for SVM-based robotic audio-visual emotion recognition
Han, M.-J., Hsu, J. I.-H., Song, K., & Chang, F. (2007) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2007.4413990 · Cited by 16
ENSEMBLE APPROACHES TO PARAMETRIC DECISION FUSION FOR BIMODAL EMOTION RECOGNITION
Kim, J., & Lingenfelser, F. (2010) · International Conference on Bio-inspired Systems and Signal Processing · doi:10.5220/0002753204600463 · Cited by 14
Multi-stream Confidence Analysis for Audio-Visual Affect Recognition
Zeng, Z., Tu, J., Liu, M., & Huang, T. S. (2005) · Lecture notes in computer science · doi:10.1007/11573548_123 · Cited by 12
Multimodal Emotion Recognition Using a Spontaneous Filipino Emotion Database
Dy, M. L. I. C., Espinosa, I. V. L., Go, P. P. V., Mendez, C. M. M., & Cu, J. (2010) · International Conference on Human-Centric Computing (HumanCom) · doi:10.1109/humancom.2010.5563314 · Cited by 11
Music-aided affective interaction between human and service robot
Park, J., Jang, G., & Seo, Y.-H. (2012) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/1687-4722-2012-5 · Cited by 11
Emotion Recognition Method Based on Multimodal Sensor Fusion Algorithm
Moon, B.-H., & Sim, K.-B. (2008) · International Journal of Fuzzy Logic and Intelligent Systems · doi:10.5391/ijfis.2008.8.2.105 · Cited by 10
Bimodal Emotion Recognition Based on Speech Signals and Facial Expression
Tu, B., & Feng-qin, Y. (2011) · Advances in intelligent and soft computing · doi:10.1007/978-3-642-25664-6_81 · Cited by 9
Speech Emotion Recognition Exploiting Acoustic and Linguistic Information Sources
Rigoll, G., Müller, R., & Schuller, B. W. (2005) · Cited by 9
Attention and Emotion Based Adaption of Dialog Systems
Hommel, S., Rabie, A., & Handmann, U. (2012) · Topics in intelligent engineering and informatics · doi:10.1007/978-3-642-33959-2_12 · Cited by 8
Visual inference of human emotion and behaviour
Gong, S., Shan, C., & Xiang, T. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322199 · Cited by 6
Crawling to Improve Multimodal Emotion Detection
Cueva, D. R., Gonçalves, R. A., Cozman, F. G., & Barretto, M. R. P. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25330-0_30 · Cited by 6
On dimensions in emotion psychology
Kaernbach, C. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771350 · Cited by 5
Research works printed but not linked: 41
- Baltrušaitis T., Proceedings of the International Conference on Multimedia and Expo (Workshop on Affective Analysis in Multimedia).
- Banda N., Proceedings of the 11th International Conference on Multimodal Interaction (ICMI).
- Chen C.-Y., Proceedings of the IEEE International Conference on Multimedia and Expo. IEEE
- Chetty G., Proceedings of the International Conference on Auditory-Visual Speech Processing, 115--120
- 10.1037/0033-2909.112.1.155 · doi:10.1037/0033-2909.112.1.155 · Paper page
- D'Mello S., Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Lukin et al. (Eds.). IOS Press
- S. K. D'Mello and A. C. Graesser. 2014. Confusion. In International Handbook of Emotions in Education R. Pekrun and L. Linnenbrink-Garcia (Eds.). Routledge New York NY 289--310. S. K. D'Mello and A. C. Graesser. 2014. Confusion. In International Handbook of Emotions in Education R. Pekrun and L. Linnenbrink-Garcia (Eds.). Routledge New York NY 289--310.
- Dy M., Proceedings of the 3rd International Conference on Human-Centric Computing. IEEE
- Emerich S., 2009, Proceedings of the 17th European Signal Processing Conference (EUSIPCO
- Eyben F., 2011, Ninth IEEE International Conference on Automatic Face and Gesture Recognition (FG
- Forbes-Riley K., Proceedings of the 4th Meeting of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 201--208
- Glodek M., Proceedings of the 11th International Workshop on Multiple Classifier Systems, Z.-H. Zhou, F. Roli, and J. Kittler (Eds.). Springer
- Graesser A., Proceedings of the 28th Annual Conference of the Cognitive Science Society, R. Sun and N. Miyake (Eds.). Cognitive Science Society
- Han M., 2007, Proceedings of the IEEE International Conference on Systems, Man and Cybernetics. IEEE
- Haq S., Proceedings of International Conference on Auditory-Visual Speech Processing, 53--58
- Haq S., Proceedings of the International Conference on Auditory-Visual Speech Processing, 185--190
- Hoch S., Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE
- Hoque M., Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG'11)
- Hussain M., Proceedings of the Australasian Data Mining Conference.
- Jiang D., Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, S. D'Mello, A. Graesser, B. Schuller, and J. Martin (Eds.). Springer-Verlag, 609--618
- Joo J.-T., 2007, Proceedings of the 8th Symposium on Advanced Intelligent Systems. 200--204
- Kanluan I., Proceedings of the 16th European Signal Processing Conference.
- Khalali Z., Proceedings of International Joint Conference on Neural Networks. IEEE
- Kim J., Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing. BIOSTEC, 460--463
- 10.1016/j.specom.2005.09.008 · DOI
- Lu K., Proceedings of the 21st International Conference on Pattern Recognition. IEEE
- Monkaresi H., Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE
- Pal P., Proceedings. of the 2006 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE
- Picard R., Affective Computing
- Rigoll G., Proceedings of the 10th International Conference Speech and Computer. 61--67
- 10.1037/1089-2680.2.3.247 · doi:10.1037/1089-2680.2.3.247 · Paper page
- Rozgic V., Proceedings of the Signal and Information Processing Association Annual Summit and Conference. IEEE
- Seppi D., Proceedings of the 9th Annual Conference of the International Speech Communication Association, 601--604
- Sutdiffe A., The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications
- Tomkins S. S., Affect Imagery Consciousness
- Tu B., Proceedings of the 6th International Conference on Intelligent Systems and Knowledge. Springer
- Vu H., IEEE International Conference on Fuzzy Systems. IEEE
- Walter S., Proceedings of the International Conference on Human-Computer Interaction, J. Jacko (Ed.). Springer
- Wang Y., Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE
- Wimmer M., Proceedings of the 3rd International Conference on Computer Vision Theory and Applications, 145--151
- Wöllmer M., Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH'10)
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Identifying Stable Patterns over Time for Emotion Recognition from EEG
Zheng, W.-L., Zhu, J.-Y., & Lu, B.-L. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2712143 · Cited by 990
A survey of multimodal sentiment analysis
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.-F., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 601
Metadata
- Authors
- D'mello, S. K., & Kory, J.
- Venue
- ACM Computing Surveys
- Published
- 2015-02-17
- Volume
- 47
- Issue
- 3
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W1985867508
- Cited by
- 628
- References
- 136
- FWCI
- 4.42
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- NSF Graduate Research Fellowship under 1122374
- Bill & Melinda Gates Foundation
- National Science Foundation (NSF) (ITR 0325428, HCC 0834847, DRL 1235958)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.