The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent
Open accessAbstract
SEMAINE has created a large audiovisual database as a part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally colored conversation. Data used to build the agents came from interactions between users and an "operator” simulating a SAL agent, in different configurations: Solid SAL (designed so that operators displayed an appropriate nonverbal behavior) and Semi-automatic SAL (designed so that users' experience approximated interacting with a machine). We then recorded user interactions with the developed system, Automatic SAL, comparing the most communicatively competent version to versions with reduced nonverbal skills. High quality recording was provided by five high-resolution, high-framerate cameras, and four microphones, recorded synchronously. Recordings total 150 participants, for a total of 959 conversations with individual SAL characters, lasting approximately 5 minutes each. Solid SAL recordings are transcribed and extensively annotated: 6-8 raters per clip traced five affective dimensions and 27 associated categories. Other scenarios are labeled on the same pattern, but less fully. Additional information includes FACS annotation on selected extracts, identification of laughs, nods, and shakes, and measures of user engagement with the automatic system. The material is available through a web-accessible database.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
While previous RGB datasets record FEs in controlled lab environments, AFEW, AMFED and SEMAINE contain faces in naturalistic environments.
“While previous RGB datasets record FEs in controlled lab environments, Acted Facial Expressions In The Wild Database (AFEW) [203], Affectiva-MIT Facial Expression Dataset (AMFED) [204] and SEMAINE [205] contain faces in naturalistic environments.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [205]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019 · see citing works
Ekman, P. (1976) · Medical Entomology and Zoology · Cited by 4,345 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Russell, J. A., & Barrett, L. F. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.76.5.805 · Cited by 3,121 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Self-citationEkman, P. (1999) · Handbook of Cognition and Emotion · doi:10.1002/0470013494.ch3 · Cited by 2,125 · see citing works
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763 · see citing works
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179 · see citing works
Valdez, P., & Mehrabian, A. (1994) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.123.4.394 · Cited by 1,104 · see citing works
Web-Based Database for Facial Expression Analysis
Pantić, M., Valstar, M., Rademaker, R. M., & Maat, L. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521424 · Cited by 1,099 · see citing works
Self-citationThe Effects of Color on Emotions
Kane, B. A., Campbell, J. E., & Katz, G. S. (2013) · PsycEXTRA Dataset · doi:10.1037/e314722004-001 · Cited by 731 · see citing works
Describing the emotional states that are expressed in speech
Cowie, R., & Cornelius, R. R. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00071-7 · Cited by 602 · see citing works
Self-citationThe INTERSPEECH 2010 paralinguistic challenge
Schuller, B. W., Steidl, S., Batliner, A. M., Burkhardt, F., Devillers, L. Y., Müller, C. A., & Narayanan, S. S. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-739 · Cited by 577 · see citing works
FEELTRACE: an instrument for recording perceived emotion in real time
Cowie, R., Douglas‐Cowie, E., Savvidou, S. M., & McMahon, E. R. (2000) · Research Portal (Queen's University Belfast) · Cited by 539 · see citing works
Self-citationLinguistic functions of head movements in the context of speech
McClave, E. Z. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00079-x · Cited by 500 · see citing works
Automatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 466 · see citing works
The Vera am Mittag German audio-visual emotional speech database
Grimm, M., Kroschel, K., & Narayanan, S. S. (2008) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2008.4607572 · Cited by 413 · see citing works
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385 · see citing works
Self-citationMind Reading: The Interactive Guide to Emotions
Junek, W. (2007) · PubMed Central · Cited by 314 · see citing works
Multimodal expression of emotion: Affect programs or componential appraisal patterns?
Scherer, K. R., & Ellgring, H. (2007) · Emotion · doi:10.1037/1528-3542.7.1.158 · Cited by 287 · see citing works
Scherer, K. R., & Ellgring, H. (2007) · Emotion · doi:10.1037/1528-3542.7.1.113 · Cited by 267 · see citing works
Action unit detection using sparse appearance descriptors in space-time video volumes
Jiang, B., Valstar, M., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771416 · Cited by 236 · see citing works
Self-citationBuilding autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185 · see citing works
Self-citationA new emotion database: considerations, sources and scope
Douglas‐Cowie, E., Cowie, R., & Schroeder, N. V. (2000) · Cited by 151 · see citing works
Self-citationBatliner, A. M., Hacker, C., Steidl, S., Nöth, E., D’Arcy, S., Russell, M. J., & Wong, M. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2g6tkyi6gw2r · Cited by 144 · see citing works
Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus
Bänziger, T., & Scherer, K. R. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_42 · Cited by 135 · see citing works
Beyond emotion archetypes: Databases for emotion modelling using neural networks
Cowie, R., Douglas‐Cowie, E., & Cox, C. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.002 · Cited by 130 · see citing works
Self-citationCanal9: A database of political debates for analysis of social interactions
Vinciarelli, A., Dielmann, A., Favre, S., & Salamin, H. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349466 · Cited by 101 · see citing works
Güneş, H., & Pantić, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15892-6_39 · Cited by 86 · see citing works
Douglas‐Cowie, E., Cowie, R., Cox, C., Amier, N., & Heylen, D. (2008) · University of Twente Research Information · Cited by 81 · see citing works
Self-citationAcoustical analysis of spectral and temporal changes in emotional speech
Kienast, M., & Sendlmeier, W. F. (2000) · Cited by 81 · see citing works
String-based audiovisual fusion of behavioural events for the assessment of dimensional affect
Eyben, F., Wöllmer, M., Valstar, M., Güneş, H., Schuller, B. W., & Pantić, M. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771417 · Cited by 54 · see citing works
Self-citationPerceiving emotion: towards a realistic understanding of the task
Cowie, R. (2009) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2009.0139 · Cited by 52 · see citing works
Self-citationNatural affect data — Collection & annotation in a learning context
Afzal, S., & Robinson, P. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349537 · Cited by 48 · see citing works
Cowie, R., Cox, C., Martin, J. C., Batliner, A. M., Heylen, D., & Karpouzis, K. C. (2010) · Cognitive technologies · doi:10.1007/978-3-642-15184-2_13 · Cited by 40 · see citing works
Self-citationMultimodal user’s affective state analysis in naturalistic interaction
Caridakis, G., Karpouzis, K. C., Wallace, M., Kessous, L., & Amir, N. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0030-8 · Cited by 38 · see citing works
Automatic Segmentation of Spontaneous Data using Dimensional Labels from Multiple Coders
Nicolaou, M. A., Güneş, H., & Pantić, M. (2010) · University of Twente Research Information · Cited by 32 · see citing works
Cost-effective solution to synchronised audio-visual data capture using multiple sensors
Lichtenauer, J. F., Shen, J., Valstar, M., & Pantić, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.004 · Cited by 32 · see citing works
Self-citationPiecing Together the Emotion Jigsaw
Cowie, R., & Schröder, M. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-30568-2_26 · Cited by 31 · see citing works
Self-citationCowie, R., Douglas‐Cowie, E., Martin, J. A., & Devillers, L. Y. (2010) · Research Portal (Queen's University Belfast) · Cited by 10 · see citing works
Self-citationAnnotating State of Mind in Meeting Data
Heylen, D., Reidsma, D., & Ordelman, R. (2006) · University of Twente Research Information · Cited by 5 · see citing works
Cowie, Douglas-Cowie, Sneddon, McRorie, Hanratty, McMahon, & McKeown, E. (2010) · Cited by 2 · see citing works
Individual and situational differences in emotional expression
Hanratty, J. (2011) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 46 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., & Mihalcea, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1050 · Cited by 1,416 · see citing works
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · see citing works
DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation
Ghosal, D., Majumder, N., Poria, S., Chhaya, N., & Gelbukh, A. F. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1015 · Cited by 766 · see citing works
Akçay, M. B., & Oğuz, K. (2019) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 743 · see citing works
AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups
Miranda-Correa, J. A., Abadi, M. K., Sebe, N., & Patras, I. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2884461 · Cited by 740 · see citing works
DISFA: A Spontaneous Facial Action Intensity Database
Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.4 · Cited by 793 · see citing works
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sarıyanidi, E., Güneş, H., & Cavallaro, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 689 · see citing works
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D’Mello, S. K., & Kory-Westlund, J. M. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 654 · see citing works
A survey of sentiment analysis in social media
Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-018-1236-4 · Cited by 594 · see citing works
Valstar, M., Schuller, B. W., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., & Pantić, M. (2013) · doi:10.1145/2512530.2512533 · Cited by 581 · see citing works
Self-citationAnalysis of EEG Signals and Facial Expressions for Continuous Emotion Detection
Soleymani, M., Asghari-Esfeden, S., Fu, Y., & Pantić, M. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2436926 · Cited by 557 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances
Poria, S., Majumder, N., Mihalcea, R. F., & Hovy, E. H. (2019) · IEEE Access · doi:10.1109/access.2019.2929050 · Cited by 532 · see citing works
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R. F., Cambria, E., & Zimmermann, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1280 · Cited by 465 · see citing works
Zhang, S., Zhang, S. S., Huang, T., & Gao, W. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 444 · see citing works
Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis
Zhang, Z., Girard, J. M., Wu, Y., Zhang, X., Liu, P., Çiftçi, U. A., Canavan, S. J., Reale, M. J., Horowitz, A., Yang, H., et al. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.374 · Cited by 438 · see citing works
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · see citing works
A Comprehensive Review of Speech Emotion Recognition Systems
Wani, T. M., Gunawan, T. S., Qadri, S. A. A., Kartiwi, M., & Eliathamby Ambikairajah, A. (2021) · IEEE Access · doi:10.1109/access.2021.3068045 · Cited by 382 · see citing works
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B. W., Kotsia, I., & Zafeiriou, S. P. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01158-4 · Cited by 388 · see citing works
Gunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360 · see citing works
Databases, features and classifiers for speech emotion recognition: a review
Swain, M., Routray, A., & Kabisatpathy, P. (2018) · International Journal of Speech Technology · doi:10.1007/s10772-018-9491-z · Cited by 360 · see citing works
Automatic Analysis of Facial Actions: A Survey
Martínez, B., Valstar, M., Jiang, B., & Pantić, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2731763 · Cited by 354 · see citing works
Self-citationLotfian, R., & Busso, C. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2736999 · Cited by 330 · see citing works
Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge
Zafeiriou, S. P., Kollias, D., Nicolaou, M. A., Papaioannou, A., Zhao, G., & Kotsia, I. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.248 · Cited by 297 · see citing works
The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results
Shen, J., Zafeiriou, S. P., Chrysos, G. G., Kossaifi, J., Tzimiropoulos, G., & Pantić, M. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.132 · Cited by 289 · see citing works
On the Acoustics of Emotion in Audio: What Speech, Music, and Sound have in Common
Weninger, F. J., Eyben, F., Schuller, B. W., Mortillaro, M., & Scherer, K. R. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00292 · Cited by 277 · see citing works
DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations
Hu, D., Wei, L., & Huai, X. (2021) · doi:10.18653/v1/2021.acl-long.547 · Cited by 291 · see citing works
FERA 2015 - second Facial Expression Recognition and Analysis challenge
Valstar, M. F., Almaev, T., Girard, J. M., McKeown, G., Mehu, M., Yin, L., Pantic, M., & Cohn, J. F. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7284874 · Cited by 252 · see citing works
Self-citationDeveloping a benchmark for emotional analysis of music
Aljanaki, A., Yang, Y.-H., & Soleymani, M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173392 · Cited by 248 · see citing works
Sewa db: A rich database for audio-visual emotion and sentiment research in the wild
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944808 · Cited by 243 · see citing works
A dataset of continuous affect annotations and physiological signals for emotion analysis
Sharma, K., Castellini, C., van den Broek, E. L., Albu‐Schäffer, A., & Schwenker, F. (2019) · Scientific Data · doi:10.1038/s41597-019-0209-0 · Cited by 246 · see citing works
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
BiERU: Bidirectional emotional recurrent unit for conversational sentiment analysis
Li, W., Shao, W., Ji, S., & Cambria, E. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.057 · Cited by 237 · see citing works
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234 · see citing works
Zhang, D., Wu, L., Sun, C., Li, S., Zhu, Q., & Zhou, G. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/752 · Cited by 230 · see citing works
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
Vijayaraghavan, G., Mala, T., Dhanasekaran, P., & Uma, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 246 · see citing works
Recognition of human emotions using EEG signals: A review
Rahman, M. M., Sarkar, A. K., Hossain, M. A., Hossain, M. S., Islam, M. R., Hossain, M. B., Quinn, J. M. W., & Moni, M. A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104696 · Cited by 226 · see citing works
Schuller, B. W., Valstar, M., Eyben, F., Cowie, R., & Pantić, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388776 · Cited by 225 · see citing works
Self-citationJiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 219 · see citing works
Ruhland, K., Peters, C., Andrist, S., Badler, J. B., Badler, N. I., Gleicher, M. L., Mutlu, B., & McDonnell, R. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12603 · Cited by 218 · see citing works
BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States
Zhalehpour, S., Onder, O., Akhtar, Z., & Erdem, Ç. E. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2553038 · Cited by 216 · see citing works
A Review on Methods and Applications in Multimodal Deep Learning
Jabeen, S., Li, X., Amin, M. S., Bourahla, O. E. F., Li, S., & Jabbar, A. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3545572 · Cited by 211 · see citing works
Emotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 195 · see citing works
A Survey of Textual Emotion Recognition and Its Challenges
Deng, J., & Ren, F. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3053275 · Cited by 201 · see citing works
Building autonomous sensitive artificial listeners
Schroder, M., Bevacqua, E., Cowie, R., Eyben, F., Gunes, H., Heylen, D., ter Maat, M., McKeown, G., Pammi, S., Pantic, M., et al. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.34 · Cited by 185 · see citing works
Self-citationSurvey on audiovisual emotion recognition: databases, features, and data fusion strategies
Wu, C., Lin, J., & Wei, W.-L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2014.11 · Cited by 185 · see citing works
A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space
Xia, R., & Liu, Y. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2512598 · Cited by 182 · see citing works
Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality
Mehta, D., Siddiqui, M. F. H., & Javaid, A. Y. (2018) · Sensors · doi:10.3390/s18020416 · Cited by 187 · see citing works
A review of recent advances in visual speech decoding
Zhou, Z., Zhao, G., Hong, X., & Pietikäinen, M. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.06.004 · Cited by 189 · see citing works
1000 songs for emotional analysis of music
Soleymani, M., Caro, M. N., Schmidt, E. M., Sha, C.-Y., & Yang, Y. (2013) · doi:10.1145/2506364.2506365 · Cited by 179 · see citing works
Sentiment Analysis: From Opinion Mining to Human-Agent Interaction
Clavel, C., & Callejas, Z. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2444846 · Cited by 179 · see citing works
Survey of Deep Representation Learning for Speech Emotion Recognition
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3114365 · Cited by 178 · see citing works
Affect-LM: A Neural Language Model for Customizable Affective Text Generation
Ghosh, S., Chollet, M., Laksana, E., Morency, L., & Scherer, S. J. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1059 · Cited by 161 · see citing works
A Dynamic Appearance Descriptor Approach to Facial Actions Temporal Modeling
Jiang, B., Valstar, M., Martínez, B., & Pantić, M. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2249063 · Cited by 146 · see citing works
Self-citationAnnotation and processing of continuous emotional attributes: Challenges and opportunities
Metallinou, A., & Narayanan, S. S. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553804 · Cited by 143 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Analysis of facial expressions in parkinson's disease through video-based automatic methods
Bandini, A., Orlandi, S., Escalante, H. J., Giovannelli, F., Cincotta, M., Reyes-Garcia, C. A., Vanni, P., Zaccara, G., & Manfredi, C. (2017) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2017.02.006 · Cited by 143 · see citing works
Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 142 · see citing works
Affective Computing: Recent Advances, Challenges, and Future Trends
Pei, G., Li, H., Lu, Y., Wang, Y., Hua, S., & Li, T. (2023) · Intelligent Computing · doi:10.34133/icomputing.0076 · Cited by 161 · see citing works
Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition
Martínez, B., & Valstar, M. (2016) · Springer eBooks · doi:10.1007/978-3-319-25958-1_4 · Cited by 136 · see citing works
Self-citationCai, Y., Li, X., & Li, J. (2023) · Sensors · doi:10.3390/s23052455 · Cited by 143 · see citing works
A Review of Key Technologies for Emotion Analysis Using Multimodal Information
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., & Wang, R. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10287-z · Cited by 137 · see citing works
Capturing Global Semantic Relationships for Facial Action Unit Recognition
Wang, Z., Li, Y., Wang, S., & Ji, Q. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.410 · Cited by 126 · see citing works
A comprehensive survey on automatic facial action unit analysis
Zhi, R., Liu, M., & Zhang, D. (2019) · The Visual Computer · doi:10.1007/s00371-019-01707-5 · Cited by 127 · see citing works
Ham, J., Li, S., Looi, J., & Eastin, M. S. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108161 · Cited by 126 · see citing works
Local Evidence Aggregation for Regression-Based Facial Point Detection
Martínez, B., Valstar, M., Binefa, X., & Pantić, M. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.205 · Cited by 125 · see citing works
Self-citationThe Ordinal Nature of Emotions: An Emerging Approach
Yannakakis, G. N., Cowie, R., & Busso, C. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2879512 · Cited by 121 · see citing works
Self-citationPetridis, S., Martínez, B., & Pantić, M. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.08.014 · Cited by 123 · see citing works
Multimodal Human-Human-Robot Interactions (MHHRI) Dataset for Studying Personality and Engagement
Çeliktutan, O., Skordos, E., & Güneş, H. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2737019 · Cited by 119 · see citing works
CHEAVD: a Chinese natural emotional audio–visual database
Li, Y., Tao, J., Chao, L., Bao, W., & Liu, Y. (2016) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-016-0406-z · Cited by 116 · see citing works
Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion
Atmaja, B. T., Sasou, A., & Akagi, M. (2022) · Speech Communication · doi:10.1016/j.specom.2022.03.002 · Cited by 119 · see citing works
Music mood and human emotion recognition based on physiological signals: a systematic review
Chaturvedi, V., Kaur, A. B., Varshney, V., Garg, A., Chhabra, G. S., & Kumar, M. (2021) · Multimedia Systems · doi:10.1007/s00530-021-00786-6 · Cited by 115 · see citing works
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., de Melo, W. C., Ullah, N., Aslam, M. H., Zeeshan, O., Denorme, T., Pedersoli, M., Koerich, A. L., Bacon, S., Cardinal, P., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00278 · Cited by 115 · see citing works
The ordinal nature of emotions
Yannakakis, G. N., Cowie, R., & Busso, C. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273608 · Cited by 112 · see citing works
Self-citationLearning representations of emotional speech with deep convolutional generative adversarial networks
Chang, J., & Scherer, S. J. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952656 · Cited by 112 · see citing works
The Indian Spontaneous Expression Database for Emotion Recognition
Happy, S. L., Patnaik, P., Routray, A., & Guha, R. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2498174 · Cited by 111 · see citing works
ChaLearn Looking at People and Faces of the World: Face AnalysisWorkshop and Challenge 2016
Escalera, S., Bagheri, M. A., Valstar, M., Torres, M. T., Martínez, B., Baró, X., Escalante, H. J., Guyon, I., Tzimiropoulos, G., Corneanu, C. A., et al. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.93 · Cited by 109 · see citing works
Self-citationiMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
Liu, X., Shi, H., Chen, H., Yu, Z., Li, X., & Zhao, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01049 · Cited by 111 · see citing works
Correcting Time-Continuous Emotional Labels by Modeling the Reaction Lag of Evaluators
Mariooryad, S., & Busso, C. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2334294 · Cited by 107 · see citing works
A survey on deep reinforcement learning for audio-based applications
Latif, S., Cuayáhuitl, H., Pervez, F., Shamshad, F., Ali, H. S., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10224-2 · Cited by 116 · see citing works
Aff-Wild2: Extending the Aff-Wild Database for Affect RecognitionPreprint
Kollias, D., & Zafeiriou, S. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07770 · Cited by 142 · see citing works
Multi-modal emotion recognition using EEG and speech signals
Wang, Q. H., Wang, M., Yang, Y., & Zhang, X. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105907 · Cited by 103 · see citing works
Dynamic interactive multiview memory network for emotion recognition in conversation
Wen, J., Jiang, D., Tu, G., Liu, C., & Cambria, E. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.009 · Cited by 102 · see citing works
Continuous emotion detection using EEG signals and facial expressions
Soleymani, M., Asghari-Esfeden, S., Pantić, M., & Fu, Y. (2014) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890301 · Cited by 98 · see citing works
Towards a Small Set of Robust Acoustic Features for Emotion Recognition: Challenges
Tahon, M., & Devillers, L. Y. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2487051 · Cited by 98 · see citing works
D'Mello, S. K., & Kory, J. (2012) · International Conference on Multimodal Interaction · doi:10.1145/2388676.2388686 · Cited by 122 · see citing works
Multi-Modal Recurrent Attention Networks for Facial Expression Recognition
Lee, J., Kim, S., Kim, S., & Sohn, K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2996086 · Cited by 96 · see citing works
A Survey on Physiological Signal-Based Emotion Recognition
Ahmad, Z., & Khan, N. (2022) · Bioengineering · doi:10.3390/bioengineering9110688 · Cited by 99 · see citing works
Ding, J., Chen, X., Lu, P., Yang, Z., Li, X., & Du, Y. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05439-1 · Cited by 95 · see citing works
First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis
Jacques, J. C. S., Güçlütürk, Y., Pérez, M., Guclu, U., Andújar, C., Baró, X., Escalante, H. J., Guyon, I., van Gerven, M. A. J., van Lier, R., et al. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2930058 · Cited by 91 · see citing works
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Welivita, A., & Pu, P. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.429 · Cited by 101 · see citing works
Stappen, L., Baird, A., Schumann, L., & Schuller, B. W. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3097002 · Cited by 92 · see citing works
Speech emotion recognition approaches: A systematic review
Hashem, A., Arif, M., & Alghamdi, M. (2023) · Speech Communication · doi:10.1016/j.specom.2023.102974 · Cited by 99 · see citing works
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R. F., Gelbukh, A. F., & Cambria, E. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016818 · Cited by 901 · see citing works
Han, J., Zhang, Z., Schmitt, M., Pantić, M., & Schuller, B. W. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123383 · Cited by 85 · see citing works
Chang, W.-Y., Hsu, S.-H., & Chien, J.-H. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.246 · Cited by 84 · see citing works
Spontaneous Expression Recognition Using Universal Attribute Model
Perveen, N., Roy, D., & Mohan, C. K. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2856373 · Cited by 78 · see citing works
- Authors
- McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M.
- Affiliations
- Queen's University Belfast (GB)
- Imperial College London (GB)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2012-01
- Volume
- 3
- Issue
- 1
- Pages
- 5-17
- ISSN
- 1949-3045 (ISSN-L)
- OpenAlex
- W2132555391
- Cited by
- 665
- References
- 46
- Citations by year
- 2026: 19 · 2025: 30 · 2024: 40 · 2023: 50 · 2022: 44 · 2021: 54 · 2020: 48 · 2019: 62 · 2018: 64 · 2017: 57 · 2016: 40 · 2015: 66 · 2014: 41 · 2013: 36 · 2012: 13
- FWCI
- 37.62
- Type
- journal article
- Language
- en
- Open access
- green
- pure.qub.ac.uk
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- University of Twente Research Information (submittedVersion)
- Research Portal (Queen's University Belfast) (submittedVersion, other-oa)
- Funders
- European Commission (211486)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- nonverbal behavior, facial expression coding, affective domain, user engagement, multimodal interaction, human-computer interaction, affective computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
