Emotion recognition in human–computer interaction
Abstract
In this paper, we outline the approach we have developed to construct an emotion-recognising system. It is based on guidance from psychological studies of emotion, as well as from the nature of emotion in its interaction with attention. A neural network architecture is constructed to be able to handle the fusion of different modalities (facial features, prosody and lexical content in speech). Results from the network are given and their implications discussed, as are implications for future direction for the research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(1994) · DigitalGeorgetown (Georgetown University Library) · Cited by 10,542 · see citing works
DESCARTES' ERROR: Emotion, Reason, and the Human Brain
Marg, E. (1995) · Optometry and Vision Science · doi:10.1002/j.1538-9235.1995.tb03988.x · Cited by 8,177 · see citing works
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019 · see citing works
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · see citing works
Russell, J. A., & Barrett, L. F. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.76.5.805 · Cited by 3,121 · see citing works
The nature of emotion: Fundamental questions.
Ekman, P., & Davidson, R. J. (1994) · Oxford University Press eBooks · Cited by 2,839 · see citing works
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,765 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Self-citationEmotion: A Psychoevolutionary Synthesis
Camras, L. A., & Plutchik, R. (1980) · The American Journal of Psychology · doi:10.2307/1422394 · Cited by 2,500 · see citing works
Descartes’ Error. Emotion, Reason and the Human Brain. New York (Grosset/Putnam) 1994.
Damasio, A. R. (1994) · Cited by 2,266 · see citing works
Acoustic profiles in vocal emotion expression.
Banse, R., & Scherer, K. R. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.3.614 · Cited by 2,253 · see citing works
What's basic about basic emotions?
Ortony, A., & Turner, T. J. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.315 · Cited by 2,015 · see citing works
Neural systems for recognizing emotion
Adolphs, R. (2002) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(02)00301-x · Cited by 1,933 · see citing works
Tooby, J., & Cosmides, L. (1990) · Ethology and Sociobiology · doi:10.1016/0162-3095(90)90017-z · Cited by 1,787 · see citing works
Towards a Cognitive Theory of Emotions
Oatley, K., & Johnson-laird, P. N. (1987) · Cognition & Emotion · doi:10.1080/02699938708408362 · Cited by 1,751 · see citing works
Russell, J. A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.102 · Cited by 1,640 · see citing works
Young, P. T., & Arnold, M. B. (1963) · The American Journal of Psychology · doi:10.2307/1419805 · Cited by 1,437 · see citing works
Emotion in the Human Face: Guidelines for Research and an Integration of Findings
Ekman, P., Friesen, W. V., & Ellsworth, P. C. (1972) · Cited by 1,279 · see citing works
Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique.
Ekman, P. (1994) · Psychological Bulletin · doi:10.1037//0033-2909.115.2.268 · Cited by 1,130 · see citing works
Murray, I. R., & Arnott, J. L. (1993) · The Journal of the Acoustical Society of America · doi:10.1121/1.405558 · Cited by 1,098 · see citing works
Hughes, C. W. (1982) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-198205000-00017 · Cited by 975 · see citing works
Basic emotions, relations among emotions, and emotion-cognition relations.
Izard, C. E. (1992) · Psychological Review · doi:10.1037/0033-295x.99.3.561 · Cited by 972 · see citing works
Evolutionary explanations of emotions
Nesse, R. M. (1990) · Human Nature · doi:10.1007/bf02733986 · Cited by 884 · see citing works
Allers, R. (1961) · New Scholasticism · doi:10.5840/newscholas196135345 · Cited by 581 · see citing works
THE DICTIONARY OF AFFECT IN LANGUAGE
Whissell, C. (1989) · Elsevier eBooks · doi:10.1016/b978-0-12-558704-4.50011-6 · Cited by 559 · see citing works
Affect and the Functional Bases of Behavior: On the Dimensional Structure of Affective Experience
Carver, C. S. (2001) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0504_4 · Cited by 357 · see citing works
Neural systems for recognition of emotional prosody: A 3-D lesion study.
Adolphs, R., Damásio, H., & Tranel, D. (2002) · Emotion · doi:10.1037/1528-3542.2.1.23 · Cited by 237 · see citing works
EMOTION: Theory, Research, and Experience
(1990) · Elsevier eBooks · doi:10.1016/b978-0-12-558705-1.50001-4 · Cited by 175 · see citing works
What is an emotion? : classic and contemporary readings
Solomon, R. C. (2003) · Oxford University Press eBooks · Cited by 164 · see citing works
A neurodynamic model of the attentional blink
Fragopanagos, N. F., Kockelkoren, S., & Taylor, J. G. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.03.010 · Cited by 80 · see citing works
Self-citationPlutchik, R. (1997) · Acta Neuropsychiatrica · doi:10.1017/s0924270800036802 · Cited by 66 · see citing works
Paying attention to consciousness
Taylor, J. G. (2003) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2003.10.002 · Cited by 64 · see citing works
Self-citationNeurological and physiological aspects
Arnold, M. B. (1960) · Columbia University Press eBooks · Cited by 61 · see citing works
Attention as Sigma-Pi controlled ACh-based feedback
Taylor, J., Hartley, M., & Taylor, N. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2005.1555839 · Cited by 9 · see citing works
Self-citationSome Reasons to Expect Universal Antecedents of Emotion
Ellsworth, P. C. (1994) · bepress Legal Repository · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Roh, E., Hwang, B.-U., Kim, D., Kim, B.-Y., & Lee, N.-E. (2015) · ACS Nano · doi:10.1021/acsnano.5b01613 · Cited by 927 · see citing works
Social signal processing: Survey of an emerging domain
Vinciarelli, A., Pantic, M., & Bourlard, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.11.007 · Cited by 842 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156 · see citing works
A survey of affect recognition methods: audio, visual and spontaneous expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322216 · Cited by 144 · see citing works
Error weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121 · see citing works
Emotion Recognition through Physiological Signals for Human-Machine Communication
Maaoui, C., & Pruski, A. (2010) · Cutting Edge Robotics 2010 · doi:10.5772/10312 · Cited by 109 · see citing works
“The experimental results indicate that the proposed method provides very stable and successful emotional classification performance as 92% over six emotional states.”
MethodA hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98 · see citing works
Emotion recognition using Kinect motion capture data of human gaits
Li, S., Cui, L., Zhu, C., Li, B., Zhao, N., & Zhu, T. (2016) · PeerJ · doi:10.7717/peerj.2364 · Cited by 71 · see citing works
Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73 · see citing works
Human Emotion Recognition With Relational Region-Level Analysis
Li, W., Dong, X., & Wang, Y. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3064918 · Cited by 43 · see citing works
Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition
Zhao, J., Li, R., Jin, Q., Wang, X., & Li, H. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746910 · Cited by 39 · see citing works
Soladié, C., Salam, H., Pelachaud, C., Stoiber, N., & Séguier, R. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388782 · Cited by 33 · see citing works
Arruti, A., Cearreta, I., Álvarez, A., Lazkano, E., & Sierra, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0108975 · Cited by 21 · see citing works
SAFER: Situation Aware Facial Emotion Recognition
Palash, M., & Bhargava, B. (2023) · arXiv · doi:10.48550/arxiv.2306.09372 · Cited by 2 · see citing works
“Additionally, ER has a wide range of applications in domains such as human-computer interactions [1], law enforcement and surveillance [2], interactive gaming, consumer behavior analysis, customer service [3], and health care [4], among others.”
Background
- Authors
- Fragopanagos, N., & Taylor, J. G.
- Affiliations
- King's College London (GB)
- Corresponding author
- Fragopanagos, N.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2005-05
- Volume
- 18
- Issue
- 4
- Pages
- 389-405
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2080830759
- Cited by
- 414
- References
- 35
- Citations by year
- 2026: 6 · 2025: 21 · 2024: 17 · 2023: 21 · 2022: 20 · 2021: 25 · 2020: 22 · 2019: 26 · 2018: 21 · 2017: 17 · 2016: 18 · 2015: 26 · 2014: 23 · 2013: 29 · 2012: 23
- FWCI
- 17.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- emotion recognition, human-computer interaction, multimodal fusion, facial features, prosody, lexical content, speech emotion, attention, neural network architecture
- MeSH terms
- Attention, Computers, Emotions, Facial Expression, Humans, Learning, Man-Machine Systems, User-Computer Interface, Neural Networks, Computer, Recognition, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review